Project

General

Profile

Bug #30675

Configure the serial console to support both BIOS and UEFI

Added by Eric Loewenthal over 1 year ago. Updated 12 months ago.

Status:
Done
Priority:
No priority
Assignee:
Benno Rice
Category:
OS
Target version:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

The serial console does not have all the output present in the video console. This is a regression, as around 11.0 both consoles had identical output. This may be related to the new bootloader.

See attached images for examples of missing output in the serial console (top) present in the video console (bottom).

The serial console does show something not present in the video console, namely the following block:

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x1d,0x0)/USB(0x0,0x0)/USB(0x0,0x0)/HD(1,GPT,2e5fadf1-6578-11e6-8c8a-0cc47a073ee2,0x22,0x32000)
   BootCurrent: 0002
   BootOrder: 0002 0003 0004 0008 0009 0005
   Probing 24 block devices.....*..+.......+.+.+.+.+.+ done
    ZFS found the following pools: freenas-boot Flexo
    UFS found no partitions
Consoles: EFI console
Command line arguments: loader.efi
Image base: 0xcb202000
EFI version: 2.31
EFI Firmware: American Megatrends (rev 4.655)

FreeBSD/amd64 EFI loader, Revision 1.1
(Sat Mar 24 08:12:38 UTC 2018 root@nemesis.tn.ixsystems.com)
Loading /boot/defaults/loader.conf
console vidconsole is invalid!
Available consoles:
    spinconsole
\
  ______             _   _           _____
 |  ____|           | \ | |   /\    / ____|
 | |__ _ __ ___  ___|  \| |  /  \  | (___
 |  __| '__/ _ \/ _ \ . ` | / /\ \  \___ \
 | |  | | |  __/  __/ |\  |/ ____ \ ____) |
 |_|  |_|  \___|\___|_| \_/_/    \_\_____/

 +============Welcome to FreeNAS============================================+
 |                                                                          |
 |  1. Boot FreeNAS [Enter]                                                 |
 |                                                                          |
 |  Options:                                                                |
 |  2. Configure Boot [O]ptions...                                          |
 |  3. Select Boot [E]nvironment...                                         |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 |                                                                          |
 +==========================================================================+

/boot/kernel/kernel text=0x15e5078 data=0x3c0390+0x5a0ca8 syms=[0x8+0x1a9850+0x8+0x1f650e]
/boot/entropy size=0x1000
/boot/kernel/ispfw.ko size 0x1002b8 at 0x26e7000
/boot/modules/freenas_sysctl.ko size 0x8690 at 0x27e8000
Booting...
Start @ 0xffffffff80325000 ...
EFI framebuffer information:
addr, size     0xf6000000, 0x1d4c00
dimensions     800 x 600
stride         800
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000

(No image comparison for this one, unfortunately, it requires good timing by the fleshy operator. The video console starts showing everything from

KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-STABLE #0 0df1a1c85(freenas/11-stable): Sat Mar 24 08:23:35 UTC 2018

onwards)

The environment is as follows:

FreeNAS-11-MASTER-201803240729
Supermicro X10SLM+-F

serialconsole.PNG (206 KB) serialconsole.PNG Eric Loewenthal, 03/25/2018 06:06 AM
serialconsole2.PNG (179 KB) serialconsole2.PNG Eric Loewenthal, 03/25/2018 06:06 AM
serialconsole4.PNG (165 KB) serialconsole4.PNG Eric Loewenthal, 03/25/2018 06:06 AM
serialconsole3.PNG (180 KB) serialconsole3.PNG Eric Loewenthal, 03/25/2018 06:06 AM
Screen Shot 2018-04-09 at 3.16.23 PM.png (375 KB) Screen Shot 2018-04-09 at 3.16.23 PM.png Vaibhav Chauhan, 04/09/2018 03:16 PM
working_eficonsole.PNG (835 KB) working_eficonsole.PNG Confirmation that the bootloader correctly uses both serial and EFI consoles Eric Loewenthal, 05/02/2018 03:38 PM
tty_comparison.PNG (939 KB) tty_comparison.PNG Serial and EFI console ttys are still not identical Eric Loewenthal, 05/02/2018 03:38 PM
15627
15630
15633
15636
16287
17160
17163

Related issues

Related to FreeNAS - Bug #35467: More serial console fixes for installerClosed
Copied to FreeNAS - Bug #35548: Explicitly make serial console preferred when it is enabledDone

History

#1 Updated by Dru Lavigne over 1 year ago

Eric: to clarify, it is the missing boot menu that is the issue? Also, to make sure we're on the same page, what do you mean by "serial" console vs "video" console?

#2 Updated by Eric Loewenthal over 1 year ago

The issue is that neither console displays the entirety of the boot process. The video console doesn't display the bootloader's menu and the serial console doesn't display a whole bunch of messages (including after boot - see serialconsole4.png - presumably because they're not being sent to this tty).

The video console is the "graphical" console, in other words, the output is VGA (in this case, it is being viewed remotely via the IPMI iKVM application). FreeBSD identifies it as ttyv0.

The serial console is an ASCII output to the server's serial port (FreeNAS is configured to use uart2 at 0x3e8, uart0 is also available at 0x3f8 but is not in use, the server doesn't have a third serial port that could account for uart1. uart2 is being remotely monitored using the IPMI serial redirection functionality, which is piping it into an SSH session established with the BMC). FreeBSD identifies this console as ttyu2, as expected for uart2.

#3 Updated by Dru Lavigne over 1 year ago

  • Assignee changed from Release Council to Erin Clark
  • Target version set to 11.2-RC2

#4 Updated by Erin Clark about 1 year ago

  • Status changed from Unscreened to Not Started
  • Assignee changed from Erin Clark to William Grzybowski

We are currently using "filesystem.file_tail_follow:/var/log/messages:500"; to get these messages. William, do you know if there is something we are doing wrong or is this something that needs to be fixed on your end?

#5 Updated by William Grzybowski about 1 year ago

  • Assignee changed from William Grzybowski to Erin Clark

I dont understand your question Erin. The ticket is about console it seems, how is this related to filesystem.file_tail_follow and UI? I must be missing something.

#6 Updated by Erin Clark about 1 year ago

  • Assignee changed from Erin Clark to Lola Yang

I think Lola implemented this in the new UI, could you look at this?

#7 Updated by Vaibhav Chauhan about 1 year ago

16287

Eric: so we are trying to determine how you are accessing the console(Serial|Video) ? are you using any of VM `VNC|Serial` display options(see attached screenshot). we are asking you this question, so that we can determine if this is a frontend or backend issue ?

#8 Updated by Eric Loewenthal about 1 year ago

This has nothing to do with VMs. The server is neither virtualized nor is it running any VMs. This is FreeNAS itself, with the new bootloader, on a Supermicro X10SLM+-F.

I see two underlying issues here, which may or may not be connected:

Issue 1: The bootloader doesn't recognize vidconsole, only spinconsole. This keeps it from actually displaying anything on VGA. Curiously, it doesn't recognize a serial console either, even though it is outputting on it (as evidenced by the output I pasted).

Issue 2: When the actual OS takes over from the bootloader, it is not outputting all messages on the serial console (VGA console is fine). The output of ttyv0 is complete after FreeBSD takes over from the bootloader, but the output of the serial console, ttyu2 on uart2, is missing messages (see the first screenshot for an example, ttyu2, on the top, is missing everything that is not bolded in ttyv0, bottom).

#9 Updated by Dru Lavigne about 1 year ago

  • Assignee changed from Eric Loewenthal to Vaibhav Chauhan

#10 Updated by Eric Loewenthal about 1 year ago

  • Assignee changed from Vaibhav Chauhan to Eric Loewenthal

This ticket from upstream may be of interest for 1):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209275

#11 Updated by Dru Lavigne about 1 year ago

  • Assignee changed from Eric Loewenthal to Vaibhav Chauhan

#12 Updated by William Grzybowski about 1 year ago

  • Category changed from Console to OS
  • Assignee changed from Vaibhav Chauhan to Alexander Motin

Alexander, any idea in this one? Does this belong to OS/Services?

There is a PR removing vidconsole but not sure that alone will help?

#13 Updated by Alexander Motin about 1 year ago

  • Assignee changed from Alexander Motin to Benno Rice

Init/rc output going to only one console should not be a new issue. I suppose it was like that forever, since serial and video consoles are different TTY devices, and user-space messages are going to only one of them. We may add a control which one to use for that, but not sure we can make them identical, even though I agree that would be very useful.

What's about the loader messages, that indeed likely to be a new issue, related to switch from GRUB to BSD loader. Benno, do you have some system with UEFI and serial console to test that?

#14 Updated by Eric Loewenthal about 1 year ago

Alexander, I'm pretty sure both ttys had complete outputs before. I went back to check an older serial console log I had lying around, and it does include some stuff that it currently missing (the plugin_load messages and the ConnectionRefusedError traceback*, for instance).
Note that this was around September of last year, so it may have been broken independently since then.

BTW: Supermicro X10 and X11 boards support Serial over LAN by SSHing into the BMC (same credentials as the GUI), cding to the sol1 interface (cd system1/sol1, typically) and entering start. The SSH session should now show the local serial console, as configured in the BIOS.
Some other systems may present a busybox shell, which may complicate things.
ASRock boards (such as the C2750D4I in the FreeNAS mini) apparently allow for SoL via ipmitool, of all things: https://forums.freenas.org/index.php?resources/how-to-use-ipmitool-sol-with-asrock-motherboards-alternative-to-jviewer.51/

If you can't get your hands on a system for testing, I might be able to provide access next wednesday.

  • The ConnectionRefusedError tracebacks have been hard to reproduce and are an unrelated, seemingly cosmetic issue. They just happened to be a decent ilustration.

#15 Updated by Nick Wolff about 1 year ago

  • Severity set to Medium

As this seems to always happens and has minor impact of possibly effecting troubleshooting I am marking as medium

#16 Updated by Benno Rice about 1 year ago

  • Status changed from Not Started to In Progress

Looks like the issue is that we set the following in all cases:

console="comconsole,vidconsole" 

This works for BIOS/VESA/VGA/whatnot but doesn't work for EFI which wants:

console="comconsole,efi" 

I'm working on a patch for this.

#17 Updated by Benno Rice about 1 year ago

  • Status changed from In Progress to Ready for Testing

#18 Updated by Eric Loewenthal about 1 year ago

Which nightly is this going to be in? 20180423something?

#19 Updated by Dru Lavigne about 1 year ago

  • Subject changed from Serial console is missing output present in VGA to Configure the serial console to support both BIOS and UEFI
  • Target version changed from 11.2-RC2 to 11.2-BETA1
  • Needs Merging changed from Yes to No

#20 Updated by Eric Loewenthal about 1 year ago

17160
17163

I can confirm that the bootloader is now correctly recognizing the EFI console, in addition to the serial console. (See attached screenshot)

I can also confirm that after the Kernel takes over, the two ttys are still not identical, which I'm pretty sure is a regression from earlier versions, as described in comment number 14. I can't confirm this 100% at the moment, because my other server's SoL isn't working at all and I don't know where my null modem cable is. (See screenshot, taken using a 2018-05-02 nightly)

There is one cosmetic oddity that didn't happen before:

JH ______             _   _           _____
 H|  ____|           | \ | |   /\    / ____|
 H| |__ _ __ ___  ___|  \| |  /  \  | (___
 H|  __| '__/ _ \/ _ \ . ` | / /\ \  \___ \
 H| |  | | |  __/  __/ |\  |/ ____ \ ____) |
 H|_|  |_|  \___|\___|_| \_/_/    \_\_____/

 H+===========HWelcome to FreeNAS===========================================H+
 H|                                                                         H|
 H| HH.HBoot FreeNAS [Enter]                                                H|
 H|                                                                         H|
 H| HOptions:                                                               H|
 H| HH.HConfigure Boot [O]ptions...                                         H|
 H| HH.HSelect Boot [E]nvironment...                                        H|
 H|                                                                         H|
 H|                                                                         H|
 H|                                                                         H|
 H|                                                                         H|
 H|                                                                         H|
 H|                                                                         H|
 H+=========================================================================H+
   H
                                                                              H        

Some characters aren't correct. If the fix isn't immediately obvious, I'll investigate further and bring it up here or upstream, as appropriate, since it's merely a cosmetic issue. My serial setup is expecting a VT100, by the way.

#21 Updated by Dru Lavigne about 1 year ago

  • Related to Bug #35467: More serial console fixes for installer added

#24 Updated by Dru Lavigne about 1 year ago

  • Copied to Bug #35548: Explicitly make serial console preferred when it is enabled added

#27 Updated by Bonnie Follweiler about 1 year ago

Nick Wolff will be testing this on 6/27/18

#28 Updated by Nick Wolff 12 months ago

  • Status changed from Ready for Testing to Passed Testing

Passed testing.

Checked that it is identical output

#29 Updated by Dru Lavigne 12 months ago

  • Status changed from Passed Testing to Done
  • Needs QA changed from Yes to No

Also available in: Atom PDF