Project

General

Profile

Bug #65994

Merge in FreeBSD boot loader fixes

Added by Tibor Gálffy 11 months ago. Updated 8 months ago.

Status:
Ready for Testing
Priority:
No priority
Assignee:
Ryan Moeller
Category:
OS
Target version:
Seen in:
Severity:
Low Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Dear Developer,
Upgrade (and 1st reboot) went thru without any problem, but after 2nd reboot system was unable tu start (gptzfsboot: error 32 lba 1).
Downloaded and install 11.1-U6 again (I have got backuped config). Splited mirrored boot device in order to have backuped 11.1-u6 system. Downloaded 11.2 image. I try to prepare installation media, but written by Unetbootin has not booted (Invalid or corrupted kernel image). Rufus (v 3.3 and 3.4) caused BSOD when 99.6% of immage was written. My PC has not started with this pendrive attached. Connecting this pendrive caused immediate BSOD. I was able to repair it only with Gparted under Linux creating a new partition table (Gparted reportet corrupter partition table).
Only Image writer was able to write image to pendrive, but after booting I got just a blinking "\".
It is an HP N54L Microserver with 5x3TiB disks in raidz2.
The simmilar issue I had a few weeks ago, I tried to upgrade to the beta version:
https://www.facebook.com/groups/118037613554/permalink/10156713699418555/

2019-01-27 15.02.42.jpg (2.76 MB) 2019-01-27 15.02.42.jpg Tibor Gálffy, 01/27/2019 07:46 AM
2019-01-27 15.42.17.jpg (2.98 MB) 2019-01-27 15.42.17.jpg Tibor Gálffy, 01/27/2019 08:09 AM
50697
50705

Related issues

Related to FreeNAS - Bug #46795: FreeBSD bootloaderClosed
Has duplicate FreeNAS - Bug #66513: RE Bug #65994 Unable to Upgrade vom 11.1 to 11.2 HP N54 L same issueClosed
Has duplicate FreeNAS - Bug #62721: OS doesn't boot after updateClosed
Has duplicate FreeNAS - Bug #68700: 11.2-RELEASE-U1 will *NOT BOOT* on a HP Microserver N40LClosed
Has duplicate FreeNAS - Bug #62847: HP Gen8 Microserver fails to boot from internal SD slotClosed
Has duplicate FreeNAS - Bug #65517: gptzfsboot errors and BTX halted on fresh FreeNAS-11.2-RELEASE install to USBClosed
Has duplicate FreeNAS - Bug #72964: 11.2 bootloader broken, gptzfsboot error 32, now incompatible on previous running hardwareClosed
Has duplicate FreeNAS - Bug #27432: bsdloader fails on Dell R730xd in UEFI modeClosed

History

#1 Updated by Tibor Gálffy 11 months ago

Sorry, I forgot:
Boot device: pair of mirrored Sandisk Ultra Fit SDCZ43 32GB USB 3.0
As installation media I used several 8GB USB2.0 flash disks, all of them has same issue.
Image I wrote under Win7 64 bit, corrupted flash drive repaired with Gparted, an old Ubuntu live installation CD (v. 12.04).
Hardware:
HP N54L Microserver AMD Turion(tm) II Neo N54L Dual-Core Processor
Memory: KINGSTON KVR1333D3E9SK2/16G, DDR3 2x8GB 1333MHz ECC CL9
Disks: 4x 3.0 TB WD Greem + 1x3.Tb WD Red

#2 Updated by William Grzybowski 11 months ago

  • Assignee changed from Release Council to Alexander Motin

#4 Updated by Christoph Borsbach 11 months ago

Hey all,
I just wanted to chime in that I noticed the same behavior:
- Upgrade 11.1-U6 --> 11.2-U1
- All is well for the first reboot.
- The error described ((gptzfsboot: error 32 lba 1)) appeared for me on the second reboot, days later.
- I have not examined the USB-Stick in question very closely, but an attempt to do an "upgrade installation" on it failed. (as in: Attempt to boot gives the same error: (gptzfsboot: error 32 lba 1))
- For the moment I reinstalled FreeNAS on another USB Stick.

Let me know if I can test anything.

My Hardware is similar to the OP:
HP N54L Microserver AMD Turion(tm) II Neo N54L Dual-Core Processor
16GB Sandisk USB2 Stick as Bootdevice
Various HDDs

#5 Updated by Alexander Motin 11 months ago

  • Assignee changed from Alexander Motin to Ryan Moeller

#6 Updated by Alexander Motin 11 months ago

  • Has duplicate Bug #66513: RE Bug #65994 Unable to Upgrade vom 11.1 to 11.2 HP N54 L same issue added

#7 Updated by Ryan Moeller 10 months ago

  • Status changed from Unscreened to Screened

#8 Updated by Eric Bonharme 10 months ago

Hey folks,

Just wanted t say I for the same hardware (only 8G memory) and got the issue: failed to boot second time around (after I’d upgraded all my jails - thankfully I had snapshots to roll back to)

Let us know if you need more detailed report,

Cheers

#9 Updated by Ryan Moeller 10 months ago

  • Has duplicate Bug #62721: OS doesn't boot after update added

#10 Updated by Ryan Moeller 10 months ago

  • Subject changed from Unable to upgrade 11.1-U6->11.2 to HP N45L Microserver unable to boot
  • Seen in changed from 11.1-U6 to 11.2-RELEASE

#11 Updated by Ryan Moeller 10 months ago

  • Subject changed from HP N45L Microserver unable to boot to HP N54L/N40L Microserver unable to boot

#12 Updated by Ryan Moeller 10 months ago

  • Subject changed from HP N54L/N40L Microserver unable to boot to HP N54L/N40L Microserver unable to boot from SanDisk Ultra Fit USB sticks

Someone on the forum has found a workaround by not using SanDisk Ultra Fit USB sticks for the boot device:

https://forums.freenas.org/index.php?threads/hp-proliant-microserver-n40l-upgrade-to-11-2-beta-1-and-2-fails.68952/page-3#post-481935

If there are other boot devices people have this issue with, or if changing to a different boot device helps, either case would be good to have reports of here!

#13 Updated by Gábor ADORJÁNI 10 months ago

Ryan Moeller wrote:

Someone on the forum has found a workaround by not using SanDisk Ultra Fit USB sticks for the boot device:

https://forums.freenas.org/index.php?threads/hp-proliant-microserver-n40l-upgrade-to-11-2-beta-1-and-2-fails.68952/page-3#post-481935

If there are other boot devices people have this issue with, or if changing to a different boot device helps, either case would be good to have reports of here!

Possible, but why has it been working until l1.1-U6? I would rather suspect the native boot loader.

I've been using these Sandisk Cruzer Fits for years, both the older USB2 and the current, USB3 versions. In my NAS, and also in an Intel NUC with SmartOS. There hasn't been any problems at all (except for the unavoidable burnout every couple of years).

#14 Updated by Ryan Moeller 10 months ago

  • Status changed from Screened to In Progress
  • Severity changed from New to Low Medium

I found reports of people having trouble booting other operating systems from the SanDisk Ultra Fit USB devices as well:
https://forums.sandisk.com/t5/All-SanDisk-USB-Flash-Drives/can-t-boot-a-Mint-Cin-17-1-live-Sandisk-Ultra-Fit-3-0-32GB/td-p/338326

It's a long thread that will take some time to read through, but I have seen some helpful information so far. The fact that these devices were previously working with GRUB suggest that whatever firmware quirk they have should have a workaround. I'll also be looking for related tickets in redmine in light of this new information, as this issue may not be specific to the AMD Microservers.

#15 Updated by Tibor Gálffy 10 months ago

Interesting...
In the next couple of days I will try to move my boot environment to another flash disk(s). I will let you know what will have been happened.

#16 Updated by Tibor Gálffy 10 months ago

I have replaced mirrored Sandisks with TDK LoR Micro flash-disk. Installed 11.2 and uploaded backuped config. Everything seems to be working.
Do we have solved the issue? :-)
The only strange thing is, that I have some problem with preparing installation media. I got BSOD on my Win7 machine during and after writing image to flash-disk...
(It was not a Sandisk)

#17 Updated by Tibor Gálffy 10 months ago

...to be absolutely accurate I have installed FreeNAS-11.2-RELEASE-U1.

#18 Updated by Ryan Moeller 10 months ago

Tibor Gálffy wrote:

Everything seems to be working.

That is excellent news indeed, Tibor! Now I'm wondering, would you be able to try temporarily booting FreeNAS from the Ultra Fit in a different machine (even like a laptop or desktop) to see if it might not be specific to the HP Microserver?

#19 Updated by Tibor Gálffy 10 months ago

50697
50705

I have made some experiments with my old desktop ( http://speccy.piriform.com/results/qVhi1R7JcUVb5mExMkN2O1m ), and HP EliteBook 820 G2 laptop. Be careful making conlusions about results with the desktop, already it had made some strange things. For example on rare occasions reboot caused by attaching an USB device, or just touching USB connectors.

Tried to start desktop several times from installation flash-disk with 11.2. I could not start it. System hanged before boot in state described on the attached photo. I tried boot without pen-drive, pressed F12 and then attached pen-drive. System has freezed. Attaching flash drive to running windows caused BSOD. Also I had difficulties to prepare this media now, and also in the past. Only with 11.2 I had this issue, previous versions were without this problem.
On my laptop start OK.

Tried to start Ultra Fit with 11.1-U7 on desktop. I got the same hanged BIOS. But I could attach it to running Windows.
Ultra Fit on laptop started without problem. I made successful update to 11.2. I tried to restart it several time, no problem. But did not boot in Microserver, and BIOS hangs in desktop.
I also tried to start desktop from TDK LoR Micro (11.2 system full functional is Microserver), BIOS hanged same way.

Ultra Fit I rewrite by Ubuntu 18.04.1 image using Rufus. Booting is OK, but the live system did not start. Photo with error message is attached. Same issue I got on laptop. The same Ubuntu image written on another flash drive started normally.
Finally I rewrite Ultra Fit by Win7 install image, it started normally on desktop and laptop too.

#20 Updated by Ryan Moeller 10 months ago

#21 Updated by Ryan Moeller 10 months ago

  • Subject changed from HP N54L/N40L Microserver unable to boot from SanDisk Ultra Fit USB sticks to Unable to boot from SanDisk Ultra Fit 3.0 USB sticks
  • Status changed from In Progress to Closed
  • Reason for Closing set to Cannot Reproduce

Thank you. These results suggest that there are multiple factors at play to bring out the quirk of the USB drive.

I have found through the SanDisk forum thread (linked previously) that there were a few production runs of various SanDisk USB devices that have this quirk affecting the boot process for several operating systems. I suspect it also depends on the particular USB controller hardware and possibly the BIOS/UEFI firmware as well.

I obtained a 32GB SanDisk Ultra Fit 1.0 (USB 3.0) but have been unable to reproduce the boot issues, so I suspect I have one from a newer run where the issue has been fixed. Furthermore, the BSD loader relies on BIOS/UEFI firmware for disk access rather than having its own USB stack, unlike some other bootloaders. As such, I do not have much hope for finding a fix for this hardware quirk.

The workaround for this issue is to use a different USB drive.

#22 Updated by Ryan Moeller 10 months ago

  • Related to Bug #68700: 11.2-RELEASE-U1 will *NOT BOOT* on a HP Microserver N40L added

#23 Updated by Dru Lavigne 10 months ago

  • Target version changed from Backlog to N/A

#24 Updated by Ryan Moeller 10 months ago

  • Related to deleted (Bug #68700: 11.2-RELEASE-U1 will *NOT BOOT* on a HP Microserver N40L)

#25 Updated by Ryan Moeller 10 months ago

  • Has duplicate Bug #68700: 11.2-RELEASE-U1 will *NOT BOOT* on a HP Microserver N40L added

#26 Updated by John Pierce 10 months ago

I had the same problem as detailed in the 'will NOT BOOT on HP Microserver' thread thats been linked to this one with several different Sandisk sticks.

Today, as I summarized in that other thread, I tested a Samsung USB 3.1 FIT Plus 32GB (model MUF-32AB/AM), and a Kingston Data Tracker G4 32gb (DTIG4/32GB), both flashed with the 11.2-U1-Release.iso, using Rufus, and both sticks booted.

I need to perform the same test one last time using the 3 or 4 different model Sandisks I have handy to ensure that /all/ of them are indeed problematic...

edit: the Samsung FIT crashed with data errors when I tried to use it as part of aa freenas-boot mirror.

#27 Updated by John Pierce 9 months ago

more data points...

USB sticks that do NOT boot 11.2-RELEASE-U1.iso on my HP Microserver N40L with the last 10/01/2013 bios...

  • SanDisk Ultra FIT 64GB USB 3.1 - SDCZ-430-064G
  • SanDisk Ultra FIT 32GB USB 3.0 - SDCZ-43-032G
  • SanDisk Ultra 32GB - SDCZ45-032G
  • SanDisk Ultra 16GB - SDCZ45-016G

#28 Updated by Ryan Moeller 9 months ago

John, since you seem to have a good collection, could you post the output of diskinfo -v DEVICE for each of those?

#29 Updated by John Pierce 9 months ago

sandisk sdcz43 ULTRA FIT usb 3.0 32gb:

ugen1.2: <SanDisk Ultra Fit> at usbus1
umass0 on uhub3
umass0: <SanDisk Ultra Fit, class 0/0, rev 2.10/1.00, addr 2> on usbus1
umass0: SCSI over Bulk-Only; quirks = 0xc100
umass0:7:0: Attached to scbus7
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <SanDisk Ultra Fit 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C531001600908100283
da0: 40.000MB/s transfers
da0: 29327MB (60062500 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

/dev/da0
512 # sectorsize
30752000000 # mediasize in bytes (29G)
60062500 # mediasize in sectors
0 # stripesize
0 # stripeoffset
3738 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
SanDisk Ultra Fit # Disk descr.
4C531001600908100283 # Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Not_Zoned # Zone Mode

sandisk sdcz430-064g ULTRA FIT usb 3.1 64gb:

ugen1.2: <SanDisk Ultra Fit> at usbus1
umass0 on uhub3
umass0: <SanDisk Ultra Fit, class 0/0, rev 2.10/1.00, addr 2> on usbus1
umass0: SCSI over Bulk-Only; quirks = 0xc100
umass0:7:0: Attached to scbus7
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <SanDisk Ultra Fit 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C531001610107122511
da0: 40.000MB/s transfers
da0: 58656MB (120127488 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

/dev/da0
512 # sectorsize
61505273856 # mediasize in bytes (57G)
120127488 # mediasize in sectors
0 # stripesize
0 # stripeoffset
7477 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
SanDisk Ultra Fit # Disk descr.
4C531001610107122511 # Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Not_Zoned # Zone Mode

sandisk sdcz44-032g ultra 32gb:

ugen1.2: <SanDisk Ultra> at usbus1
umass0 on uhub3
umass0: <SanDisk Ultra, class 0/0, rev 2.00/1.26, addr 2> on usbus1
umass0: SCSI over Bulk-Only; quirks = 0xc100
umass0:7:0: Attached to scbus7
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <SanDisk Ultra 1.26> Removable Direct Access SPC-3 SCSI device
da0: Serial Number 200515357111ADC32087
da0: 40.000MB/s transfers
da0: 30532MB (62530624 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

/dev/da0
512 # sectorsize
32015679488 # mediasize in bytes (30G)
62530624 # mediasize in sectors
0 # stripesize
0 # stripeoffset
3892 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
SanDisk Ultra # Disk descr.
200515357111ADC32087 # Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Not_Zoned # Zone Mode

sandisk sdcz44-016g ultra 16gb:

ugen1.2: <SanDisk Firebird USB Flash Drive> at usbus1
umass0 on uhub3
umass0: <SanDisk Firebird USB Flash Drive, class 0/0, rev 2.00/1.26, addr 2> on usbus1
umass0: SCSI over Bulk-Only; quirks = 0xc100
umass0:7:0: Attached to scbus7
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <SanDisk Ultra 1.26> Removable Direct Access SPC-3 SCSI device
da0: Serial Number 4C532000000718112062
da0: 40.000MB/s transfers
da0: 15267MB (31266816 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

/dev/da0
512 # sectorsize
16008609792 # mediasize in bytes (15G)
31266816 # mediasize in sectors
0 # stripesize
0 # stripeoffset
1946 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
SanDisk Ultra # Disk descr.
4C532000000718112062 # Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Not_Zoned # Zone Mode

kingston DTIG4/32GB DataTraveler G4 32GB

ugen1.2: <Kingston DataTraveler 3.0> at usbus1
umass0 on uhub3
umass0: <Kingston DataTraveler 3.0, class 0/0, rev 2.10/1.10, addr 2> on usbus1
umass0: SCSI over Bulk-Only; quirks = 0x8100
umass0:7:0: Attached to scbus7
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 60A44C4251CAE2C1372A146E
da0: 40.000MB/s transfers
da0: 29568MB (60555264 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

/dev/da0
512 # sectorsize
31004295168 # mediasize in bytes (29G)
60555264 # mediasize in sectors
0 # stripesize
0 # stripeoffset
3769 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
Kingston DataTraveler 3.0 # Disk descr.
60A44C4251CAE2C1372A146E # Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Not_Zoned # Zone Mode

Samsung FIT 32 USB 3.1

ugen1.2: <Samsung Flash Drive FIT> at usbus1
umass0 on uhub3
umass0: <Samsung Flash Drive FIT, class 0/0, rev 2.10/11.00, addr 2> on usbus1
umass0: SCSI over Bulk-Only; quirks = 0x4000
umass0:7:0: Attached to scbus7
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <Samsung Flash Drive FIT 1100> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 0324618110004694
da0: 40.000MB/s transfers
da0: 30594MB (62656641 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

/dev/da0
512 # sectorsize
32080200192 # mediasize in bytes (30G)
62656641 # mediasize in sectors
0 # stripesize
0 # stripeoffset
3900 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
Samsung Flash Drive FIT # Disk descr.
0324618110004694 # Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Not_Zoned # Zone Mode

#30 Updated by Dru Lavigne 9 months ago

  • Status changed from Closed to Unscreened
  • Target version changed from N/A to Backlog
  • Reason for Closing deleted (Cannot Reproduce)

#31 Updated by Ryan Moeller 9 months ago

  • Status changed from Unscreened to Blocked
  • Reason for Blocked set to Dependent on a related task to be completed

Updates to the bootloader are in the pipeline to be merged to the stable/11 branch upstream. I am hopeful one of these changes will resolve this issue, because the problem isn't present in my testing with FreeBSD 12. There is significant code reorganization and cleanup that makes it impractical to cherry-pick fixes out of order for testing. I am tracking the changes upstream for now.

#32 Updated by Ryan Moeller 9 months ago

  • Reason for Blocked changed from Dependent on a related task to be completed to Waiting for feedback

There has been significant work to the bootloader merged into our freenas/11-stable branch. If someone would like to test a build:
https://download.freenas.org/11.3/MASTER/201902231007/x64/FreeNAS-11.3-MASTER-201902231007-e132e27.iso

#33 Updated by Juergen Schramm 9 months ago

Can confirm Sandisk USB 32 GB Stick runs fine, no waiting during bootup as in 11.2x on a Transcend USB Stick. Import Config file from 11.1U7 without any errors.

New WebGui runs very well as 11.1U7 on a HP Microserver N54.

#34 Updated by Ryan Moeller 9 months ago

That's encouraging. I'd like to see feedback from a few more testers, and I'll look into what might be possible to backport for 11.2.

#35 Updated by Ryan Moeller 9 months ago

I've also confirmed the issue is fixed in 11.3 for the test setup I'm using in the lab. Still interested in seeing more feedback.

#36 Updated by Christian Weissmann 9 months ago

I can confirm that the 11.3 solved the issue for me too.
I was using a SanDisk Extreme 16GB in the first place which worked perfect.

I ordered 2x SanDisk Ultra Flair 16GB for a mirrored installation and got stock at the first boot after the installation.
I tried everything and nothing worked but this new version did the trick.

Thanks!

#37 Updated by Ryan Moeller 8 months ago

  • Status changed from Blocked to In Progress
  • Reason for Blocked deleted (Waiting for feedback)

At this point I'm confident the issue is fixed for 11.3, and I am now attempting to determine if a minimal patch for 11.2 is possible.

#38 Updated by Ryan Moeller 8 months ago

  • Target version changed from Backlog to 11.2-U4

#39 Updated by Ryan Moeller 8 months ago

  • Status changed from In Progress to Ready for Testing
  • Needs Merging changed from Yes to No

#41 Updated by Dru Lavigne 8 months ago

  • Subject changed from Unable to boot from SanDisk Ultra Fit 3.0 USB sticks to Merge in FreeBSD boot loader fixes

#42 Updated by Ryan Moeller 8 months ago

  • Related to Bug #62847: HP Gen8 Microserver fails to boot from internal SD slot added

#43 Updated by John Pierce 8 months ago

Ryan, re #42, this bug #65994 is about the G7 AMD Neo (N40L, N54L) series of Microservers, which are quite different than the Intel based Gen8 models.

now maybe the 'new' FreeBSD boot loader is the common thread, I'll buy that.

#44 Updated by Ryan Moeller 8 months ago

  • Related to deleted (Bug #62847: HP Gen8 Microserver fails to boot from internal SD slot)

#45 Updated by Ryan Moeller 8 months ago

  • Has duplicate Bug #62847: HP Gen8 Microserver fails to boot from internal SD slot added

#46 Updated by Ryan Moeller 8 months ago

John Pierce wrote:

now maybe the 'new' FreeBSD boot loader is the common thread, I'll buy that.

Indeed, I believe this issue is not specific to only the combinations of hardware described here.

#47 Updated by Ryan Moeller 8 months ago

  • Has duplicate Bug #65517: gptzfsboot errors and BTX halted on fresh FreeNAS-11.2-RELEASE install to USB added

#48 Updated by Ryan Moeller 8 months ago

  • Has duplicate Bug #72964: 11.2 bootloader broken, gptzfsboot error 32, now incompatible on previous running hardware added

#49 Updated by Ryan Moeller 8 months ago

  • Has duplicate Bug #27432: bsdloader fails on Dell R730xd in UEFI mode added

Also available in: Atom PDF