Project

General

Profile

Bug #7435

FreeNAS 9.3 Install failed with Sandisk Ultra Fit USB (USB_ERR_IOERROR)

Added by David Y about 4 years ago. Updated over 1 year ago.

Status:
Closed: Not To Be Fixed
Priority:
Expected
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Build FreeNAS-11.0-U4 (54848d13b)
Platform Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Memory 24435MB

ChangeLog Required:
No

Description

Encountered an error installing 9.3 with with Sandisk Ultra Fit drive as I've installed 9.3 successfully on the same machine with other non Sandisk USB drives. The install process works fine, but after the reboot, it errors out with USB_ERR_IOERROR and drops into manual mount.

FreeNAS image used: FreeNAS-9.3-STABLE-201412312006.iso
Hardware: Dell PowerEdge R515 with Perc H200 card

Console Output:

uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
uhub2: 3 ports with 3 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub4: 3 ports with 3 removable, self powered
failure at /fusion/jkh/93/FN/FreeBSD/src/sys/dev/mps/mps_sas_lsi.c:667/mpssas_add_device()! Could not get ID for device with handle 0x0009
mpssas_fw_work: failed to add device with handle 0x9
...
usb_alloc_device: set address 3 failed (USB_ERR_TIMEOUT, ignored)
usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_TIMEOUT, ignored)
usbd_setup_device_descr: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_TIMEOUT, ignored)
usbd_setup_device_descr: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_TIMEOUT, ignored)
usbd_setup_device_descr: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
ugen2.3: <Unknown> at usbus2 (disconnected)
uhub_reattach_port: could not allocate new device
...
Trying to mount root from zfs:freenas-boot/ROOT/default []...
Mounting from zfs:freenas-boot/ROOT/default failed with error 2.

Loader variables:
vfs.root.mountfrom=zfs:freenas-boot/ROOT/default

Manual root filesystem specification:
<fstype>:<device> [options]
Mount <device> using filesystem <fstype>
and with specified (optional) option list.

eg. ufs:/dev/da0s1a
zfs:tank
cd9660:/dev/acd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)

? List valid disk boot devices
. Yield 1 second (for background tasks)
<empty line> Abot manual input

mountroot>

degraded.PNG (121 KB) degraded.PNG Jason Gould, 03/06/2015 12:03 PM
2359

Related issues

Related to FreeNAS - Bug #6857: GEOM_MULTIPATH does strange things with (formerly?) GPT devicesClosed2014-11-29
Related to FreeNAS - Bug #6842: Installer doesn't check mirrored boot devices for GPTClosed2014-11-28

History

#1 Updated by Jordan Hubbard about 4 years ago

  • Category set to 1
  • Assignee set to Sean Fagan
  • Target version set to Unspecified

Here's another candidate!

#3 Updated by Sean Fagan about 4 years ago

Candidate for what?

Looks like this thumb drive doesn't work with freebsd.

You want I should order one?

#4 Updated by Sean Fagan about 4 years ago

  • Status changed from Unscreened to Screened

#5 Updated by David Y about 4 years ago

Came across Bug# 6857 https://bugs.pcbsd.org/issues/6857

Could the issue be related to Sandisk using non-printable characters in T10-based identifier as mentioned in the other bug?

The fix was supposedly in the 9.3 branch at commit:0322603d. Any idea if the image I used from FreeNAS-9.3-STABLE-201412312006.iso had the fix in it?

Thanks

#6 Updated by Cpu Roast about 4 years ago

I have 2 of these Sandisk Ultra Fit USB sticks, the 16GB model, successfully running 9.3 in a mirror.

I've cleanly installed FN with diff builds twice on these and have no such install issues.

#7 Updated by Cpu Roast about 4 years ago

Cpu Roast wrote:

I have 2 of these Sandisk Ultra Fit USB sticks, the 16GB model, successfully running 9.3 in a mirror.

I've cleanly installed FN with diff builds twice on these and have no such install issues.

#8 Updated by David Y about 4 years ago

Thanks for the feedback, what system are you running on? Were you able to run it successfully without it in a mirror? Bug 6857 seem to suggest that the non-printable character in the vendor ID triggers the multipath module even when you don't have 2 drives in there (which is my case).

I'm also running the 32GB model. Not sure if there's a limitation with the size of the drive or if the 2 models are just built differently.

#9 Updated by Justin Wong about 4 years ago

As another datapoint, I have the same drive, same capacity (32gb), and get stuck in the same spot.

#10 Updated by Sean Fagan about 4 years ago

Well, I've ordered a couple of these (one 16g and one 32g), we'll see what happens with them.

#11 Updated by Sean Fagan about 4 years ago

  • Status changed from Screened to Investigation

So the good news is that it reproduces with the ones that arrived today (the 32g -- I'm doing I/O tests on the 16g right now).

Bad news is I have no idea why; the 16g drive mounts just fine on FreeBSD 9.3, and of course it mounted during the installer.

I'll try to see what the 32g looks like to my FreeBSD box.

#12 Updated by Sean Fagan about 4 years ago

  • Category changed from 1 to 18

Okay, it imported just fine there.

It also imports just fine in FreeNAS. So it's just a problem with booting?

Have to change this to kernel, I think. Maybe grub.

#13 Updated by Sean Fagan about 4 years ago

  • Assignee changed from Sean Fagan to Xin Li

Xin, I have this thumb drive (although it's currently got the pool name "bad-boot" so that would need to be changed), and I don't know why booting wouldn't work. Do you?

#14 Updated by Hector Gongora about 4 years ago

I am also having the same issue but with the SanDisk ultra 32 gb regular usb 3.0 drive. My model is not the fit but shows same symptoms.

#15 Updated by Sean Fagan about 4 years ago

  • Assignee changed from Xin Li to Kris Moore

Sending to Kris for the moment, to see if he has any ideas. (I, personally, am wondering if the same change made to the kernel needs to be made to the boot loader, since the device works as everything other than root.)

I have a pair of them to use.

#16 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

Sean,

I'm not sure this has anything to do with GRUB. GRUB seems to be getting past this, and then its failing when the kernel loads / tries to mount root right? Grub doesn't do any specific USB passthrough or anything, once the kernel is loaded, it's on its own. The only environment variable GRUB sets is the vfs.root.mountfrom variable, which has nothing to do with USB.

#17 Updated by Jason Gould about 4 years ago

Seeing the same thing on a pair of 32GB sandisk ultra fit usb's. Strangest thing happened. I had created the mirrored usb's on another PC. The first time I plugged them into my machine everything looked to be working. IN the web GUI however it showed the boot device was operating in a degraded state because 1 of the sticks wasn't working correctly. So I shutdown and am trying to just get it installed on 1 stick and to boot. It keeps failing with the same error as above. I'm going to try installing and booting from the OTHER USB drive to see if maybe only 1 of them boots correctly and the other is different in some way? Will report back if I find anything.

#18 Updated by Jason Gould about 4 years ago

Well, same issue on both sticks. I have no idea how I got it to at least boot the first time.
Moved back over to an old 9.2.1.7 install on my old trusty lexar 4GB and it booted right up.
Hope someone figures this out.

#19 Updated by Jason Gould about 4 years ago

Same exact install on the USB stick, bring it home, plug into home desktop PC, boots up no problem. No IOERROR's and no failed with error 2. There are some interesting lines right after it booted correctly. See image at;
http://s22.postimg.org/u9njufndd/IMG_20150223_202551.jpg

I have a feeling that now if I try to boot this on the machine at work it will work like last time. Will attempt tomorrow.

#20 Updated by Jason Gould about 4 years ago

Just to update a bit on this. Since I'm building a new FreeNAS system, I've been able to do some tests.
On a SuperMicro X10SL7-F motherboard, with XHCI off, and 2 of these USB sticks installed in the back USB 2.0 ports, I was able to boot into the latest release. This is with them installed to both USB sticks in a mirror. I still see some of the USB_ERROR messages. HOWEVER, if I try to use either of them by themselves, it fails to boot as mentioned above.

I feel like this has something to do with GRUB. Perhaps the delay to detect USB drives isn't sufficient for these USB sticks. See https://bugs.pcbsd.org/issues/7254 and https://bugs.pcbsd.org/issues/6967, as I feel they are related.

#21 Updated by Jason Gould about 4 years ago

Spoke to soon. Although it booted the first time using the two disks in a mirror, the very next reboot fails. Although I think this had already been merged into the latest release, I'm going to give it a try...https://forums.freenas.org/index.php?threads/workaround-semi-fix-for-mountroot-issues-with-9-3.26071/

#22 Updated by Jason Gould about 4 years ago

I should also mention that after the reboot fails, the BIOS doesn't even SEE the USB sticks anymore! Like they aren't even plugged into the computer. Crazy...

#23 Updated by Jason Gould about 4 years ago

2359

I've now been able to get it to boot consistently, however there are still real issues.

I did what was mentioned at https://forums.freenas.org/index.php?threads/workaround-semi-fix-for-mountroot-issues-with-9-3.26071/
At 50000 it boots into the web interface. However, every single time the system is rebooted, one of the sticks in the mirror shows as "unavailable". See the attached picture. I even took this stick out of the server, reformatted it, and replaced the unavailable disk. It re-silvered and status shows as healthy after this. Reboot and it goes unavailable again.

[root@freenas] ~# zpool status
  pool: freenas-boot
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        freenas-boot                                    DEGRADED     0     0     0
          mirror-0                                      DEGRADED     0     0     0
            3198367488699364907                         UNAVAIL      0     0     0  was /dev/da0p2
            gptid/3ec67152-c387-11e4-980d-0cc47a335ac4  ONLINE       0     0     0

errors: No known data errors

[root@freenas] ~# zpool status -v
  pool: freenas-boot
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Mar  6 08:53:34 2015
        543M scanned out of 942M at 5.54M/s, 0h1m to go
        542M resilvered, 57.65% done
config:

        NAME                                            STATE     READ WRITE CKSUM
        freenas-boot                                    DEGRADED     0     0     0
          mirror-0                                      DEGRADED     0     0     0
            replacing-0                                 UNAVAIL      0     0     0
              3198367488699364907                       UNAVAIL      0     0     0  was /dev/da0p2
              da1p2                                     ONLINE       0     0     0  (resilvering)
            gptid/3ec67152-c387-11e4-980d-0cc47a335ac4  ONLINE       0     0     0

errors: No known data errors

[root@freenas] ~# zpool status -v
  pool: freenas-boot
 state: ONLINE
  scan: resilvered 942M in 0h2m with 0 errors on Fri Mar  6 08:56:18 2015
config:

        NAME                                            STATE     READ WRITE CKSUM
        freenas-boot                                    ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            da1p2                                       ONLINE       0     0     0
            gptid/3ec67152-c387-11e4-980d-0cc47a335ac4  ONLINE       0     0     0

errors: No known data errors
[root@citranofreenas] ~#

Perhaps something wrong with that one stick? Perhaps manufactured differently?
Willing to provide any details to help for as long as I still have the system in testing.

#24 Updated by Jason Gould about 4 years ago

Still playing around with this. The one usb still goes unavailable after a reboot, no matter what I do. Take it offline, reformat, replace, and it works fine. Reboot again and same usb stick goes unavailable. Also noticed it doesn't get the same information for glabel...

[root@freenas] ~# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: resilvered 1.26G in 0h3m with 0 errors on Mon Mar  9 17:14:57 2015
config:

        NAME                                            STATE     READ WRITE CKSUM
        freenas-boot                                    ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/534769e2-c421-11e4-9f97-0cc47a335ac4  ONLINE       0     0     0
            da1p2                                       ONLINE       0     0     0

errors: No known data errors
[root@freenas] ~# glabel status
                                      Name  Status  Components
gptid/532d5211-c421-11e4-9f97-0cc47a335ac4     N/A  da0p1
gptid/534769e2-c421-11e4-9f97-0cc47a335ac4     N/A  da0p2
gptid/cdccd4f2-c6a0-11e4-a62a-0cc47a335ac4     N/A  da1p1
[root@freenas] ~#
[root@freenas] ~# gpart show
=>      34  60751805  da0  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  60750768    2  freebsd-zfs  (29G)
  60751832         7       - free -  (3.5k)

=>      34  60751805  da1  GPT  (29G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  60750768    2  freebsd-zfs  (29G)
  60751832         7       - free -  (3.5k)

I don't really know much about glabel, but it looks like the good stick has two glabels (da0p1, da0p2) that correlate to the two partitions on the stick. And if I am reading it correctly, the first partition is the bios-boot, second partition is the actual zfs (where I believe freenas is located). I find it interesting that the stick that doesn't work is not assigned a gptid for the 2nd partition (freebsd-zfs) like the first one. That is the partition with the glabel that appears to be used for identification in the zpool. Don't know if that means anything at all (probably not), but never know.

#25 Updated by David Y about 4 years ago

Still looks to be related to Bug#6857.

https://bugs.pcbsd.org/issues/6857

Very similar symptoms (device disappearing). Can one of the developers take a look or at least kick it over to Alexander Motin so he can chime in?

Thanks

#26 Updated by Jason Gould about 4 years ago

So, turns out just pulling out the "unavailable" usb stick and plugging it back in causes it to come online and the pool status to go from degraded to online. No need to reformat it, do a zpool replace, or anything.

Here is what it looks like after this;

[root@freenas] ~# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: resilvered 488K in 0h0m with 0 errors on Tue Mar 10 09:58:36 2015
config:

        NAME                                            STATE     READ WRITE CKSUM
        freenas-boot                                    ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/534769e2-c421-11e4-9f97-0cc47a335ac4  ONLINE       0     0     0
            gptid/cde70046-c6a0-11e4-a62a-0cc47a335ac4  ONLINE       0     0     0

errors: No known data errors

Here is the information related to bug #6857
Again, after pulling and plugging it back in;

[root@freenas] ~# geom disk list
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 31104958464 (29G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: SanDisk Ultra Fit
   lunname: SanDisk Ultra Fit       4C531001600111103501
   lunid: SanDisk Ultra Fit       4C531001600111103501
   ident: 4C531001600111103501
   fwsectors: 63
   fwheads: 255

Geom name: da1
Providers:
1. Name: da1
   Mediasize: 31104958464 (29G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: SanDisk Ultra Fit
   lunname: SanDisk Ultra Fit       4C530001210111104070
   lunid: SanDisk Ultra Fit       4C530001210111104070
   ident: 4C530001210111104070
   fwsectors: 63
   fwheads: 255
[root@freenas] ~# sg_vpd -p di da0
Device Identification VPD page:
  Addressed logical unit:
    designator type: T10 vendor identification,  code set: ASCII
      vendor id: SanDisk
      vendor specific: Ultra Fit
[root@freenas] ~# sg_vpd -p di -H da0
Device Identification VPD page:
 00     00 83 00 30 02 01 00 2c  53 61 6e 44 69 73 6b 00    ...0...,SanDisk.
 10     55 6c 74 72 61 20 46 69  74 00 00 00 00 00 00 00    Ultra Fit.......
 20     34 43 35 33 31 30 30 31  36 30 30 31 31 31 31 30    4C53100160011110
 30     33 35 30 31                                         3501
[root@freenas] ~# sg_vpd -p di da1
Device Identification VPD page:
  Addressed logical unit:
    designator type: T10 vendor identification,  code set: ASCII
      vendor id: SanDisk
      vendor specific: Ultra Fit
[root@freenas] ~# sg_vpd -p di -H da1
Device Identification VPD page:
 00     00 83 00 30 02 01 00 2c  53 61 6e 44 69 73 6b 00    ...0...,SanDisk.
 10     55 6c 74 72 61 20 46 69  74 00 00 00 00 00 00 00    Ultra Fit.......
 20     34 43 35 33 30 30 30 31  32 31 30 31 31 31 31 30    4C53000121011110
 30     34 30 37 30                                         4070

I should note that at no point in any of my testings has the USB_ERR_IOERROR during bootup stopped, even when I'm able to get freenas to boot it still shows these messages.

#27 Updated by Benjamin Gilles about 4 years ago

I just wanted to add that I am also experiencing this exact same issue with the Ultra Fit 32GB flash drives and on the same motherboard(X10SL7-F) as Jason above. Here is a link to where I had done some testing and documentation

https://forums.freenas.org/index.php?threads/new-build-seeking-opinions-and-a-few-clarifications.27870/page-2#post-184109

#28 Updated by David Y about 4 years ago

@Jason Gould I should note that at no point in any of my testings has the USB_ERR_IOERROR during bootup stopped, even when I'm able to get freenas to boot it still shows these messages.

From what I can gather, you don't get the USB_ERR_IOERROR might be because your testing with 2 mirrored USB devices and for whatever reason, one drive is still available to "boot through" to the OS. Like you mentioned, you do lose one of the drives and it doesn't return until you remove and plug the drive back in.

I'm only running a single USB drive so when the boot drive disappears, it's gone. There's no secondary drive to boot from. The drive does reappear when I unplug and plug it back in as well, but by then, I'm already dropped into the mountroot prompt and can't continue.

Either way, this needs to be looked at for those of us who don't use 2 USB drives.

#29 Updated by Jordan Hubbard about 4 years ago

We have looked at it, believe me. The problem is, it requires a very specific motherboard / USB stick manufacturer combination to even reproduce it. We have no idea why, or what the possible fix might be.

#30 Updated by Benjamin Gilles about 4 years ago

Jordan

If you want any more details from my system let me know. I just got these USB drives and am thinking about sending them back to Amazon in my return period but that still gets me about 20 days I can play with them, and I don't have any data on this system as of yet. Don't even have the hard drives powered up anymore as I was starting and stopping the system to do the above testing.

#31 Updated by Jordan Hubbard about 4 years ago

Well, we have a reproduction case locally, as Sean outlines above, we just can't see to agree whether it's a GRUB issue or a Kernel issue, and the two folks involved in both areas are currently pointing their fingers at one another. :)

#32 Updated by Jason Gould about 4 years ago

David Y wrote:

@Jason Gould I should note that at no point in any of my testings has the USB_ERR_IOERROR during bootup stopped, even when I'm able to get freenas to boot it still shows these messages.

From what I can gather, you don't get the USB_ERR_IOERROR might be because your testing with 2 mirrored USB devices and for whatever reason, one drive is still available to "boot through" to the OS. Like you mentioned, you do lose one of the drives and it doesn't return until you remove and plug the drive back in.

I'm only running a single USB drive so when the boot drive disappears, it's gone. There's no secondary drive to boot from. The drive does reappear when I unplug and plug it back in as well, but by then, I'm already dropped into the mountroot prompt and can't continue.

Either way, this needs to be looked at for those of us who don't use 2 USB drives.

I think you may have misunderstood me. The USB_ERR_IRERROR DOES NOT stop. Even if the system boots from the other USB drive, you still get those errors.

Jordan Hubbard wrote:

We have looked at it, believe me. The problem is, it requires a very specific motherboard / USB stick manufacturer combination to even reproduce it. We have no idea why, or what the possible fix might be.

Jordan, I know you say you guys have reproduced this and seem to believe you at least know somewhat where the issue is coming from. Could you clear that up for me at least a little? I assume it isn't anything that has been mentioned to this point in other tickets? The issue I'm seeing doesn't seem to match ticket #6857 (doesn't look like they violate SCSI standards). After more testing it doesn't appear to be related to "kFreeBSD.kern.cam.boot_delay".

I've been adding my findings because;
  1. I have a pretty decent variety of machines to test on, both consumer and enterprise. Z87 with i5, Z68 with i5, older E8500 machines, new 1150/Xeon E3, 2011/E5.
  2. I'm running mirrored while seems most have been testing single sticks
  3. I seem to actually have gotten further than most since I'm actually able to get FreeNAS to boot, proving that at least it does boot from these sticks
  4. It helps me keep track of things I've run and outputs, lol

It doesn't make sense to me why if I have 2 of these in a mirror it magically boots (albeit in a degraded state, but still). While booting from the one "good" stick doesn't work at all (literally doesn't boot if you remove the bad stick from the machine). I find that incredibly strange. Almost as if, even though the 2nd bad stick isn't identified during boot (doesn't even show in camcontrol devlist) that it still is somehow required to get the system to boot.

I appreciate the team at least looking at this and trying to get other teams to fix the issue.
I'm left with thinking it is something related to GRUB, but that is my personal feeling. Doesn't TrueNAS use something other than grub?

Is there a set of affordable USB sticks that I can get that are KNOWN WORKING?

#33 Updated by Jordan Hubbard about 4 years ago

Jason: We don't know where the issue is coming from. We don't know why some sticks exhibit this behavior and some don't. All we have are some extremely vague theories which are not, unfortunately, backed up by any particular piece of code or hardware configuration or we'd already have debugging shims in place and/or specific reproduction steps for people to follow in debugging this. As you yourself point out, the problem manifests itself even more mysteriously on your system. Perhaps a specific BIOS version or manufacturer is to blame? Perhaps a certain brand of USB controller chip, on-board certain motherboards? We don't know. What we do know is that all the machines we ship work just fine, so we know where it does NOT occur, but that does not help us much.

#34 Updated by 4M8 l93 about 4 years ago

I would like you to point at my finding from #7414: From time to time I am stuck at mountroot with my (single) thumb drive being powered off seconds after the kernel has loaded! I can tell that it's powered off because it has an LED that is on when it's powered on and blinks when it's accessed. Can you tell if both thumb drives are powered on when you run into the issue?
As the thumb drive is powered off after the kernel has loaded I would rule out GRUB as a cause on my machine.

#35 Updated by Jordan Hubbard about 4 years ago

  • Category set to 137
  • Assignee changed from Kris Moore to Xin Li

#36 Updated by Jason Gould about 4 years ago

I noticed today that when I perform the process of unplugging the "unavailable" USB stick and plug it back in, that it shows up in Storage -> View Disks. It looks like it even tries to use SMART on it, like it is a regular disk. Which doesn't seem right. The USB stick that does work correctly does not show up there, only in the "boot" section. And as I mentioned before, it is not being assigned a gptid label for the freebsd-zfs partition.

I was reading this thread;
https://forums.freebsd.org/threads/labelling-existing-partitions-with-glabel.41438/
Since maybe it has something to do with the glabel. Might see if it helps at all.

When you reboot the system, the stick disappears from camcontrol, glabel, gpart show, and so on. It is as if the device isn't even there. However the USB stick is powered on (as noted by the red light on the stick).

I think this USB stick is originally always the "FIRST" stick it attempts to boot from as well.

#37 Updated by Kay Bee Null about 4 years ago

Any updates here? I sent a rather generic complaint about not being able to boot from a SanDisk 16GB Ultra Fit to ASRock (my board is an E3C226D2I) and their response asked for more information but also gave a link to this page and asked if my problem might be related to this bug report. Wow! Guess I am not the only frustrated ASRock owner sitting around waiting for this bug to be fixed... I see the problem has been assigned to a FreeBSD category. Just as an added piece of information, I stuck a SanDisk 16GB Ultra Fit into my pfSense box (SuperMicro A1SRi-2758F) and got a very similar failure leading to the mountroot prompt.

#38 Updated by Jordan Hubbard about 4 years ago

Unfortunately, we are no closer to figuring this out today than we were 2 months ago.

#39 Updated by David Y about 4 years ago

Kay Bee Null wrote:

Any updates here? I sent a rather generic complaint about not being able to boot from a SanDisk 16GB Ultra Fit to ASRock (my board is an E3C226D2I) and their response asked for more information but also gave a link to this page and asked if my problem might be related to this bug report. Wow! Guess I am not the only frustrated ASRock owner sitting around waiting for this bug to be fixed... I see the problem has been assigned to a FreeBSD category. Just as an added piece of information, I stuck a SanDisk 16GB Ultra Fit into my pfSense box (SuperMicro A1SRi-2758F) and got a very similar failure leading to the mountroot prompt.

I reported the original bug and have been trying to suggest it's related to Bug#6857, but don't think anyone is listening.

https://bugs.pcbsd.org/issues/6857

The fact it happens on pfsense tells me it's even more likely that the issue with handling non-printable T10 identifiers on FreeBSD is not fixed.

#40 Updated by Kay Bee Null about 4 years ago

The T10 identifier bug seems to have been addressed in FreeBSD 10 (https://lists.freebsd.org/pipermail/svn-src-stable-10/2014-December/003972.html) and it will probably make it into pfSense in their latest update to 2.2.1. Does anyone know if this issue has been addressed in FreeBSD 9.3??

Edit: The issue has been fixed in the FreeBSD 9 branch (https://lists.freebsd.org/pipermail/svn-src-stable-9/2014-December/007373.html). Has this fix been incorporated into FreeNAS 9.3??

#41 Updated by Kay Bee Null about 4 years ago

Just took a look at the code in the FreeNAS 9.3 Standard Jail template (http://download.freenas.org/jails/9.3/x64/freenas-standard-9.3-RELEASE.tgz) and it does not include the svn commit: r275604. No surprise I guess. The template is stamped at a time before the fix was committed. How difficult is it to get this change into the next SU?

#42 Updated by Kay Bee Null about 4 years ago

Where is the source for the base-os-9.3-STABLE that is packaged with FreeNAS located?

#44 Updated by Xin Li about 4 years ago

  • Related to Bug #6857: GEOM_MULTIPATH does strange things with (formerly?) GPT devices added

#45 Updated by Xin Li about 4 years ago

  • Related to Bug #6842: Installer doesn't check mirrored boot devices for GPT added

#46 Updated by Xin Li about 4 years ago

  • Status changed from Investigation to Closed
  • Assignee changed from Xin Li to Alexander Motin

Resolved by a change from Sasha.

#47 Updated by Kay Bee Null about 4 years ago

Hate to open a closed ticket but I don't quite understand the resolution here. It looks like the change that resolves the T10-ID bug was introduced by Alexander Motin into the trueos-9.3-STABLE branch in December of last year. However, I am still seeing this problem now and am running the most current software. How is this issue resolved?

#48 Updated by Jordan Hubbard about 4 years ago

  • Status changed from Closed to Screened

I think closing this may have been a bit premature (or at least wishful thinking). So the fix was applied, but did not fix this problem. So there is a different problem. Unfortunately, we are now no closer to a resolution, however, since we don't have a clue as to what is going on or why this fails with such a limited range of devices.

Where did you obtain this exact device? Do you have a serial # or any other information that might help us in obtaining the precise make/model of device that causes these errors?

#49 Updated by Alexander Motin about 4 years ago

I suppose this stick has more then one bug. Earlier I fixed bug with LUN ID reported by this device. According to data in this ticket it really fixed situation when more then one stick of such model used in FreeNAS. But it seems like this stick also has some compatibility problems with some USB controllers (some AMD one in this case). I have neither this controller nor this stick, and I am not expert in USB transport (and this new problem seems has nothing to do with generic SCSI layer). So I doubt that I/we can do anything reasonable about this case. It is cheaper to buy different stick, that may live better with this USB controller.

#50 Updated by Kay Bee Null almost 4 years ago

I have successfully installed FreeNAS-9.3-current onto 3 separate 16GB SanDisk Ultra Fit flash drives and all of them fail to boot. I see now that I am not getting USB_ERR_IOERROR errors. The boot output looks clean until it fails to mount the ROOT drive with error 2 and drops down to the mountroot prompt. I have finally decided to shoot my inner OCD and give up on these sticks.

#51 Updated by Aaron Oneal almost 4 years ago

This is still happening with newer Sandisk drives as well. I just bought the one below and hit the same error 2 mountroot prompt after install and reboot. Unfortunately, I didn't know about this bug before purchasing. :-(

SanDisk Ultra Fit™ CZ43 32GB USB 3.0 Low-Profile Flash Drive Up To 130MB/s Read- SDCZ43-032G-G46
http://www.amazon.com/gp/product/B00LLER2CS/

Any good place to put a warning about not buying this brand until the bug is fixed?

#52 Updated by Jochem Kuijpers over 3 years ago

What's the status on this? After a few hours of troubleshooting I finally reached this page. So I'll have to buy another USB or use one of my older USB2.0 ones.

It seems this bug is more relevant than 'Nice to have'. I'd like to see it at least on Important, especially since these USBs are quite popular. Fixing this will probably also prevent future products from failing to boot if they happen to trigger the same issue.

Can someone please invest some effort into solving this?

#53 Updated by Jordan Hubbard over 3 years ago

Patches happily accepted. This is marked "nice to have" because we don't have an obvious fix for this or anyone super-knowledgable with FreeBSD's USB stack on tap to figure this out.

#54 Updated by Miles Ingram over 3 years ago

  • Priority changed from Nice to have to Expected
  • Target version changed from Unspecified to 261

Hey guys. I was running 9.2.1.9 earlier today and decided to upgrade to 9.3. I made a huge mistake. I was running the 9.2.1.9 on the 32GB flash drive you guys are referring to. It worked fine. I assumed upgrading would be fine. Well I am getting all the same errors as above. I really don't know anything about Freenas, I just followed a tutorial to set it up. I had all my stuff set up on Plex with 6 3TB drives running a RaidZ2. PLEASE tell me all my data isn't gone. Am I going to have to reinstall Freenas now that the flash drive won't boot? Is all my plex data gone? Can someone tell me how to proceed? BTW I am running a small H871-plus motherboard if that helps any.

#55 Updated by Miles Ingram over 3 years ago

  • Target version changed from 261 to Unspecified

Sorry didn't see that it changed the priority for everyone didn't know I had the permission. I was filling it out for me.

#56 Updated by Alexander Motin over 3 years ago

Version reverts within 9.3 branch are as trivial as reverting ZFS snapshot. Revert to 9.2 on the other side is more difficult. You'll indeed have to reinstall it. If you backed up your FreeNAS configuration before the upgrade, then you can easily restore everything as it was before. If you haven't, then it will be more difficult. You may try to apply config from 9.3 to 9.2, but I am not sure it will work, since 9.2 has no idea about 9.3 configs. In any case your data should not be lost if you do everything right, but you may need to reconfigure FreeNAS from scratch to access them.

#57 Updated by Miles Ingram over 3 years ago

Thanks Alex. I was able to put 9.3 onto an older 8 gig flash drive I had. I would rather use a newer version of Freenas on an older drive than a drive that isn't supported as of yet. I will perhaps move to the 32gb low profile flash drive once Freenas supports it, as it looks better than the ugly thing sticking out of the side of the pc right now. 9.3 recognized my old volume instantly so that was great. I had to reinstall and configure plex and cifs again but it wasn't too bad, overall a hour or so of work overall. I will definitely see if I can backup my metadata from Plex in the future. Not only does it take forever to download all the metadata but I have to manually put in what I have already watched now which will take an immense about of time as there are a lot of movies and t.v. shows. Thank you for your help. Looking forward to the next version of Freenas supporting this flash drive as it is a popular drive (I think it was the best seller in flash drives on Amazon when I bought it).

#58 Updated by david stievenard over 3 years ago

hi

that's my first post here, as I face the same problem, let me just give the informations I have :

freenas version : curent at this date

same problem with
- a couple of sandisk cruzer fit 32gb usb3.0 (cz43) [on usb2.0 ports] on a supermicro A1SRi-2758F
- I noticed that sometimes the usb keys are not recognized in the bios also...I still can not figure out why and when they appear or not

no problem with
- a couple of sandisk cruzer fit 32g usb2.0 (cz33) on a asrockrack C2750D4i
- a sandisk 16g (cz52) on a supermicro X9DRi

#59 Updated by Ryan Rust over 3 years ago

I'm seeing this exact same issue with a SanDisk Ultra Fit CZ43 (USB 3.0) plugged into StarTech 2 Port USB Motherboard Header Adapter (USBMBADAPT2) on a ASROCK Q1900-ITX J1900 R.

Interestingly, I've been using this hardware configuration since May 2015 and I didn't start seeing it until applying some of the updates in September 2015. Another interesting bit: The first time it occurred, I avoided updating until the beginning of October. When I applied the updates that had queued up during this time, the issue did not occur. However, while applying new updates in late October, it started happening again. It is also occurring when I attempt to apply the latest updates for the 9.3-STABLE train. So, I'm stuck at FreeNAS-9.3-STABLE-201509282017 for the time being.

Sorry about the date ranges, I wasn't paying attention to the specific updates that were queued up/applied.

I'm able to resolve this when it occurs by manually selecting the older (working) boot environment in GRUB.

#60 Updated by Vyshakh P over 3 years ago

This is my first post here,
I am also facing the same issue. I had bought a new Scandisk Ultra Usb3 32Gig Drive, and on which i am getting the same issue as USB_ERR_IOERROR, and after that usb wont be detected by the bios. I am using the usb3 on usb 2 port. I was having another 16Gig usb2 Scandisk blade, which works fine. Is there a way to come out of this issue, if there is no fix for this identified , I can return the usb3 and will go for usb2 drive.

#61 Updated by Jochem Kuijpers over 3 years ago

Vyshakh P wrote:

Is there a way to come out of this issue, if there is no fix for this identified , I can return the usb3 and will go for usb2 drive.

As far as we know, there is not. It seems to be a problem specific with this SanDisk product. It's not a problem with some other USB 3.0 drives, so you could get another USB 3.0 drive if you want. My Kingston DataTraveler 100 G3 (16GB USB3.0) works with the latest FreeNas.

I still think this should be fixed, whether it is FreeNas' fault or SanDisks. You're definitely not the first and probably not the last person to arrive at this bugreport.

#62 Updated by Alexander Motin about 3 years ago

  • Status changed from Screened to Closed: Not To Be Fixed

There is no plans to significantly touch USB stack in FreeNAS 9.3. New version will come with FreeNAS 10.

#63 Updated by Aleksandar Krsteski almost 3 years ago

I also have SanDisk Ultra Fit 32GB USB drive and had the same issue on HP MicroServer Gen8 with FreeNAS 9.10. I was able to install to SanDisk Ultra Fit 32GB, but after that FreeNAS never booted from it.

The solution for me was to load xhci.ko USB 3.0 driver module after ispfw.ko by editing GRUB boot command. To achieve this you can press 'e' key on GRUB prompt to edit the boot, and add following line after ispfw.ko line:

kfreebsd_module_elf /ROOT/default/@/boot/kernel/xhci.ko

After that you can press Ctrl-X to boot FreeNAS. There is some related info on following link on how to enable USB 3.0 support: https://blog.voltagex.org/2016/01/08/freenas-9-3-on-the-supermicro-5028a-tn4-a1sri-2758f/. Please note that I had to add the xhci.ko module after ispfw.ko since it didn't work if xhci.ko was added before ispfw.ko.

To make the change permanent and avoid manually editing GRUB boot commands on every boot you can add following two Tunnables (System->Tunnables) through FreeNAS web interface once you are booted:

ispfw_load="NO" 
xhci_load="YES" 

I do not use any Qlogic controllers so disabling ispfw module works for me. If you need this module, you will need to find a way to make xhci.ko load in GRUB after ispfw.ko.

Hope this info helps in fixing the issue for FreeNAS 10.

#64 Updated by Sean Fagan almost 3 years ago

FN10 has xhci compiled into the kernel.

#65 Updated by Aleksandar Krsteski almost 3 years ago

I was under impression that there won't be kernel changes between 9.10 and 10. But it is good to hear that xhci will get officially enabled in FN10.

In that case this issue might get fixed, which is great and we might get out of the box working install with previously problematic USB 3.0 drives.

#66 Updated by Alexander Motin almost 3 years ago

Aleksandar Krsteski wrote:

I was under impression that there won't be kernel changes between 9.10 and 10.

9.10 and 10 actually have exactly the same kernel, but this thread started from 9.3.

#67 Updated by Aleksandar Krsteski almost 3 years ago

I know that the issue originated from 9.3, but it was still not fixed in 9.10 although we have a workaround for 9.10 as I explained in my post.

Anyway, we still need to make sure xhci.ko is loaded for some USB 3.0 drives to work even on latest kernel (9.10 or 10). By default, this is not the case in 9.10, but it is nice to know it will be handled properly for 10.

#68 Updated by Sean Fagan almost 3 years ago

sigh

You can't load xhci.ko on an FN9.10 system, because it's already built into the kernel.

#69 Updated by Aleksandar Krsteski almost 3 years ago

Hmmm... now I am puzzled why issue is fixed for me with following changes:
- Adding following line directly into GRUB

kfreebsd_module_elf /ROOT/default/@/boot/kernel/xhci.ko

- Or adding following Tunable through interface once we are booted for first time:
xhci_load="YES" 

Also note that issue might be related with disabling/enabling of ispfw.ko, as I explained in my first post.

#70 Updated by Aleksandar Krsteski almost 3 years ago

I have done more tests on my HP MicroServer Gen8 with FreeNAS 9.10 on SanDisk Ultra Fit 32GB drive, testing all combinations of ispfw.ko and xhci.ko loading through GRUB. I got following result:

Module Load & Order | Works
--------------------|------
NONE                | Yes
xhci.ko             | Yes
ispfw.ko, xhci.ko   | Yes
ispfw.ko            | No    (this is default with FreeNAS 9.10)
xhci.ko, ispfw.ko   | No

I have verified above results multiple times and the behavior is always consistent with the table. I don't know what to conclude from this, but it is obvious that ispfw.ko module plays a role in the issue.

Also I have verified that xhci.ko is compiled into kernel since I am getting following log in dmesg if I add xhci.ko through GRUB:

module_register: cannot register pci/xhci from kernel; already loaded from xhci.ko
Module pci/xhci failed to register: 17

But it seems that xhci.ko takes precedence since log says already loaded from xhci.ko.

For the time being I will go with first option by just disabling ispfw.ko. If you need some more testing or debugging information do not hesitate to ask me, I will do what I can to help.

#71 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Target version changed from Unspecified to N/A

#72 Updated by Simon Lock over 1 year ago

  • Hardware Configuration updated (diff)

Has this error again resurfaced with FN 11-U4 regarding Sandisk Ultra Fit USB? One of my mirrored boot usb drives (2x64 Gb Sandisk Ultra Fit) failed shortly after installing 11-U4 although I can now confirm that this usb drive is actually totally healthy. I've now physically replaced the failed usb drive with a totally new one and all now appears to be good. However, the same inexplicable failure happened a few months ago with an earlier install of FN 11.

#73 Updated by Tyler Vincent over 1 year ago

Simon Lock wrote:

Has this error again resurfaced with FN 11-U4 regarding Sandisk Ultra Fit USB? One of my mirrored boot usb drives (2x64 Gb Sandisk Ultra Fit) failed shortly after installing 11-U4 although I can now confirm that this usb drive is actually totally healthy. I've now physically replaced the failed usb drive with a totally new one and all now appears to be good. However, the same inexplicable failure happened a few months ago with an earlier install of FN 11.

I'm also seeing this on 11-U4. I have a brand new Sandisk Ultra Fit and cannot get my install to mirror it's boot to it.

#74 Updated by Dru Lavigne over 1 year ago

Tyler: this ticket was closed over a year ago. Please create a new one and attach a debug to it so that a dev can review it.

Also available in: Atom PDF