Project

General

Profile

Bug #27787

USB disk fails to identify properly on USB3 controller

Added by Matt Kessler over 1 year ago. Updated over 1 year ago.

Status:
Closed
Priority:
No priority
Assignee:
Alexander Motin
Category:
Middleware
Target version:
Seen in:
Severity:
Reason for Closing:
Cannot Reproduce
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Build FreeNAS-11.1-RELEASE
Platform Intel(R) Celeron(R) CPU G1610T @ 2.30GHz
Memory 12215MB
Manufacturer: HP
Product Name: ProLiant MicroServer Gen8

ChangeLog Required:
No

Description

I revisited an old issue I had: Importing +1TB from a NTFS formatted USB3 drive.
(I'm doing a lot of off site backups and do not have the internet bandwith,
so I'm using 3 of those: WD Elements USB3, 2.5 HDD, 1.5TB, 1 NTFS partition)

In freenas 9.10 rsync was being used for the import and it failed in times(/var/run/importcopy/vol_rsync_stats).

In 11.1 I did not find any log with traces of what's going on.

Starting from GUI, I'm getting:

Import Disk
Import of Volume /dev/da1s1 Failed.

Reason: Mount failed 12 -> b'', b"NTFS signature is missing.\nFailed to mount '/dev/da1s1': Invalid argument\nThe device '/dev/da1s1' doesn't seem to have a valid NTFS.\nMaybe the wrong device
is used? Or the whole disk instead of a\npartition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?\n" 

My disk:
root@freenas:~ # camcontrol devlist
<ST3000VN000-1HJ166 SC60>          at scbus0 target 0 lun 0 (ada0,pass0)
<SanDisk Ultra Fit 1.00>           at scbus7 target 0 lun 0 (da0,pass1)
<WD Elements 10B8 1007>            at scbus8 target 0 lun 0 (pass2,da1)

root@freenas:~ # gpart list da1
Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 2930210815
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da1s1
   Mediasize: 1500266889216 (1.4T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawtype: 7
   length: 1500266889216
   offset: 1048576
   type: ntfs
   index: 1
   end: 2930210815
   start: 2048
Consumers:
1. Name: da1
   Mediasize: 1500267937792 (1.4T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

This is a testing box, but same hardware is being used in 3 locations.

Where should I start looking now?
Is this the same code as in 9.10 at all?

Cheers

GUI-failedImportText_01.png (9.78 KB) GUI-failedImportText_01.png Matt Kessler, 01/13/2018 02:40 PM
importOK-butFilesMissing.png (24.9 KB) importOK-butFilesMissing.png 15.000 files missing when browsing the network share from a windows box Matt Kessler, 01/17/2018 09:22 AM
13895
13939

History

#1 Updated by Dru Lavigne over 1 year ago

  • Status changed from Unscreened to 15

Matt: please attach an 11.1 debug (System -> Advanced -> Save Debug) created after a failed import.

#2 Updated by Matt Kessler over 1 year ago

I'd really like to do some more digging myself, as I have/had +40 devices and had issues in freenas 9, 9.10 and 11 but never got anywhere with this.
(well others too… e.g. #7307, #11535, #11028, but the tool does not seem to be a love child: https://redmine.ixsystems.com/issues/7307#note-2)

These things would help me very much:
  • What method is being used to read the partition information and is there a log, please?
    (I think sometimes it seems to try reading data from the extended partition)
  • Where did this logfile (/var/run/importcopy/vol_rsync_stats) go, please?
  • Is there a known issue with foreign char sets and are there any recommendations?
    From an attempt, which got to ~15% I can see some problems:
     root@freenas:/# ls '/var/run/importcopy/tmpdir/dev/da3s1/Music/Music A-Z/abc' | grep '?'
     Amon D??l II
     Blue ?yster Cult
     

    And in Samba stuff like this
     Conversion error: illegal character: Sissel Kyrkjebø
     Conversion error: illegal character: Pippi Långstrump
     Conversion error: illegal character: DÖF
     Conversion error: illegal character: Tommy Körberg & Sissel Kyrkjebö
     Conversion error: illegal character: Die Ärzte
     Conversion error: illegal character: Peter Jöbeck
     Conversion error: illegal character: Queensrÿche
     Conversion error: illegal character: Julsånger
     

@Dru
As a Sysadmin I always preach, to not submit any logs etc. ever, for security reasons.
In Europe we are now dealing with a new General Data Protection Regulation (GDPR),
which I would violate if this was a one of my other boxes.
Maybe you folks at IX could discuss this topic in-house and draw some conclusions for future freenas releases.
And I'll have a look, whats being send.
Thanks

#3 Updated by Dru Lavigne over 1 year ago

  • Private changed from No to Yes

Note that tickets are hidden while a debug is attached and once the dev has the info needed, the debugs are deleted.

We have an open ticket now on dealing with the GDPR.

#4 Updated by Matt Kessler over 1 year ago

13939

I'm attaching the debug dump.

Also, I tested importing from a FAT32 500GB, which somewhat works,
but there are ~15.000 files missing when browsing the network share from a windows box!
(see above note on foreign char sets issues)

I wish this tool was more of a file explorer typ of thing.
Just imagine end users trying to import 1.5TB family pics and videos from a 4TB USB drive…
They got a single Ethernet-port from the ISP, which they use for the freenas box and then
start coping 1.5TB over WIFI — not such a pleasant experience.
Instead they will try importing 4TB to a hopefully large enough pool and delete 2.5TB after the import — if all went well…

#5 Updated by Dru Lavigne over 1 year ago

  • Status changed from 15 to Unscreened
  • Assignee changed from Release Council to Vladimir Vinogradenko
  • Target version set to 11.1-U2

#6 Updated by Vladimir Vinogradenko over 1 year ago

  • Status changed from Unscreened to 15

Matt Kessler, the problem you're having with your NTFS drive is shown in GUI:

NTFS signature is missing.
Failed to mount '/dev/da1s1': Invalid argument
The device '/dev/da1s1' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?

The command used to mount drive is
kldload fuse
/usr/local/bin/ntfs-3g -o ro /dev/da1s1 /var/run/importcopy/tmpdir/dev/da1s1

From what I know, this message only appears if file system is really broken. If you want to investigate this yourself, you're welcome :)

#7 Updated by Vladimir Vinogradenko over 1 year ago

I also see 455.19G import to /mnt/Sea3/share/Music500 that has failed (rsync failed with exit code 23 which means "some files could not be transferred"). Did UI show you that this import was successful?

Unfortunately rsync output is lost in that case which will be fixed. But now, can you please run

/usr/local/bin/rsync --info=progress2 --modify-window=1 -rltvh --no-perms /var/run/importcopy/tmpdir/dev/da3s1 /mnt/Sea3/share/Music500

(that is the exact command) to see what exactly was wrong?

#8 Updated by Vladimir Vinogradenko over 1 year ago

Regarding Conversion error: illegal character: I suppose we can introduce iocharset option for importing FAT volumes.

Thank you for your logs, Matt, they have been a great help in improving this feature!

#9 Updated by Matt Kessler over 1 year ago

Vladimir Vinogradenko wrote:

From what I know, this message only appears if file system is really broken. If you want to investigate this yourself, you're welcome :)

Thanks Vladimir
Broken in what sense? Bad partition layout, broken mbr…?
The above WD Elements 10B8 1007 is hooked up to a win 2008 server for nightly CRM dumps.
In the morning, I then walk to 3 locations down the street, plug it into 3 different Ubuntu 16.04 desktop computers and copy the backup(over the local network) to a freenas box.
I do this in all 3 building and none of these systems ever have a problem with the drive.
We got 3 of those WD USB3 about 2 yrs ago because the desktop computers in those buildings were only USB2 and 100Mbit,
I had the issues on and off back then already, when using the import tool and eventually gave up.

#10 Updated by Matt Kessler over 1 year ago

ah, forgot

here I tried two USB3 thumb drives, both fine on my daily driver, a KDEneon ThinkPad — sorry no more PCbsd…


#+1515798740
kldload fuse
#+1515798774
ntfs-3g /dev/da1s1 /mnt/stick01
#+1515798797
ntfs-3g /dev/da2s1 /mnt/stick01

these were both fine.

A third test failed with a thumb drive I'm still using here and there.
This looks wild… do you see a particular reason why this would fail?

Geom name: da1s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 7862271
first: 0
entries: 124797
scheme: EBR
Providers:
1. Name: da1s5
   Mediasize: 1939865600 (1.8G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2097152
   Mode: r0w0e0
   rawtype: 7
   length: 1939865600
   offset: 1048576
   type: ntfs
   index: 1
   end: 3790847
   start: 0
2. Name: da1s6
   Mediasize: 2083520512 (1.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1943011328
   Mode: r0w0e0
   rawtype: 7
   length: 2083520512
   offset: 1941962752
   type: ntfs
   index: 60173
   end: 7862271
   start: 3790848
Consumers:
1. Name: da1s1
   Mediasize: 4025483264 (3.7G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r0w0e0

Geom name: iso9660/neon%20user%20lts%2020171213-00:46
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 7864319
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: iso9660/neon%20user%20lts%2020171213-00:46s1
   Mediasize: 4025483264 (3.7G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r0w0e0
   rawtype: 5
   length: 4025483264
   offset: 1048576
   type: ebr
   index: 1
   end: 7864319
   start: 2048
Consumers:
1. Name: iso9660/neon%20user%20lts%2020171213-00:46
   Mediasize: 4026531840 (3.8G)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: iso9660/neon%20user%20lts%2020171213-00:46s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 7862271
first: 0
entries: 124797
scheme: EBR
Providers:
1. Name: iso9660/neon%20user%20lts%2020171213-00:46s5
   Mediasize: 1939865600 (1.8G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2097152
   Mode: r0w0e0
   rawtype: 7
   length: 1939865600
   offset: 1048576
   type: ntfs
   index: 1
   end: 3790847
   start: 0
2. Name: iso9660/neon%20user%20lts%2020171213-00:46s6
   Mediasize: 2083520512 (1.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1943011328
   Mode: r0w0e0
   rawtype: 7
   length: 2083520512
   offset: 1941962752
   type: ntfs
   index: 60173
   end: 7862271
   start: 3790848
Consumers:
1. Name: iso9660/neon%20user%20lts%2020171213-00:46s1
   Mediasize: 4025483264 (3.7G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r0w0e0


This last results my be from a different freenas box, but identical hardware is in use.
As I said I'm revisiting and would like to figure this, but in my notes I sometimes forget to write down the name of the box.
If I ever give up, I'm gonna send you all my drives ;-)

#11 Updated by Matt Kessler over 1 year ago

Vladimir Vinogradenko wrote:

I also see 455.19G import to /mnt/Sea3/share/Music500 that has failed (rsync failed with exit code 23 which means "some files could not be transferred"). Did UI show you that this import was successful?

Unfortunately rsync output is lost in that case which will be fixed. But now, can you please run
[...]
(that is the exact command) to see what exactly was wrong?

Sorry, I let this run over night, so not sure about the UI.
(It was the 2nd test from another 500GB drive and I only watched the beginning)
The drive has been removed by now.

So the logfile (/var/run/importcopy/vol_rsync_stats) should have been there and this will be fixed!?

#12 Updated by Matt Kessler over 1 year ago

Gotcha…

Just run one more test with the drive I attached 2 days ago
Here is what caused rsync to struggle:

/usr/local/bin/rsync --info=progress2 --modify-window=1 -rltvh --no-perms /var/run/importcopy/tmpdir/dev/da2s1/ /mnt/Sea3/ximport/usb2_1
…
sent 26.55G bytes  received 17.14K bytes  26.54M bytes/sec
total size is 30.94G  speedup is 1.17
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1180) [sender=3.1.2]

oh well, this file could have a better name:
Thunderbird and S_MIME certificates_ fixing “Unable to sign message.” error _ johnny’s jotter.mht

Now behold, behold
This transfer was from dev/da2 (USB2 port) -- attached 2 days ago

Jan 15 22:10:55 freenas ugen0.6: <Terra USThree> at usbus0
Jan 15 22:10:55 freenas umass2 on uhub4
Jan 15 22:10:55 freenas umass2: <Terra USThree, class 0/0, rev 2.10/1.00, addr 6> on usbus0
Jan 15 22:10:55 freenas umass2:  SCSI over Bulk-Only; quirks = 0xc100
Jan 15 22:10:55 freenas umass2:9:2: Attached to scbus9
Jan 15 22:10:55 freenas da2 at umass-sim2 bus 2 scbus9 target 0 lun 0
Jan 15 22:10:55 freenas da2: <Terra USThree PMAP> Removable Direct Access SPC-4 SCSI device
Jan 15 22:10:55 freenas da2: Serial Number 07083B88B71C4C20
Jan 15 22:10:55 freenas da2: 40.000MB/s transfers
Jan 15 22:10:55 freenas da2: 29573MB (60566016 512 byte sectors)
Jan 15 22:10:55 freenas da2: quirks=0x2<NO_6_BYTE>

Here it is being removed today
Jan 17 22:47:42 freenas ugen0.6: <Terra USThree> at usbus0 (disconnected)
Jan 17 22:47:42 freenas umass2: at uhub4, port 1, addr 6 (disconnected)
Jan 17 22:47:42 freenas da2 at umass-sim2 bus 2 scbus9 target 0 lun 0
Jan 17 22:47:42 freenas da2: <Terra USThree PMAP> s/n 07083B88B71C4C20 detached
Jan 17 22:47:42 freenas (da2:umass-sim2:2:0:0): Periph destroyed
Jan 17 22:47:42 freenas umass2: detached

And here the very same thumb drive is inserted at a USB3 port
Jan 17 22:48:04 freenas ugen1.2: <Phison 2307 Boot ROM> at usbus1
Jan 17 22:48:04 freenas umass1 on uhub2
Jan 17 22:48:04 freenas umass1: <Phison 2307 Boot ROM, class 0/0, rev 3.00/1.10, addr 1> on usbus1
Jan 17 22:48:04 freenas umass1:  SCSI over Bulk-Only; quirks = 0xc100
Jan 17 22:48:04 freenas umass1:8:1: Attached to scbus8
Jan 17 22:48:04 freenas da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
Jan 17 22:48:04 freenas da1: < 2307 PRAM 1.00> Removable Direct Access SCSI device
Jan 17 22:48:04 freenas da1: 400.000MB/s transfers
Jan 17 22:48:04 freenas da1: Attempt to query device size failed: NOT READY, Medium not present
Jan 17 22:48:04 freenas da1: quirks=0x2<NO_6_BYTE>

And just now — repeat — and it's all good

Jan 17 23:47:14 freenas ugen0.5: <Terra USThree> at usbus0
Jan 17 23:47:14 freenas umass1 on uhub4
Jan 17 23:47:14 freenas umass1: <Terra USThree, class 0/0, rev 2.10/1.00, addr 5> on usbus0
Jan 17 23:47:14 freenas umass1:  SCSI over Bulk-Only; quirks = 0xc100
Jan 17 23:47:14 freenas umass1:8:1: Attached to scbus8
Jan 17 23:47:14 freenas da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
Jan 17 23:47:14 freenas da1: <Terra USThree PMAP> Removable Direct Access SPC-4 SCSI device
Jan 17 23:47:14 freenas da1: Serial Number 07083B88B71C4C20
Jan 17 23:47:14 freenas da1: 40.000MB/s transfers
Jan 17 23:47:14 freenas da1: 29573MB (60566016 512 byte sectors)
Jan 17 23:47:14 freenas da1: quirks=0x2<NO_6_BYTE>
Jan 17 23:48:14 freenas daemon[2985]:     2018/01/17 23:48:14 [WARN] agent: Check 'service:nas-health' is now warning
Jan 17 23:48:28 freenas ugen0.5: <Terra USThree> at usbus0 (disconnected)
Jan 17 23:48:28 freenas umass1: at uhub4, port 2, addr 5 (disconnected)
Jan 17 23:48:28 freenas da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
Jan 17 23:48:28 freenas da1: <Terra USThree PMAP> s/n 07083B88B71C4C20 detached
Jan 17 23:48:28 freenas (da1:umass-sim1:1:0:0): Periph destroyed
Jan 17 23:48:28 freenas umass1: detached

Jan 17 23:48:40 freenas ugen1.2: <Terra USThree> at usbus1
Jan 17 23:48:40 freenas umass1 on uhub2
Jan 17 23:48:40 freenas umass1: <Terra USThree, class 0/0, rev 3.00/1.00, addr 1> on usbus1
Jan 17 23:48:40 freenas umass1:  SCSI over Bulk-Only; quirks = 0xc100
Jan 17 23:48:40 freenas umass1:8:1: Attached to scbus8
Jan 17 23:48:40 freenas da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
Jan 17 23:48:40 freenas da1: <Terra USThree PMAP> Removable Direct Access SPC-4 SCSI device
Jan 17 23:48:40 freenas da1: Serial Number 07083B88B71C4C20
Jan 17 23:48:40 freenas da1: 400.000MB/s transfers
Jan 17 23:48:40 freenas da1: 29573MB (60566016 512 byte sectors)
Jan 17 23:48:40 freenas da1: quirks=0x2<NO_6_BYTE>

Now this is is a FAT32, so for me we can remove the ntfs from the Subject.
It has always been my hunch, this has to do with USB3.
This would also explain why it is all sorts of devices and only on and off, I think…


root@freenas:~ # camcontrol devlist
<ST3000VN000-1HJ166 SC60>          at scbus0 target 0 lun 0 (ada0,pass0)
<SanDisk Ultra Fit 1.00>           at scbus7 target 0 lun 0 (da0,pass1)
<Terra USThree PMAP>               at scbus8 target 0 lun 0 (pass2,da1)

root@freenas:~ # gpart list da1
Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 60566015
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da1s1
   Mediasize: 31005671424 (29G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 4128768
   Mode: r0w0e0
   rawtype: 12
   length: 31005671424
   offset: 4128768
   type: !12
   index: 1
   end: 60566015
   start: 8064
Consumers:
1. Name: da1
   Mediasize: 31009800192 (29G)
   Sectorsize: 512
   Mode: r0w0e0

Any thoughts?

#13 Updated by Matt Kessler over 1 year ago

And here is why I want USB3 to work at all times :(

sent 30.95G bytes received 17.43K bytes 71.89M bytes/sec
total size is 30.94G speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1180) [sender=3.1.2]

before on USB2
sent 26.55G bytes received 17.14K bytes 26.54M bytes/sec
total size is 30.94G speedup is 1.17
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1180) [sender=3.1.2]

#14 Updated by Dru Lavigne over 1 year ago

  • Status changed from 15 to Investigation

#15 Updated by Dru Lavigne over 1 year ago

  • Status changed from Investigation to In Progress

#16 Updated by Vladimir Vinogradenko over 1 year ago

Matt, sorry, I was not receiving notifications for some reason.

Do you mean that the very same drive, plugged into USB 2.0 port, works, but if you plug it into USB 3.0 port, it does not? (or vice versa).

#17 Updated by Vladimir Vinogradenko over 1 year ago

  • Status changed from In Progress to Blocked
  • Reason for Blocked set to Waiting for feedback

#18 Updated by Matt Kessler over 1 year ago

Yes, Vladimir
this is the very same <Terra USThree PMAP> s/n 07083B88B71C4C20!
Please read log below "Now behold, behold"…
short version it was working 2 days in USB2 port; faild when switching to USB3 port;
worked fine on same USB3 port 1h later.

Jan 15 22:10:55 freenas da2: <Terra USThree PMAP> Removable Direct Access SPC-4 SCSI device
Jan 15 22:10:55 freenas da2: Serial Number 07083B88B71C4C20
Jan 15 22:10:55 freenas da2: 40.000MB/s transfers

Fails -- note, some details are missing

And here the very same thumb drive is inserted at a USB3 port

Jan 17 22:48:04 freenas ugen1.2: <Phison 2307 Boot ROM> at usbus1
Jan 17 22:48:04 freenas umass1 on uhub2
Jan 17 22:48:04 freenas umass1: <Phison 2307 Boot ROM, class 0/0, rev 3.00/1.10, addr 1> on usbus1
Jan 17 22:48:04 freenas umass1:  SCSI over Bulk-Only; quirks = 0xc100
Jan 17 22:48:04 freenas umass1:8:1: Attached to scbus8
Jan 17 22:48:04 freenas da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
Jan 17 22:48:04 freenas da1: < 2307 PRAM 1.00> Removable Direct Access SCSI device
Jan 17 22:48:04 freenas da1: 400.000MB/s transfers
Jan 17 22:48:04 freenas da1: Attempt to query device size failed: NOT READY, Medium not present
Jan 17 22:48:04 freenas da1: quirks=0x2<NO_6_BYTE>

Eject and reinsert
Jan 17 23:48:40 freenas da1: <Terra USThree PMAP> Removable Direct Access SPC-4 SCSI device
Jan 17 23:48:40 freenas da1: Serial Number 07083B88B71C4C20
Jan 17 23:48:40 freenas da1: 400.000MB/s transfers

#19 Updated by Dru Lavigne over 1 year ago

  • Status changed from Blocked to Not Started
  • Assignee changed from Vladimir Vinogradenko to Alexander Motin

Sasha: what are your thoughts on this one?

#20 Updated by Alexander Motin over 1 year ago

  • Subject changed from Import from a ntfs formatted usb3 disk fails to USB disk fails to identify properly on USB3 controller
  • Status changed from Not Started to Blocked

I would say that this device just looks broken to me, like its firmware can't start sometimes on USB3, or at least can't do it within expected time, reporting some dummy interfaces and device names. I am going to try asking FreeBSD's USB stack maintainer about it, but could you tell more about the device first?

Do those logs at the end belong to WD Elements USB3 you mentioned in the beginning?

Also I think that `usbconfig dump_device_desc` and `usbconfig dump_all_config_desc` for the respective device in both of its states could be useful.

#21 Updated by Matt Kessler over 1 year ago

Short summary, Alexander

I'm a sys admin and this is a testing box, but same hardware is being used in 3 locations.
So in total there are 4×HP servers, 3×WD Elements USB3 and ~30 thumb drives — often 3-5 of a kind.
I have seen most imports failing on ntfs formatted devices, as my job is, to transfer backups/dumps from a windows CRM to off site locations without internet connection.

So far Vladimir has found some issues with rsync an non ascii characters.

During my testing with other devices I could confirm the rsync/non ascii character issue also happens with Fat32 devices — resulting in missing files.

As for over two years with 9.3, 9.10 and now 11.1 I have had failing imports with more or less every device on all 4 servers, only sometimes I had successful imports, too.
At all times I'm using the USB3 ports, as I want the speed and they are easy to get to for me, so I also use them in case I have just al small transfer from a USB2 thumb drive.

By now I'm wondering, if the rsync issue leads to some improper unmount or strange device not ready state!?
When the import failed in the past I would often re-try and the freenas gui would not find the device anymore — which I always thought it was more a freenas bug.
This is the code — and I'm unsure what happens in case of rsync errors — but I no good with python

https://redmine.ixsystems.com/projects/freenas/repository/revisions/master/entry/src/middlewared/middlewared/plugins/pool.py#L258
and
https://redmine.ixsystems.com/projects/freenas/repository/revisions/master/entry/src/middlewared/middlewared/plugins/pool.py#L103

And at last… those logs at the end belong to one of 3 "Terra USThree PMAP" USB3 thumb drives(from a very reliable german computer manufacturer — a bit like IX in the US ☺) — which I used for further testing at the time. And I has been removed since then, so I got no `usbconfig dump_device_desc`, sorry.
What two states are you referring to USB2 and USB3 prot?

Well, for what it's woth, I found my old bug report…
https://redmine.ixsystems.com/issues/14477#change-72336

Thanks

#22 Updated by Alexander Motin over 1 year ago

Matt, lets not mix everything together, from encoding problem though middleware mounting code to USB device issues. Lets do one thing at a time, starting form the most basic ones. You've shown that some of your USB deices has problems with being detected (even reports different names, etc.) when plugged into USB3 port. Is that right? Obviously if device is not functional, other things by definition won't work. So, is there still and open question about those thumb drives, that require attention, or you already thrown them away? If still open -- please provide info. Otherwise lets move to some other issue. You say second insert after failed import does not work?

#23 Updated by Matt Kessler over 1 year ago

I wish it wasn't such a can of worms. From my point of view, it's that imports fail, for unknown reasons.

Just to make this clear non of those USB HDDs or thumb drives ever caused problems on non freenas systems:

Matt Kessler wrote:

Vladimir Vinogradenko wrote:

From what I know, this message only appears if file system is really broken. If you want to investigate this yourself, you're welcome :)

Thanks Vladimir
Broken in what sense? Bad partition layout, broken mbr…?
The above WD Elements 10B8 1007 is hooked up to a win 2008 server for nightly CRM dumps.
In the morning, I then walk to 3 locations down the street, plug it into 3 different Ubuntu 16.04 desktop computers and copy the backup(over the local network) to a freenas box.
I do this in all 3 building and none of these systems ever have a problem with the drive.
We got 3 of those WD USB3 about 2 yrs ago because the desktop computers in those buildings were only USB2 and 100Mbit,
I had the issues on and off back then already, when using the import tool and eventually gave up.

Please read #note-12
This is not me mixing things… it is a series of events.

#24 Updated by Vladimir Vinogradenko over 1 year ago

Matt, please note, this task is now only about your USB 2.0/3.0 drives issue. For all other bugs you've found I've created separate issues, they are clear to me and will be fixed soon.

#25 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 1 year ago

  • Due date set to 02/12/2018

Due date updated to reflect the code freeze for 11.1U2.

#26 Updated by Alexander Motin over 1 year ago

  • Target version changed from 11.1-U2 to 11.2-RC2

Matt Kessler wrote:

Just to make this clear non of those USB HDDs or thumb drives ever caused problems on non freenas systems:

Then it would be good to get some kind of systematized input about that side of the problem, related to USB behavior. Lets focus on some device you'd like to try to fix, collect the data and push them to FreeBSD USB maintainer.

The problem of the most USB devices is that they are tested only with Windows, and at least in past were not passing any significant certification for standards violation. As result they are working on Windows, but often not other OS. I even had devices working on older versions of Windows, but no on new ones, just because new versions behave differently and devices are really broken.

#27 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 1 year ago

  • Due date deleted (02/12/2018)

#28 Updated by Alexander Motin over 1 year ago

  • Status changed from Blocked to Closed
  • Reason for Closing set to Cannot Reproduce
  • Reason for Blocked deleted (Waiting for feedback)

#29 Updated by Dru Lavigne over 1 year ago

  • Target version changed from 11.2-RC2 to N/A
  • Private changed from Yes to No

#30 Updated by Dru Lavigne over 1 year ago

  • File deleted (debug-freenas-20180117172051.tgz)

Also available in: Atom PDF