Project

General

Profile

Bug #9123

Duplication of large files in Mac OSX 10.10.2 over smb results in corrupted files.

Added by Dennis Juhler Aagaard over 5 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Nice to have
Assignee:
John Hixson
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

When duplicating a large file 4+GB the outcome gets corrupted. I have tried with videofiles. The command used are high-lighting the file and pushing cmd+d. The finder goes into a I/O lock with spinning beachball, network activity are almost zero which leads me to think that it is doing a copy-on-server despite OSX don't support that. After a minute of I/O lock the newly duplicated files appear and I/O lock are gone. When playing the video file everything are great but after about 4GB played the video are playing weird green frames and weird looping. I will test this with AFP also.
Attached are a log.smbd with log level set to debug.

History

#1 Updated by Dennis Juhler Aagaard over 5 years ago

Did a cmd+d over AFP with a 13,6GB video file. Beside having I/O lock in finder with spinning beachball, it did the duplication without a problem and no corruption. When moving up in size of the file to 27,2GB I get a dialog with "The operation can’t be completed because an item with the name “test3.mov” already exists." and the duplication fails.

Something weird are going on behind the scenes here too. I saw this behaviour in FreeNAS 9.2.1.7 and then made a test rig with FreeNAS 9.3-STABLE-201503270027 to see if the updated versions of SAMBA and Netatalk did any good on this issue. The issues has changed a bit but the outcome is the same.

I have tried to put "log level = default:maxdebug" in Global auxiliary parameters: of AFP settings, but I dont see any debug logging in /var/log/messages any suggestions to getting debug logging to work in Netatalk?

#2 Updated by Dennis Juhler Aagaard over 5 years ago

  • File netatalk.debug added

Redirected the Netatalk debug log to a new file in the logs.
log file = /var/log/netatalk.debug
log level = default:maxdebug

Attached is the Netatalk debug log recorded while getting the failure.
I know these 2 issues might be related or might not. If you would like me to make 2 separate bug reports, please say so. :-)

#3 Updated by John Hixson over 5 years ago

  • Status changed from Unscreened to 15
  • Priority changed from Important to Nice to have
  • Target version changed from 213 to Unspecified

Can you go to system->advanced->"save debug" and attach to this ticket?

#4 Updated by Dennis Juhler Aagaard over 5 years ago

  • File debug-freenas-20150410090609.tar added

Debug file attached. :-)

#5 Updated by John Hixson over 5 years ago

Dennis Juhler Aagaard wrote:

Redirected the Netatalk debug log to a new file in the logs.
log file = /var/log/netatalk.debug
log level = default:maxdebug

Attached is the Netatalk debug log recorded while getting the failure.
I know these 2 issues might be related or might not. If you would like me to make 2 separate bug reports, please say so. :-)

Yes. At this point, I am unclear if this problem is over CIFS or AFP. Can you be more clear please? or is it both? How are you verifying there is corruption? If you copy a file, and say, do a sha256 checksum on the FreeNAS side, and then on the destination, is it the same? does it differ?

#6 Updated by Dennis Juhler Aagaard over 5 years ago

Ok. I see that the 2 issues are unrelated. Please disregard the AFP issue I will open a ticket for that issue.
So to get back to the CIFS/SMB issue. If you in Mac OSX connected via SMB duplicates a large file 4+GB on the FreeNAS, the out come gets corrupted.
Yes I have made a sha256 of both files but they are the same, but something happens to the video file.
The files are equal in size but sha256 for original file is SHA256 (test2.mov) = 3402034ddb27fa25d2323293373b03b4f790cf6f4b23a3cdcced34a0c633977e
and sha256 for duplicated file is SHA256 (test2 copy.mov) = 3402034ddb27fa25d2323293373b03b4f790cf6f4b23a3cdcced34a0c633977e
I will post a link with the 2 files for you to download. The original and the duplicated file so you can see what kind of corruption. :-)

#7 Updated by Dennis Juhler Aagaard over 5 years ago

Download link for the original file and corrupted duplicate.
https://owncloud.stjernholms.com/public.php?service=files&t=75c41d43e655e054b8d69ec4d1170434
Its zip'ed.
bear in mind that its 8.9GB

#8 Updated by John Hixson over 5 years ago

  • Status changed from 15 to Investigation

Dennis Juhler Aagaard wrote:

Download link for the original file and corrupted duplicate.
https://owncloud.stjernholms.com/public.php?service=files&t=75c41d43e655e054b8d69ec4d1170434
Its zip'ed.
bear in mind that its 8.9GB

Dennis, I am downloading the archive, however, given that the sha256 is identical, I don't see how there could be corruption?

#9 Updated by John Hixson over 5 years ago

Dennis, I downloaded the file but can't unzip it ;-) At least with the unzip program I have. Anyhow, given the sha256 is identical, the copy is no different from the original. So if the copy is corrupted, the original must be corrupted as well.

#10 Updated by John Hixson over 5 years ago

  • Status changed from Investigation to 15

Dennis,

Would it be possible to do a teamviewer session? I think it would be a lot easier to just show me what you think is happening.

#11 Updated by John Hixson over 5 years ago

Dennis, see my previous post ;-)

#12 Updated by Dennis Juhler Aagaard over 5 years ago

Yes. Sorry. Hard weekend with a lot to do. ;-)
When do you have some time? I am located in Europe GMT+1. But i will try to see if i can setup teamviewer on a client for you to play with.
The FreeNAS box is just a test unit for me to reproduce what my clients finds of different issues.
So I will setup a client, and then we maybe can make a skype conversation tonight (my time)?

#13 Updated by John Hixson over 5 years ago

Dennis Juhler Aagaard wrote:

Yes. Sorry. Hard weekend with a lot to do. ;-)
When do you have some time? I am located in Europe GMT+1. But i will try to see if i can setup teamviewer on a client for you to play with.
The FreeNAS box is just a test unit for me to reproduce what my clients finds of different issues.
So I will setup a client, and then we maybe can make a skype conversation tonight (my time)?

This works for me. I don't have skype though. I have various IM clients though. I'm pretty flexible as far as times to do a teamviewer session provided we agree on it ahead of time. Just let me know.

#14 Updated by John Hixson over 5 years ago

I'm still waiting to hear from Dennis on a good time to do this

#15 Updated by Dennis Juhler Aagaard over 5 years ago

Sure, can you send me an email at dennis at kulmosen.dk, I can then send you Teamviewer credentials to the setup.
I tried updating to 10.10.3 to see if that solved anything, but no. :-S

#16 Updated by John Hixson over 5 years ago

I sent you an email ;-)

#17 Updated by John Hixson over 5 years ago

I'm still waiting to hear from Dennis

#18 Updated by John Hixson over 5 years ago

  • Status changed from 15 to Closed: Cannot reproduce

I can't reproduce this, haven't had any other reports of this and haven't heard back from Dennis, so I'm closing this.

#19 Updated by John Hixson over 5 years ago

  • Status changed from Closed: Cannot reproduce to Screened

Back to screened. I received an email from Dennis with necessary info.

#20 Updated by John Hixson over 5 years ago

I'm hoping to look at this over the weekend.

#21 Updated by John Hixson over 5 years ago

I didn't get a chance this weekend, but will this week.

#22 Updated by John Hixson over 5 years ago

Dennis, I've still got your credentials. Provided they still work, I will at least take a look at this over the weekend and let you know what I think.

#23 Updated by John Hixson over 5 years ago

  • Seen in changed from to 9.3-RELEASE

well, I'm sorry, I got preoccupied this weekend (again). I will be looking at this this week though ;-) Promise ;-)

#24 Updated by John Hixson over 5 years ago

  • Status changed from Screened to 15

Hi Dennis,

I logged into your system, however, I'm not sure what to do after that ;-) Can we coordinate a time where you can demonstrate your issue(s) to me?

#25 Updated by Dennis Juhler Aagaard over 5 years ago

Sure. As said before the time difference is a challenge. I live in Denmark, Europe. which is CEST or UTC+2. I will try and make a spot Thursday my evening at 21.00 (12.00 at iXSystem in San Jose) But please wait for my confirmation, just need to sync with wife's schedule. :-)

#26 Updated by John Hixson over 5 years ago

Dennis Juhler Aagaard wrote:

Sure. As said before the time difference is a challenge. I live in Denmark, Europe. which is CEST or UTC+2. I will try and make a spot Thursday my evening at 21.00 (12.00 at iXSystem in San Jose) But please wait for my confirmation, just need to sync with wife's schedule. :-)

Hi Dennis,

Time difference isn't that big of a problem for me as long as we agree on it beforehand. I do tend to work late ;-) Let's try for 12 noon PDT (19h UTC) as you suggested. Please email me your info beforehand ()

#27 Updated by Dennis Juhler Aagaard over 5 years ago

  • File debug-freenas-20150528215402.tar added

#28 Updated by Dennis Juhler Aagaard over 5 years ago

  • File samba4.zip added

#29 Updated by John Hixson over 5 years ago

  • Status changed from 15 to Investigation

Spent time with Dennis on this one. I witnessed the issue personally. It's obvious that a server side copy is being performed. I didn't find anything unusually strange, however I did not spend a lot of time looking around the system. I had Dennis do a couple copies while having samba logging cranked up. Dennis has sent me the logs and I'm going to be reviewing them for anything abnormal. Dennis has also left the system available for me to poke around, so I will also be doing that to check into this further if I need to do so.

#30 Updated by Dennis Juhler Aagaard over 5 years ago

Connecting with SMB1 from OSX solves the corrupted duplications. But then we are back to old school and very slow SMB1.
OSX should be supporting SMB3

SupportiMac:~ supportimac$ smbutil statshares -m /Volumes/FreeNASSMB

==================================================================================================
SHARE ATTRIBUTE TYPE VALUE ==================================================================================================
FreeNASSMB
SERVER_NAME 10.0.10.135
USER_ID 502
SMB_NEGOTIATE AUTO_NEGOTIATE
SMB_VERSION SMB_3.0
SMB_SHARE_TYPE DISK
SIGNING_SUPPORTED TRUE
EXTENDED_SECURITY_SUPPORTED TRUE
LARGE_FILE_SUPPORTED TRUE
FILE_IDS_SUPPORTED TRUE
DFS_SUPPORTED TRUE
MULTI_CREDIT_SUPPORTED TRUE
ENCRYPTION_SUPPORTED TRUE


But when forcing the max and min protocols to SMB3 in FreeNAS, the share is not connectable.

#31 Updated by Dennis Juhler Aagaard over 5 years ago

Even when forcing OSX to use SMB2.1, it will try to make a copy on server when duplicating, and the result is corruption. Something weird is going on in Apple's SMB stack.
When i get the time i will try to setup another system and try to replace Apple's SMB stack with a version from HomeBrew or the like. :-)

#32 Updated by John Hixson over 5 years ago

I'm hoping to get some time this week to review the logs. I haven't yet had a chance to do so.

#33 Updated by John Hixson over 5 years ago

I'm still hoping to find time to look at the logs

#34 Updated by John Hixson over 5 years ago

I'll probably be able to review the logs tomorrow.

#35 Updated by John Hixson over 5 years ago

John Hixson wrote:

I'll probably be able to review the logs tomorrow.

I can probably look at this sometime this week ;-) I'm sorry for the delay.

#36 Updated by John Hixson over 5 years ago

I haven't had a chance to look at this yet, possibly today.

#37 Updated by Reqlez Guy over 5 years ago

I can tell you that Apple is going crazy with their SMB stack... it has lots of bugs. Trying to make a mac shop happy with SMB is pretty hard, permission issues, disconnect issues, etc. And the mac os text editor is bugged as well, won't let us make changes and save the changes to SMB share but word for mac works just fine.

Maybe since their UNIX director left they been having issues writing clean code ;-)

#38 Updated by John Hixson over 5 years ago

I'm hoping to look at this sometime this week.

#39 Updated by John Hixson over 5 years ago

Still hoping to look at this

#40 Updated by John Hixson about 5 years ago

This is still on my radar for next week.

#41 Updated by John Hixson about 5 years ago

Still hoping to do this sometime this week

#42 Updated by John Hixson about 5 years ago

still hoping to look at this but haven't yet

#43 Updated by John Hixson about 5 years ago

still haven't had a chance to look at this.

#44 Updated by John Hixson about 5 years ago

still haven't had a chance to look at this.

#45 Updated by John Hixson about 5 years ago

well, looks like this is pushed to next week again.

#46 Updated by Dennis Juhler Aagaard about 5 years ago

Due to company vacation, the test setup has been shutdown. It will be back online from the 27'th of July.
I know that this is not high in priority, but I have customers that are waiting for a fix on this. They are using very time consuming work arounds, but are patiently waiting.
Happy summer for you and yours. :-)

#47 Updated by John Hixson about 5 years ago

Dennis Juhler Aagaard wrote:

Due to company vacation, the test setup has been shutdown. It will be back online from the 27'th of July.
I know that this is not high in priority, but I have customers that are waiting for a fix on this. They are using very time consuming work arounds, but are patiently waiting.
Happy summer for you and yours. :-)

Hi Dennis,

I'm sorry I haven't had time to dig into this again. I will get to it though ;-) If lots of folks were reporting this it would be a higher priority, but honestly you're the only one. I did see another ticket where transfers over AFP were being corrupted, not sure if it's related or not, but might be worth looking at.

#48 Updated by John Hixson about 5 years ago

Still on hold

#49 Updated by John Hixson about 5 years ago

I haven't had time to look at this yet.

#50 Updated by John Hixson about 5 years ago

I haven't had time to look at this yet.

#51 Updated by John Hixson about 5 years ago

I'll possibly get a chance to review this this week

#52 Updated by John Hixson about 5 years ago

Still shooting for this week to look at this some more

#53 Updated by John Hixson about 5 years ago

Dennis,

I've been looking at the logs. It's going to be a while, since they are rather big. However, before doing so I reviewed this ticket. Given that the files are identical in size and have an identical sha256, I do not feel that there is corruption occurring. I believe there to be something else wrong here. Perhaps playing movies off a CIFS share? Maybe some numbers need to be tuned? Either way, it's been a while and I would like to do another teamviewer session when you are available again.

#54 Updated by John Hixson about 5 years ago

Still looking at this, will continue tomorrow.

#55 Updated by John Hixson about 5 years ago

Still up in the air on this. I really want to look at your system again.

#56 Updated by Dennis Juhler Aagaard about 5 years ago

Sure, I am on vacation right now and the system has been shut down during the holidays. Company policy.
I will return next monday and fire it all up again for you. :-)

#57 Updated by John Hixson about 5 years ago

  • Status changed from Investigation to 15

Hi Dennis,

It is monday and I haven't heard from you ;-) Can you please email me your availability as well as your teamviewer info again please?

#58 Updated by John Hixson about 5 years ago

  • Status changed from 15 to Closed: Cannot reproduce

Still no reply. This has been open forever, and I'm unable to reproduce it. The sha256's are the same, the file sizes are the same. I do not believe any corruption is occurring. It was suggested it could be your video player software, or any number of other things.

#59 Updated by Dennis Juhler Aagaard about 5 years ago

Sorry John, I have a huge pile of work after the holidays. I am planning to get your access back on track to the system on Monday or Tuesday.
The checksums are not the same, that was me not doing it correctly. So the problem are there. I can reproduce it on several systems running various versions of FreeNAS. It only takes a Mac connecting via SMB and then make some duplicates or copy by dragging the data plus holding down ALT-key.
I am planning to install the newest Mac OS X Beta El Capitan, to see if the issue has been resolved client-side.

#60 Updated by John Hixson about 5 years ago

  • Status changed from Closed: Cannot reproduce to Investigation

#61 Updated by John Hixson about 5 years ago

  • Status changed from Investigation to 15

Hi Dennis,

Any updates ?

#62 Updated by John Hixson about 5 years ago

Still waiting to hear from Dennis on this. Dennis, any updates? You claim the sha256's are not the same now? I swear I watched you do it. Either way, I really need to see this again. I need movement on this ticket to happen or I will have to close it.

#63 Updated by Dennis Juhler Aagaard about 5 years ago

Are you available within 1.5 hour? I am on way to work now and can make a teamviewer session as soon as i get there. I can call you when i am ready if you give me your number at my email dennis at kulmosen.dk

#64 Updated by John Hixson about 5 years ago

Dennis Juhler Aagaard wrote:

Are you available within 1.5 hour? I am on way to work now and can make a teamviewer session as soon as i get there. I can call you when i am ready if you give me your number at my email dennis at kulmosen.dk

Dennis,

Please let me know your availability. My email address is . I do need to know ahead of time.

#65 Updated by John Hixson about 5 years ago

John Hixson wrote:

Dennis Juhler Aagaard wrote:

Are you available within 1.5 hour? I am on way to work now and can make a teamviewer session as soon as i get there. I can call you when i am ready if you give me your number at my email dennis at kulmosen.dk

Dennis,

Please let me know your availability. My email address is . I do need to know ahead of time.

Dennis,

Any updates?

#66 Updated by John Hixson about 5 years ago

  • Status changed from 15 to Investigation

John Hixson wrote:

John Hixson wrote:

Dennis Juhler Aagaard wrote:

Are you available within 1.5 hour? I am on way to work now and can make a teamviewer session as soon as i get there. I can call you when i am ready if you give me your number at my email dennis at kulmosen.dk

Dennis,

Please let me know your availability. My email address is . I do need to know ahead of time.

Dennis,

Any updates?

Received email from Dennis, trying to coordinate a time.

#67 Updated by John Hixson about 5 years ago

I was finally able to schedule a session with Dennis tonight. I spent several hours on this and witnessed the issues first hand.

So what is happening is that files larger than 4GB are being corrupted when copying using OS X. If a file resides on a FreeNAS share, and you either "copy" or "duplicate" it, it will do so, creating a copy of the same file size, but with a different sha256 and corruption. In the case here, these are movie files and the corruption only occurs after 4GB and it is easy to see by skipping the movie past the 4GB mark.

I tested the same issue on his network using the same FreeNAS share on a Windows box and no corruption occurred.

To help with debugging this issue, I repeated the copy on both Mac OS X and Windows, while logging individually to separate files and have requested Dennis upload those to this ticket. I have also asked him to make the movie available to me (which he has) so that I can continue debugging this issue locally.

At least at this point, I can verify corruption is indeed occurring, it's specific to Mac OS X, and it only occurs past 4GB.

#68 Updated by Dennis Juhler Aagaard about 5 years ago

  • File samba_corruption_logs.tgz added

I have collected the Log file as John mentions.

Link to the movie file is: http://stjernholms.com:8000/_MnzM-J6-p1fTWR
The link is active 1 month from today.

Still planning on testing this issue with the Beta Mac OS X 10.11 (El Capitan) when I get the change.

#69 Updated by John Hixson about 5 years ago

3594

I'm still working on this. I've got your movie downloaded and on a CIFS share. I've got OS X Yosemite 10.10.4 to test with. I can't reproduce your case exactly since when I try and "duplicate" the file, it never succeeds because it fails with the attached screenshot. This is proving to be difficult ;-)

#70 Updated by John Hixson about 5 years ago

To add to my previous post, the duplicate never gets past about ~300M or so.

#71 Updated by John Hixson about 5 years ago

I haven't had a chance to get back to this yet from the other day. My attempts to duplicate the issue failed miserably however. I'll probably be able to spend more time on this over the weekend.

#72 Updated by Dennis Juhler Aagaard about 5 years ago

Please let me know if I can do anything on this matter. :-)

#73 Updated by John Hixson about 5 years ago

  • Status changed from Investigation to 15

Dennis Juhler Aagaard wrote:

Please let me know if I can do anything on this matter. :-)

Dennis,

I might need to schedule another session with you where I can reproduce the issue and look at the logs again. Let me know if you can do that.

#74 Updated by John Hixson about 5 years ago

  • Status changed from 15 to Investigation

#75 Updated by Jordan Hubbard over 4 years ago

  • Status changed from Investigation to Closed

BRB: No feedback after 5 months, closing.

#76 Updated by Dennis Juhler Aagaard over 4 years ago

Sry, I didn't see that you where waiting for my input again. :-S
To follow up on this issue, it seems to have been an Apple bug which has vanished with the introduction of El Capitan. I can't reproduce the issue on my client running OS X 10.11.3. So with which release it has been solved in, I don't know.

Result of 22.61 GB large file duplicated as mentioned in the bug report.

$ shasum HD\ CAM\ TV\ MASTERS\ PRORES\ 422\ HQ.mov HD\ CAM\ TV\ MASTERS\ PRORES\ 422\ HQ\ copy.mov
30b5f20d66dd27d9742ddfaae2c045e6864eefda HD CAM TV MASTERS PRORES 422 HQ.mov
30b5f20d66dd27d9742ddfaae2c045e6864eefda HD CAM TV MASTERS PRORES 422 HQ copy.mov

#77 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Target version changed from Unspecified to N/A

#78 Updated by Dru Lavigne almost 3 years ago

  • File deleted (log.smbd)

#79 Updated by Dru Lavigne almost 3 years ago

  • File deleted (netatalk.debug)

#80 Updated by Dru Lavigne almost 3 years ago

  • File deleted (debug-freenas-20150410090609.tar)

#81 Updated by Dru Lavigne almost 3 years ago

  • File deleted (debug-freenas-20150528215402.tar)

#82 Updated by Dru Lavigne almost 3 years ago

  • File deleted (samba4.zip)

#83 Updated by Dru Lavigne almost 3 years ago

  • File deleted (samba_corruption_logs.tgz)

Also available in: Atom PDF