Project

General

Profile

Feature #5904

vfs_fruit (better Samba/Netatalk integration)

Added by Thomas Kaiser about 4 years ago. Updated 28 days ago.

Status:
Unscreened
Priority:
Nice to have
Assignee:
William Grzybowski
Category:
Services
Target version:
Estimated time:
Sprint:
Severity:
Medium
Backlog Priority:
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Description

Starting with Samba 4.2 a new VFS module called vfs_fruit (instead of vfs_apple) will be available to Samba that will address the most common challenges accessing the same set of data from OS X clients via AFP and SMB: File/record locking, encoding quirks introduced by Apple and especially access to Finder metadata and ressource forks. The author of the new module outlined the problems in [1]

In my opinion we need early adoption of this new VFS module as soon as Samba 4.2 is integrated in FreeNAS since it will both enhance compatiblity between Netatalk/Samba and also be a perfect starting point for the replacement of AFP with SMB2 for OS X 10.9/10.10 clients later (Samba 4.3 is expected to contain the loads of work to make Spotlight searches possible over SMB2 [2]). I put the manual page of vfs_fruit online: http://pastebin.com/TtSBeKB0

In our case (ZFS based) it's also necessary to stack the other necessary VFS modules vfs_catia and vfs_streams_xattr correctly and to config it all accordingly:

[cross platform share]
vfs objects = catia fruit streams_xattr
fruit:resource = xattr
fruit:metadata = netatalk
fruit:locking = netatalk
fruit:encoding = private

Best,

Thomas

[1] http://sambaxp.org/fileadmin/user_upload/SambaXP2014-DATA/wed/track1/Ralph_Boehme-AppledancesSamba.pdf
[2] https://lists.samba.org/archive/samba-technical/2014-August/101791.html

History

#1 Updated by John Hixson about 4 years ago

  • Status changed from Unscreened to Screened

#2 Updated by Nicki Messerschmidt about 4 years ago

I think this would be a very useful feature for FreeNas. I second this feature request.

#3 Updated by Thomas Kaiser about 4 years ago

BTW: Ralph the developer responsible for Samba's new vfs_fruit module shares a lot of knowledge in this list post regarding OS X as SMB client:

https://lists.samba.org/archive/samba/2014-September/184761.html

Since Apple officially declared AFP end of live it's just a matter of time until Apple ships OS X without an AFP client so users have to think about switching protocols... and file server daemons. Samba 4.3 is expected to be one of the few '100 percent OS X compliant' SMB servers but to make a switch from Netatalk/AFP to Samba/SMB hasslefree it's necessary to configure both daemons appropriately.

I believe this should be one of the design goals regarding Samba running in FreeNAS/TrueNAS since both will automatically become more attractive for OS X installations since many admins will realize that Apple's switch of protocols doesn't mean a MS Windows based fileserver can serve both OS X and Windows clients since Windows will never implement Apple proprietary SMB extensions like Spotlight support or the AAPL stuff Ralph mentions in his list post above leading to painfully slow SMB share browsing from OS X clients.

Unfortunately I've been wrong regarding storage of Finder metadata and ressource forks in ZFS Extended Attributes since FreeBSD's ZFS implementation lacks the necessary xattr APIs available on Solaris. Therefore it must read "fruit:resource = file" instead. Both Netatalk and Samba should be configured to store the stuff in ._ AppleDouble files and vfs_fruit should hide the inner details from clients.

#4 Updated by Damjan Perenic over 3 years ago

I vote for this feature as well as I believe it is a must have feature for FreeNAS. I keep my photos on FreeNAS and some programs work awfully slow without this plugin (e.g. CaptureOne). Browsing through directories is much much slower than on the local drive. I tested it on Samba-beta and CaptureOne for example, works noticeably faster.

#5 Updated by Jordan Hubbard over 2 years ago

Possible calsoft candidate

#6 Updated by Jordan Hubbard over 2 years ago

  • Assignee changed from John Hixson to Jakub Klama

#7 Updated by Anonymous over 2 years ago

freeNAS-9.10 (FreeBSD freenas.local 10.3-RC1 FreeBSD 10.3-RC1 #2 8d7981f(freebsd10))
on-wards /usr/local/lib/shared-modules/vfs/fruit.so is the shared library that introduces vfs_fruit module in freeNAS.

Till 9.3-RELEASE-p31
Samba version: 4.1.21 was used which did not had vfs_fruit module support.

As suggested by http://download.freenas.org/9.10/MASTER/201603290530/ReleaseNotes
Filesharing:
Samba (SMB filesharing) updated from version 4.1 to 4.3.4 freeNAS-9.10 and on-wards.
This is also seen in freeNAS-10.2 (10.3-BETA2) which has Samba version: 4.3.4-GIT-UNKNOWN and have vfs_fruit module support.

The source code is also reachable at samba-4.2.9/source3/modules/vfs_fruit.c

#9 Updated by Jakub Klama over 2 years ago

  • Assignee changed from Jakub Klama to Anonymous

Reassigning to Calsoft team

#10 Updated by Anonymous about 2 years ago

Looks like vfs_fruit support is already added to FreeNAS9.10(also mentioned by Rohit above). Also related ordering Bug(Bug#16325) looks resolved now. What exactly is now expected to be done here?

#11 Updated by Josh Paetzel about 2 years ago

  • Status changed from Screened to Closed: Not Applicable

Nothing.

#12 Avatar?id=14398&size=24x24 Updated by Kris Moore about 2 years ago

  • Target version changed from 111 to N/A

#13 Updated by John Hixson about 2 years ago

  • Status changed from Closed: Not Applicable to Fix In Progress
  • Assignee changed from Anonymous to John Hixson
  • Target version changed from N/A to 9.10.2

This is a thing.

#14 Avatar?id=14398&size=24x24 Updated by Kris Moore almost 2 years ago

  • Target version changed from 9.10.2 to 9.10.2-U1

#15 Updated by Evan Champion almost 2 years ago

I tried this with my Macs (10.11 and 10.12) and found that using the recommended settings makes the share incompatible with an existing "normal" SMB share (without fruit) used by a Mac. Specific differences were:

- catia filename mapping is not the same as samba's default, so some files are not accessible. catia does not seem to be required and removing catia addressed this.
- metadata stored in a different xattr. Using fruit:metadata = stream addressed this.
- fruit sets ACLs on all new files, whereas my samba share (with dataset type = UNIX, case sensitive) did not. Using fruit:nfs_aces = no addressed this.

It seems like one needs to understand the specific deployment scenario and align to that:

- if one is replacing/augmenting an existing netatalk environment then perhaps the recommended configuration above using catia and netatalk-aligned fruit: settings are correct (I have not tried; I only use afp for time machine until samba support's Apple's requirements)
- if one is augmenting an existing SMB share then catia may be an issue and fruit:metadata = stream is required, fruit:locking should not be set.
- if one is adding fruity SMB to an existing NFS share, one might want to set other parameters to get the resulting file handling to match as closely as possible to native access from UNIX, which seemed to be: use catia, fruit:metadata = stream, fruit:encoding = native, streams_xattr:prefix = user., streams_xattr:store_stream_type = no. This is a bit speculative as I didn't test with an NFS client.
- perhaps fruit:nfs_aces is appropriate for Windows datasets, but not a UNIX dataset without ACLs. Unlike samba by default, fruit creates files with ACLs set unless fruit:nfs_aces = no.

In all cases fruit:resource = file is required on FreeBSD's zfs for support of large resource forks.

A much bigger issue however is that even with no special smb.conf (using only FreeNAS' default settings) and no fruit: special settings, I have strange issues with access to the share that prevent further use. Here are examples:

- From Terminal I can create, list, change, view, then delete a given file.
- From Finder I can see all files and can view files, but cannot copy, delete them. The behaviour is that you delete the file, it goes away for a couple seconds, and then appears again.
- If I copy a file into the share using Finder, I can not delete it using Terminal either -- but I can view or change it. The error given on delete is either file not found, or resource busy if I have just edited the file.
- If I login to FreeNAS via ssh and remove the file using the exact filename that was "not found" from Terminal, I can remove the file fine.

This is on 9.10.1-U4, UNIX dataset, case sensitive; I also tried making a Windows dataset with same behaviour. I have not figured out how to get this to work reasonably. Are people using this successfully running a nightly build with a newer version of samba perhaps?

Back to Apple's requirements for time machine support (https://developer.apple.com/library/content/releasenotes/NetworkingInternetWeb/Time_Machine_SMB_Spec/) there are a number of changes required. These are described in the vfs_fruit feature request for time machine support (vfs_fruit feature request discussing time machine requirements: https://bugzilla.samba.org/show_bug.cgi?id=12380).

- samba needs to be configured to support for durable file handles. smb.conf(5) says:
durable handles are only enabled if kernel oplocks = no,
kernel share modes = no, and posix locking = no, i.e. if the share
is configured for CIFS/SMB2 only access, not supporting
interoperability features with local UNIX processes or NFS
operations.

- F_FULLFSYNC support needs to be advertised via fruit:advertise_fullsync = true, and some code that not appear to be in a production samba release yet (pull request: https://github.com/samba-team/samba/pull/64).

- Specific requirements for Bonjour advertising. I have not checked in FreeNAS implements already.

#16 Avatar?id=14398&size=24x24 Updated by Kris Moore almost 2 years ago

  • Target version changed from 9.10.2-U1 to 9.10.3

#17 Updated by John Hixson over 1 year ago

  • Status changed from Fix In Progress to Investigation

Much work has been done. Much more is needed.

#18 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Target version changed from 9.10.3 to 9.10.4

#19 Updated by Thomas Kaiser over 1 year ago

John Hixson wrote:

Much work has been done. Much more is needed.

Just for the record: Ralph (SerNet dev) sent me his somewhat cleaned up smb.conf. We tested the settings at various customers and performance seemed to be fine though we ran into some weird issues that might be related to locking (but I asked for the worst case and graphics department delivered):

[global]
    netbios name = slowserver    
    server role = standalone
    workgroup = SLOW

    idmap config * : backend = tdb
    idmap config * : range = 10000-20000
    disable netbios = yes
    smb ports = 445

    debug pid = yes
    log file = /var/log/samba/%m.log
    log level = 2

    read only = no
    ea support = yes
    store dos attributes = yes

    # for OS X symblink bug
    aio read size = 1024
    aio write size = 1024

[osx]
    path = /shares/osx
    vfs objects = catia fruit streams_xattr acl_xattr

#20 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Target version changed from 9.10.4 to 11.1

#21 Updated by Dru Lavigne about 1 year ago

  • Status changed from Investigation to 46

John: is there anything that stills need to be fixed here or was it resolved with the commit on the related bug?

#22 Updated by John Hixson about 1 year ago

  • Status changed from 46 to Screened

Dru Lavigne wrote:

John: is there anything that stills need to be fixed here or was it resolved with the commit on the related bug?

There is much work that needs to be done here. This is a more general ticket about AFP and SMB integration, which sadly still needs to happen even with AFP going away.

#23 Avatar?id=14398&size=24x24 Updated by Kris Moore about 1 year ago

  • Target version changed from 11.1 to 11.2-BETA1

#24 Avatar?id=14398&size=24x24 Updated by Kris Moore 11 months ago

  • Target version changed from 11.2-BETA1 to 11.3

#25 Avatar?id=14398&size=24x24 Updated by Kris Moore 10 months ago

  • Status changed from Screened to Not Started

#26 Updated by John Hixson 8 months ago

  • Assignee changed from John Hixson to Timur Bakeyev

#27 Avatar?id=13649&size=24x24 Updated by Ben Gadd 7 months ago

  • Target version changed from 11.3 to Backlog

#28 Updated by Timur Bakeyev 6 months ago

  • Severity set to Medium

Seems that requires Samba 4.8 to be fully functional...

#29 Updated by John Hixson 5 months ago

  • Assignee changed from Timur Bakeyev to John Hixson

#30 Updated by John Hixson 4 months ago

  • Category changed from OS to Services

#32 Updated by Dru Lavigne 29 days ago

  • Assignee changed from John Hixson to William Grzybowski

#33 Updated by William Grzybowski 28 days ago

  • Status changed from Not Started to Unscreened
  • Target version changed from Backlog to 11.3

Andrew, does 11.3 seems like a good target for this since we gonna have samba 4.9?

Also available in: Atom PDF