Bug #26994
Add zfs_space and zfsacl as default modules to VFS objects
Description
Description¶
Decided to upgrade my production FreeNAS-9.10.2-U1 (86c7ef5) box to the latest FreeNAS-11.0-U4 (54848d13b) via GUI. The upgrade was successful with no noticeable errors. All services started and it joined to Active Directory successfully. Shares to ESXi via NFS and some SMB shares issued via folder redirection to Windows machines from Active Directory worked as expected. Two shares however presented me with an "Access Denied" error message. Checking permissions via Windows Server 2012, Windows 10 and Windows 7 machines showed missing permissions. Only the owner and owner group were listed with all permissions missing. A check beside the "Special Permissions" was the only box checked and it was grayed out (See attachment 01.jpg). Continuing to 'Advanced' would show the owners correct permissions (See attachment 02.jpg). Any attempt to change permissions or add a user threw an error "Unable to save permissions...The parameter is incorrect." (See attachment 03.jpg which shows an attempt to add a new group.) There were several other folders that continued to work, checking permissions on those showed similar problems to the ones not accessible. It would seem that the permissions were still there just not registering on Windows Machines. This was confirmed by running getfacl <share>
which listed all permissions as expected.
Setup¶
- Windows Active Directory environment forest level 2012.
- Multiple versions of Windows clients from Win7-Win10.
- FreeNAS Server AMD Opteron 2419EE 32G RAM 8Tib raidz2
Troubleshooting¶
FreeNAS 11.0¶
- Starting with FreeNAS I first rebooted the system twice watching for any errors. The only thing I noticed was this bit:
@winbindd not running? (check /var/run/samba/winbindd.pid) smbd not running? (check /var/run/samba/smbd.pid) nmbd not running? (check /var/run/samba/smbd.pid)
- Checking each of those files showed unique PID's in each file. This is likely irrelevant.
- I verified that Active Directory was connected by the usual commands.
[root@host] ~# wbinfo -t checking the trust secret for domain <Sanitized-Domain> via RPC calls succeeded
[root@host] ~# wbinfo -u administrator guest krbtgt #Sanitized users....
[root@host] ~# wbinfo -g winrmremotewmiusers__ dhcp users dhcp administrators domain computers #Sanitized groups....
- Also attempted setting ACL's with
setfacl
which would successfully change permissions however Windows machines did not recognize the change and continued presenting the same errors. - Under the SMB properties I added
ntlm auth = yes
and restarted. No change. - Added additional perameters from another post with similar issues.
lanman auth = no ntlm auth = yes client NTLMv2 = yes client lanman auth = no client plaintext = no
- After the reboot SMB failed the sanity check. Removed those and restarted and sanity check passed.
Windows¶
- Windows Server 2012 released KB3041857 for an issue with network resources that would give this error. However this was already installed during a monthly roll-up. Uninstalling this patch and reinstalling did not change anything.
- Tried to take ownership of the share but it gave the same error and would not let you proceed.
- Tried removing the share completely and re-adding it.
net use /delete
and also flushed DNSipconfig /flushdns
- Verified group policies permitted the shares and tested other local shares from another Windows Server instance.
FreeNAS 9.10
- Rolled back to 9.10.2-U1 and all shares and permissions were working as expected.
Resolution¶
- After the roll back to 9.10.2-U1 I verified all SMB configs and tried 11.0-U4 again.
- After the reboot the same symptoms presented themselves on the same shares.
- Removed one of the SMB shares and re-added it. This resolved my issue.
- Comparing previous config to the new config generated all VFS Objects had been cleared (See attachment 04.png).
- If you create a new share the following objects are automatically added:
zfs_space zfsacl streams_xattr aio_pthread
- Adding
zfsacl
to the impacted shares allowed all permissions to be accessible and changeable again in Windows (obviously!)
Misc¶
During my troubleshooting I saw that sometime after 9.10-U1 there was a Samba update from 4.1 to 4.3.4. Because I made such a big jump this could have been present before FreeNAS 11. I checked the upgrade notes for several of the versions in between the two releases and saw no mention of having to re-add zfsacl's.
Also, I have debug dumps from before and after the upgrade if someone could provide a secure way of sending those I would be happy to provide them.
Related issues
Associated revisions
History
#1
Updated by James Thompson about 2 years ago
Checking into this further it appears that 9.10U1 did not include zfs_space & zfsacl VFS objects in the drop-down menu as they are "always enabled." However 11.0 has added them to the menu and was removed from the note as an "Always enabled" object under table 10.4.2 in the User Guide under Sharing > SMB.
#2
Updated by Dru Lavigne about 2 years ago
- Status changed from Unscreened to 15
James, does enabling those resolve this for you? If not, please attach a debug (System -> Advanced -> Save Debug).
#3
Updated by James Thompson about 2 years ago
Yes. Adding zfsacl
resolved the issue. I took it a bit further and matched my existing shares to a newly created share and added:
zfs_space zfsacl streams_xattr aio_pthread
#4
Updated by Dru Lavigne about 2 years ago
- Status changed from 15 to Unscreened
- Assignee changed from Release Council to John Hixson
John: should there be a migration for this?
#5
Updated by John Hixson about 2 years ago
- Status changed from Unscreened to Screened
- Target version set to 11.1-U1
#6
Updated by John Hixson about 2 years ago
Dru Lavigne wrote:
John: should there be a migration for this?
There is a migration for this already. I'm not sure what went wrong here, but it seems as though the migration (and perhaps others) did not run. Can you see if /data/update.failed exists?
#7
Updated by John Hixson about 2 years ago
- Status changed from Screened to 15
#8
Updated by James Thompson about 2 years ago
/data/update.failed
does not exist. I am going to attempt an upgrade again either tonight or tomorrow night and pull the debugs again.
When reading AD and ldap dumps I noticed that a few of the shares had zfs_space zfsacl
on them. These shares were the ones that allowed me to open them but they did not allow me to edit the ACL's. After I went through and applied zfs_space zfsacl
to them I verified all shares showed ACL's and allowed me to edit. I'm curious why if zfs_space zfsacl
was indeed listed on the shares why it did not 'register' until after re-applying them.
When I go through it again I'm hoping to have a better understanding of this. While I'm doing this is there anything you all would like me to look for?
#9
Updated by Dru Lavigne about 2 years ago
- Status changed from 15 to Investigation
#10
Updated by Ben Gadd almost 2 years ago
- Assignee changed from John Hixson to Timur Bakeyev
#11
Updated by Ben Gadd almost 2 years ago
- Related to Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module added
#12
Updated by Ben Gadd almost 2 years ago
- Related to deleted (Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module )
#13
Updated by Ben Gadd almost 2 years ago
- Precedes Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module added
#14
Updated by Ben Gadd almost 2 years ago
- Precedes deleted (Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module )
#15
Updated by Ben Gadd almost 2 years ago
- Related to Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module added
#16
Updated by Dru Lavigne almost 2 years ago
- Status changed from Investigation to 15
James: where you able to update, and if so, does the issue persist in 11.1? If it does, please attach a new debug from the 11.1 system.
#17
Updated by James Thompson almost 2 years ago
- File freenas.error.01.PNG freenas.error.01.PNG added
- File freenas.error.02.PNG freenas.error.02.PNG added
Sorry for the delay Dru.
I did some digging and I wish I could say I had better results. I did clear up a few things though.
First - As noted before not all of the shares are experiencing this issue. Only the first four of eight.
Second - I believe I lost a bit of timeline while going through my troubleshooting previously. The remaining four shares had the correct vfs_objects attached and did not need to be reapplied for them to work. They made the transition without issue.
Third - To the extent of my troubleshooting the end result is the same. Adding zfsacl
has thus far been my only successful resolution to getting the shares working.
I did try doing the update to 11.0 via ISO and saw a few warnings that could be normal. They came when the migration scripts were running. Likely nothing but I'll attach them just in case.
Per request I also attempted the update to 11.1 and the issue persists.
#18
Updated by James Thompson almost 2 years ago
- File debug.11_1.webguiupgrade.tgz added
- File debug.after11ISOupgrade.tgz added
- Private changed from No to Yes
I have a couple debugs here that I hope helps.
#19
Updated by James Thompson almost 2 years ago
- Private changed from Yes to No
#20
Updated by James Thompson almost 2 years ago
- Private changed from No to Yes
#22
Updated by Dru Lavigne almost 2 years ago
- Status changed from 15 to Unscreened
#23
Updated by Timur Bakeyev almost 2 years ago
- Status changed from Unscreened to Screened
#24
Updated by Timur Bakeyev almost 2 years ago
- File deleted (
debug.after11ISOupgrade.tgz)
#25
Updated by Timur Bakeyev almost 2 years ago
- File deleted (
debug.11_1.webguiupgrade.tgz)
#26
Updated by Timur Bakeyev almost 2 years ago
- Status changed from Screened to 15
Hi, James!
If you could switch back to 9.10(or where it was before 11.0) and take a dump of the config DB, well, at least sharing
part - it could be helpful to restore the sequence of actions that lead to such a result. In case you'd like to do so, here is the command:
# sqlite3 /data/freenas-v1.db "select * from sharing_cifs_share"
Or, maybe even better:
# sqlite3 /data/freenas-v1.db ".dump sharing_cifs_share"
We are trying to reproduce this issue in out environment, but it could be that there is something special in your particular configuration.
#27
Updated by James Thompson almost 2 years ago
Sure thing. Here is the output both commands generated on 9.10.2-U1
[root@galaxy] ~# sqlite3 /data/freenas-v1.db "select * from sharing_cifs_share" /mnt/vol1/documents|1||documents|0|1|0||0|||0||0|0|1|1 /mnt/vol1/backups|1||backups|0|2|0||0|||0||0|0|2|1 /mnt/vol1/media|1||media|0|3|0||0|||0||0|0|3|1 /mnt/vol1/vhd|1||vhd|0|7|0||0|||0||0|0|4|1 /mnt/vol1/backups/lauri|1||lauri|0|2|0||0|||0||0|0|5|1 /mnt/vol1/userdata|1||userdata|0|8|0||0||windows ad datastore|0|aio_pthread,streams_xattr|1|0|6|1 /mnt/vol1/software|1||software|0||0||0|||0|aio_pthread,streams_xattr|0|0|7|1 /mnt/vol1/pictures|1||pictures|0||0||0|||0|aio_pthread,streams_xattr|0|0|8|1 [root@galaxy] ~# sqlite3 /data/freenas-v1.db ".dump sharing_cifs_share" PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE "sharing_cifs_share" ("cifs_path" varchar(255), "cifs_default_permissions" bool NOT NULL, "cifs_hostsallow" text NOT NULL, "cifs_name" varchar(120) NOT NULL, "cifs_guestok" bool NOT NULL, "cifs_storage_task_id" integer NULL, "cifs_showhiddenfiles" bool NOT NULL, "cifs_hostsdeny" text NOT NULL, "cifs_ro" bool NOT NULL, "cifs_auxsmbconf" text NOT NULL, "cifs_comment" varchar(120) NOT NULL, "cifs_home" bool NOT NULL, "cifs_vfsobjects" varchar(255) NOT NULL, "cifs_recyclebin" bool NOT NULL, "cifs_guestonly" bool NOT NULL, "id" integer PRIMARY KEY, "cifs_browsable" bool NOT NULL); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/documents',1,'','documents',0,1,0,'',0,'','',0,'',0,0,1,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/backups',1,'','backups',0,2,0,'',0,'','',0,'',0,0,2,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/media',1,'','media',0,3,0,'',0,'','',0,'',0,0,3,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/vhd',1,'','vhd',0,7,0,'',0,'','',0,'',0,0,4,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/backups/lauri',1,'','lauri',0,2,0,'',0,'','',0,'',0,0,5,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/userdata',1,'','userdata',0,8,0,'',0,'','windows ad datastore',0,'aio_pthread,streams_xattr',1,0,6,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/software',1,'','software',0,NULL,0,'',0,'','',0,'aio_pthread,streams_xattr',0,0,7,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/pictures',1,'','pictures',0,NULL,0,'',0,'','',0,'aio_pthread,streams_xattr',0,0,8,1); COMMIT;
#28
Updated by Timur Bakeyev almost 2 years ago
Great, thanks a lot!
/mnt/vol1/documents|1||documents|0|1|0||0|||0||0|0|1|1 /mnt/vol1/backups|1||backups|0|2|0||0|||0||0|0|2|1 /mnt/vol1/media|1||media|0|3|0||0|||0||0|0|3|1 /mnt/vol1/vhd|1||vhd|0|7|0||0|||0||0|0|4|1 /mnt/vol1/backups/lauri|1||lauri|0|2|0||0|||0||0|0|5|1 /mnt/vol1/userdata|1||userdata|0|8|0||0||windows ad datastore|0|aio_pthread,streams_xattr|1|0|6|1 /mnt/vol1/software|1||software|0||0||0|||0|aio_pthread,streams_xattr|0|0|7|1 /mnt/vol1/pictures|1||pictures|0||0||0|||0|aio_pthread,streams_xattr|0|0|8|1
I think that nails it. Shares, which didn't have anything in the vfs_objects field were not upgraded to get zfsacl module.
Thanks a lot, again, for the provided information, that helped us to locate the problem.
At this point, I guess, it's OK to close the ticket?
#29
Updated by James Thompson almost 2 years ago
Just to clarify would this be considered expected behavior?
#30
Updated by Timur Bakeyev almost 2 years ago
I would define it as a bug in the migration process, so it's going to be fixed in 11.1-U1.
#31
Updated by James Thompson almost 2 years ago
Got it thanks. Just to add to this - in the 9.10.2-U1 dump I pulled. Under CIFS it shows zfsacl and zfs_space for vfs objects
Example¶
[backups] path = /mnt/vol1/backups printable = no veto files = /.snapshot/.windows/.mac/.zfs/ writeable = yes browseable = yes shadow:snapdir = .zfs/snapshot shadow:sort = desc shadow:localtime = yes shadow:format = auto-%Y%m%d.%H%M-1d shadow:snapdirseverywhere = yes vfs objects = shadow_copy2 zfs_space zfsacl hide dot files = yes guest ok = no nfs4:mode = special nfs4:acedup = merge nfs4:chown = true zfsacl:acesort = dontcare
#32
Updated by Dru Lavigne almost 2 years ago
- Status changed from 15 to Screened
#33
Updated by Timur Bakeyev almost 2 years ago
- Status changed from Screened to Fix In Progress
#34
Updated by Timur Bakeyev almost 2 years ago
- Target version changed from 11.1-U1 to 11.1-U2
#35
Updated by Dru Lavigne almost 2 years ago
- Status changed from Fix In Progress to Not Started
#36
Updated by Ben Gadd almost 2 years ago
- Due date set to 02/12/2018
Due date updated to reflect the code freeze for 11.1U2.
#37
Updated by Ben Gadd almost 2 years ago
- Severity set to New
#38
Updated by Timur Bakeyev almost 2 years ago
- Status changed from Not Started to In Progress
#39
Updated by Timur Bakeyev almost 2 years ago
Master:
https://github.com/freenas/freenas/pull/827
11.1-stable
https://github.com/freenas/freenas/pull/828
11.0-stable
https://github.com/freenas/freenas/pull/829
truenas-11.0-stable
https://github.com/freenas/freenas/pull/830
#40
Updated by Timur Bakeyev almost 2 years ago
- Related to Bug #23439: ZFS VFS objects are optional added
#41
Updated by Kris Moore almost 2 years ago
- Status changed from In Progress to Done
- Needs Doc changed from Yes to No
- Needs Merging changed from Yes to No
#42
Updated by Dru Lavigne almost 2 years ago
- Subject changed from vfs_objects being dropped while upgrading from 9.10-U1 to 11.0-U4 yielding windows error "The Parameter is incorrect" to Add zfs_space and zfsacl as default modules to VFS objects
- Private changed from Yes to No