Add zfs_space and zfsacl as default modules to VFS objects
Decided to upgrade my production FreeNAS-9.10.2-U1 (86c7ef5) box to the latest FreeNAS-11.0-U4 (54848d13b) via GUI. The upgrade was successful with no noticeable errors. All services started and it joined to Active Directory successfully. Shares to ESXi via NFS and some SMB shares issued via folder redirection to Windows machines from Active Directory worked as expected. Two shares however presented me with an "Access Denied" error message. Checking permissions via Windows Server 2012, Windows 10 and Windows 7 machines showed missing permissions. Only the owner and owner group were listed with all permissions missing. A check beside the "Special Permissions" was the only box checked and it was grayed out (See attachment 01.jpg). Continuing to 'Advanced' would show the owners correct permissions (See attachment 02.jpg). Any attempt to change permissions or add a user threw an error "Unable to save permissions...The parameter is incorrect." (See attachment 03.jpg which shows an attempt to add a new group.) There were several other folders that continued to work, checking permissions on those showed similar problems to the ones not accessible. It would seem that the permissions were still there just not registering on Windows Machines. This was confirmed by running
getfacl <share> which listed all permissions as expected.
- Windows Active Directory environment forest level 2012.
- Multiple versions of Windows clients from Win7-Win10.
- FreeNAS Server AMD Opteron 2419EE 32G RAM 8Tib raidz2
- Starting with FreeNAS I first rebooted the system twice watching for any errors. The only thing I noticed was this bit:
@winbindd not running? (check /var/run/samba/winbindd.pid) smbd not running? (check /var/run/samba/smbd.pid) nmbd not running? (check /var/run/samba/smbd.pid)
- Checking each of those files showed unique PID's in each file. This is likely irrelevant.
- I verified that Active Directory was connected by the usual commands.
[root@host] ~# wbinfo -t checking the trust secret for domain <Sanitized-Domain> via RPC calls succeeded
[root@host] ~# wbinfo -u administrator guest krbtgt #Sanitized users....
[root@host] ~# wbinfo -g winrmremotewmiusers__ dhcp users dhcp administrators domain computers #Sanitized groups....
- Also attempted setting ACL's with
setfaclwhich would successfully change permissions however Windows machines did not recognize the change and continued presenting the same errors.
- Under the SMB properties I added
ntlm auth = yesand restarted. No change.
- Added additional perameters from another post with similar issues.
lanman auth = no ntlm auth = yes client NTLMv2 = yes client lanman auth = no client plaintext = no
- After the reboot SMB failed the sanity check. Removed those and restarted and sanity check passed.
- Windows Server 2012 released KB3041857 for an issue with network resources that would give this error. However this was already installed during a monthly roll-up. Uninstalling this patch and reinstalling did not change anything.
- Tried to take ownership of the share but it gave the same error and would not let you proceed.
- Tried removing the share completely and re-adding it.
net use /deleteand also flushed DNS
- Verified group policies permitted the shares and tested other local shares from another Windows Server instance.
- Rolled back to 9.10.2-U1 and all shares and permissions were working as expected.
- After the roll back to 9.10.2-U1 I verified all SMB configs and tried 11.0-U4 again.
- After the reboot the same symptoms presented themselves on the same shares.
- Removed one of the SMB shares and re-added it. This resolved my issue.
- Comparing previous config to the new config generated all VFS Objects had been cleared (See attachment 04.png).
- If you create a new share the following objects are automatically added:
zfs_space zfsacl streams_xattr aio_pthread
zfsaclto the impacted shares allowed all permissions to be accessible and changeable again in Windows (obviously!)
During my troubleshooting I saw that sometime after 9.10-U1 there was a Samba update from 4.1 to 4.3.4. Because I made such a big jump this could have been present before FreeNAS 11. I checked the upgrade notes for several of the versions in between the two releases and saw no mention of having to re-add zfsacl's.
Also, I have debug dumps from before and after the upgrade if someone could provide a secure way of sending those I would be happy to provide them.
#1 Updated by James Thompson about 3 years ago
Checking into this further it appears that 9.10U1 did not include zfs_space & zfsacl VFS objects in the drop-down menu as they are "always enabled." However 11.0 has added them to the menu and was removed from the note as an "Always enabled" object under table 10.4.2 in the User Guide under Sharing > SMB.
#8 Updated by James Thompson about 3 years ago
/data/update.failed does not exist. I am going to attempt an upgrade again either tonight or tomorrow night and pull the debugs again.
When reading AD and ldap dumps I noticed that a few of the shares had
zfs_space zfsacl on them. These shares were the ones that allowed me to open them but they did not allow me to edit the ACL's. After I went through and applied
zfs_space zfsacl to them I verified all shares showed ACL's and allowed me to edit. I'm curious why if
zfs_space zfsacl was indeed listed on the shares why it did not 'register' until after re-applying them.
When I go through it again I'm hoping to have a better understanding of this. While I'm doing this is there anything you all would like me to look for?
#17 Updated by James Thompson about 3 years ago
- File freenas.error.01.PNG freenas.error.01.PNG added
- File freenas.error.02.PNG freenas.error.02.PNG added
Sorry for the delay Dru.
I did some digging and I wish I could say I had better results. I did clear up a few things though.
First - As noted before not all of the shares are experiencing this issue. Only the first four of eight.
Second - I believe I lost a bit of timeline while going through my troubleshooting previously. The remaining four shares had the correct vfs_objects attached and did not need to be reapplied for them to work. They made the transition without issue.
Third - To the extent of my troubleshooting the end result is the same. Adding
zfsacl has thus far been my only successful resolution to getting the shares working.
I did try doing the update to 11.0 via ISO and saw a few warnings that could be normal. They came when the migration scripts were running. Likely nothing but I'll attach them just in case.
Per request I also attempted the update to 11.1 and the issue persists.
#26 Updated by Timur Bakeyev about 3 years ago
- Status changed from Screened to 15
If you could switch back to 9.10(or where it was before 11.0) and take a dump of the config DB, well, at least
sharing part - it could be helpful to restore the sequence of actions that lead to such a result. In case you'd like to do so, here is the command:
# sqlite3 /data/freenas-v1.db "select * from sharing_cifs_share"
Or, maybe even better:
# sqlite3 /data/freenas-v1.db ".dump sharing_cifs_share"
We are trying to reproduce this issue in out environment, but it could be that there is something special in your particular configuration.
#27 Updated by James Thompson about 3 years ago
Sure thing. Here is the output both commands generated on 9.10.2-U1
[root@galaxy] ~# sqlite3 /data/freenas-v1.db "select * from sharing_cifs_share" /mnt/vol1/documents|1||documents|0|1|0||0|||0||0|0|1|1 /mnt/vol1/backups|1||backups|0|2|0||0|||0||0|0|2|1 /mnt/vol1/media|1||media|0|3|0||0|||0||0|0|3|1 /mnt/vol1/vhd|1||vhd|0|7|0||0|||0||0|0|4|1 /mnt/vol1/backups/lauri|1||lauri|0|2|0||0|||0||0|0|5|1 /mnt/vol1/userdata|1||userdata|0|8|0||0||windows ad datastore|0|aio_pthread,streams_xattr|1|0|6|1 /mnt/vol1/software|1||software|0||0||0|||0|aio_pthread,streams_xattr|0|0|7|1 /mnt/vol1/pictures|1||pictures|0||0||0|||0|aio_pthread,streams_xattr|0|0|8|1 [root@galaxy] ~# sqlite3 /data/freenas-v1.db ".dump sharing_cifs_share" PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE "sharing_cifs_share" ("cifs_path" varchar(255), "cifs_default_permissions" bool NOT NULL, "cifs_hostsallow" text NOT NULL, "cifs_name" varchar(120) NOT NULL, "cifs_guestok" bool NOT NULL, "cifs_storage_task_id" integer NULL, "cifs_showhiddenfiles" bool NOT NULL, "cifs_hostsdeny" text NOT NULL, "cifs_ro" bool NOT NULL, "cifs_auxsmbconf" text NOT NULL, "cifs_comment" varchar(120) NOT NULL, "cifs_home" bool NOT NULL, "cifs_vfsobjects" varchar(255) NOT NULL, "cifs_recyclebin" bool NOT NULL, "cifs_guestonly" bool NOT NULL, "id" integer PRIMARY KEY, "cifs_browsable" bool NOT NULL); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/documents',1,'','documents',0,1,0,'',0,'','',0,'',0,0,1,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/backups',1,'','backups',0,2,0,'',0,'','',0,'',0,0,2,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/media',1,'','media',0,3,0,'',0,'','',0,'',0,0,3,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/vhd',1,'','vhd',0,7,0,'',0,'','',0,'',0,0,4,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/backups/lauri',1,'','lauri',0,2,0,'',0,'','',0,'',0,0,5,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/userdata',1,'','userdata',0,8,0,'',0,'','windows ad datastore',0,'aio_pthread,streams_xattr',1,0,6,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/software',1,'','software',0,NULL,0,'',0,'','',0,'aio_pthread,streams_xattr',0,0,7,1); INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/pictures',1,'','pictures',0,NULL,0,'',0,'','',0,'aio_pthread,streams_xattr',0,0,8,1); COMMIT;
#28 Updated by Timur Bakeyev about 3 years ago
Great, thanks a lot!
/mnt/vol1/documents|1||documents|0|1|0||0|||0||0|0|1|1 /mnt/vol1/backups|1||backups|0|2|0||0|||0||0|0|2|1 /mnt/vol1/media|1||media|0|3|0||0|||0||0|0|3|1 /mnt/vol1/vhd|1||vhd|0|7|0||0|||0||0|0|4|1 /mnt/vol1/backups/lauri|1||lauri|0|2|0||0|||0||0|0|5|1 /mnt/vol1/userdata|1||userdata|0|8|0||0||windows ad datastore|0|aio_pthread,streams_xattr|1|0|6|1 /mnt/vol1/software|1||software|0||0||0|||0|aio_pthread,streams_xattr|0|0|7|1 /mnt/vol1/pictures|1||pictures|0||0||0|||0|aio_pthread,streams_xattr|0|0|8|1
I think that nails it. Shares, which didn't have anything in the vfs_objects field were not upgraded to get zfsacl module.
Thanks a lot, again, for the provided information, that helped us to locate the problem.
At this point, I guess, it's OK to close the ticket?
#31 Updated by James Thompson about 3 years ago
Got it thanks. Just to add to this - in the 9.10.2-U1 dump I pulled. Under CIFS it shows zfsacl and zfs_space for
[backups] path = /mnt/vol1/backups printable = no veto files = /.snapshot/.windows/.mac/.zfs/ writeable = yes browseable = yes shadow:snapdir = .zfs/snapshot shadow:sort = desc shadow:localtime = yes shadow:format = auto-%Y%m%d.%H%M-1d shadow:snapdirseverywhere = yes vfs objects = shadow_copy2 zfs_space zfsacl hide dot files = yes guest ok = no nfs4:mode = special nfs4:acedup = merge nfs4:chown = true zfsacl:acesort = dontcare