Project

General

Profile

Bug #26994

Add zfs_space and zfsacl as default modules to VFS objects

Added by James Thompson about 2 years ago. Updated almost 2 years ago.

Status:
Done
Priority:
No priority
Assignee:
Timur Bakeyev
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Description

Decided to upgrade my production FreeNAS-9.10.2-U1 (86c7ef5) box to the latest FreeNAS-11.0-U4 (54848d13b) via GUI. The upgrade was successful with no noticeable errors. All services started and it joined to Active Directory successfully. Shares to ESXi via NFS and some SMB shares issued via folder redirection to Windows machines from Active Directory worked as expected. Two shares however presented me with an "Access Denied" error message. Checking permissions via Windows Server 2012, Windows 10 and Windows 7 machines showed missing permissions. Only the owner and owner group were listed with all permissions missing. A check beside the "Special Permissions" was the only box checked and it was grayed out (See attachment 01.jpg). Continuing to 'Advanced' would show the owners correct permissions (See attachment 02.jpg). Any attempt to change permissions or add a user threw an error "Unable to save permissions...The parameter is incorrect." (See attachment 03.jpg which shows an attempt to add a new group.) There were several other folders that continued to work, checking permissions on those showed similar problems to the ones not accessible. It would seem that the permissions were still there just not registering on Windows Machines. This was confirmed by running getfacl <share> which listed all permissions as expected.

Setup

  • Windows Active Directory environment forest level 2012.
  • Multiple versions of Windows clients from Win7-Win10.
  • FreeNAS Server AMD Opteron 2419EE 32G RAM 8Tib raidz2

Troubleshooting

FreeNAS 11.0

  • Starting with FreeNAS I first rebooted the system twice watching for any errors. The only thing I noticed was this bit:
    @winbindd not running? (check /var/run/samba/winbindd.pid)
    smbd not running? (check /var/run/samba/smbd.pid)
    nmbd not running? (check /var/run/samba/smbd.pid)
  • Checking each of those files showed unique PID's in each file. This is likely irrelevant.
  • I verified that Active Directory was connected by the usual commands.
    [root@host] ~# wbinfo -t
    checking the trust secret for domain <Sanitized-Domain> via RPC calls succeeded

    [root@host] ~# wbinfo -u
    administrator
    guest
    krbtgt
    #Sanitized users....

    [root@host] ~# wbinfo -g
    winrmremotewmiusers__
    dhcp users
    dhcp administrators
    domain computers
    #Sanitized groups....
  • Also attempted setting ACL's with setfacl which would successfully change permissions however Windows machines did not recognize the change and continued presenting the same errors.
  • Under the SMB properties I added ntlm auth = yes and restarted. No change.
  • Added additional perameters from another post with similar issues.
    lanman auth = no
    ntlm auth = yes
    client NTLMv2 = yes
    client lanman auth = no
    client plaintext = no
  • After the reboot SMB failed the sanity check. Removed those and restarted and sanity check passed.

Windows

  • Windows Server 2012 released KB3041857 for an issue with network resources that would give this error. However this was already installed during a monthly roll-up. Uninstalling this patch and reinstalling did not change anything.
  • Tried to take ownership of the share but it gave the same error and would not let you proceed.
  • Tried removing the share completely and re-adding it.
  • net use /delete and also flushed DNS ipconfig /flushdns
  • Verified group policies permitted the shares and tested other local shares from another Windows Server instance.

FreeNAS 9.10
  • Rolled back to 9.10.2-U1 and all shares and permissions were working as expected.

Resolution

  • After the roll back to 9.10.2-U1 I verified all SMB configs and tried 11.0-U4 again.
  • After the reboot the same symptoms presented themselves on the same shares.
  • Removed one of the SMB shares and re-added it. This resolved my issue.
  • Comparing previous config to the new config generated all VFS Objects had been cleared (See attachment 04.png).
  • If you create a new share the following objects are automatically added:
    zfs_space
    zfsacl
    streams_xattr
    aio_pthread
  • Adding zfsacl to the impacted shares allowed all permissions to be accessible and changeable again in Windows (obviously!)

Misc

During my troubleshooting I saw that sometime after 9.10-U1 there was a Samba update from 4.1 to 4.3.4. Because I made such a big jump this could have been present before FreeNAS 11. I checked the upgrade notes for several of the versions in between the two releases and saw no mention of having to re-add zfsacl's.

Also, I have debug dumps from before and after the upgrade if someone could provide a secure way of sending those I would be happy to provide them.

01.jpg (255 KB) 01.jpg James Thompson, 12/03/2017 07:55 PM
02.jpg (353 KB) 02.jpg James Thompson, 12/03/2017 07:55 PM
04.PNG (5.43 KB) 04.PNG James Thompson, 12/03/2017 07:56 PM
05.PNG (6.7 KB) 05.PNG James Thompson, 12/03/2017 07:56 PM
03.jpg (395 KB) 03.jpg James Thompson, 12/03/2017 07:56 PM
freenas.error.01.PNG (230 KB) freenas.error.01.PNG James Thompson, 12/18/2017 09:26 PM
freenas.error.02.PNG (187 KB) freenas.error.02.PNG James Thompson, 12/18/2017 09:26 PM
13280
13281
13282
13283
13284
13475
13476

Related issues

Related to FreeNAS - Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module Closed: Duplicate2017-12-042017-12-04
Related to FreeNAS - Bug #23439: ZFS VFS objects are optionalResolved2017-04-18

Associated revisions

Revision 697a6a42 (diff)
Added by Timur I. Bakeyev almost 2 years ago

Always add 'zfs_space' and 'zfsacl' modules to vfsobjects if not present yet At this point we may have shares with empty vfsobjects, inherited from the FreeNAS 9.* and 10.*. We still want them to get default 'zfs_space' and 'zfsacl' modules. Ticket: #26994 (cherry picked from commit 94228b09efe22e26f59c04274476850c87dbe135) Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>

Revision 95cd45f3 (diff)
Added by Timur I. Bakeyev almost 2 years ago

Always add 'zfs_space' and 'zfsacl' modules to vfsobjects if not present yet At this point we may have shares with empty vfsobjects, inherited from the FreeNAS 9.* and 10.*. We still want them to get default 'zfs_space' and 'zfsacl' modules. Ticket: #26994 (cherry picked from commit 94228b09efe22e26f59c04274476850c87dbe135) Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>

Revision b750fbf9 (diff)
Added by Timur I. Bakeyev almost 2 years ago

Always add 'zfs_space' and 'zfsacl' modules to vfsobjects if not present yet At this point we may have shares with empty vfsobjects, inherited from the FreeNAS 9.* and 10.*. We still want them to get default 'zfs_space' and 'zfsacl' modules. Ticket: #26994 (cherry picked from commit 94228b09efe22e26f59c04274476850c87dbe135) Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>

History

#1 Updated by James Thompson about 2 years ago

Checking into this further it appears that 9.10U1 did not include zfs_space & zfsacl VFS objects in the drop-down menu as they are "always enabled." However 11.0 has added them to the menu and was removed from the note as an "Always enabled" object under table 10.4.2 in the User Guide under Sharing > SMB.

#2 Updated by Dru Lavigne about 2 years ago

  • Status changed from Unscreened to 15

James, does enabling those resolve this for you? If not, please attach a debug (System -> Advanced -> Save Debug).

#3 Updated by James Thompson about 2 years ago

Yes. Adding zfsacl resolved the issue. I took it a bit further and matched my existing shares to a newly created share and added:

zfs_space
zfsacl
streams_xattr
aio_pthread

#4 Updated by Dru Lavigne about 2 years ago

  • Status changed from 15 to Unscreened
  • Assignee changed from Release Council to John Hixson

John: should there be a migration for this?

#5 Updated by John Hixson about 2 years ago

  • Status changed from Unscreened to Screened
  • Target version set to 11.1-U1

#6 Updated by John Hixson about 2 years ago

Dru Lavigne wrote:

John: should there be a migration for this?

There is a migration for this already. I'm not sure what went wrong here, but it seems as though the migration (and perhaps others) did not run. Can you see if /data/update.failed exists?

#7 Updated by John Hixson about 2 years ago

  • Status changed from Screened to 15

#8 Updated by James Thompson about 2 years ago

/data/update.failed does not exist. I am going to attempt an upgrade again either tonight or tomorrow night and pull the debugs again.

When reading AD and ldap dumps I noticed that a few of the shares had zfs_space zfsacl on them. These shares were the ones that allowed me to open them but they did not allow me to edit the ACL's. After I went through and applied zfs_space zfsacl to them I verified all shares showed ACL's and allowed me to edit. I'm curious why if zfs_space zfsacl was indeed listed on the shares why it did not 'register' until after re-applying them.

When I go through it again I'm hoping to have a better understanding of this. While I'm doing this is there anything you all would like me to look for?

#9 Updated by Dru Lavigne about 2 years ago

  • Status changed from 15 to Investigation

#10 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Assignee changed from John Hixson to Timur Bakeyev

#11 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Related to Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module added

#12 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Related to deleted (Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module )

#13 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Precedes Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module added

#14 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Precedes deleted (Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module )

#15 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Related to Bug #25843: SMBD is crashing due missing 'vfs_zfsacl' module added

#16 Updated by Dru Lavigne almost 2 years ago

  • Status changed from Investigation to 15

James: where you able to update, and if so, does the issue persist in 11.1? If it does, please attach a new debug from the 11.1 system.

#17 Updated by James Thompson almost 2 years ago

13475
13476

Sorry for the delay Dru.

I did some digging and I wish I could say I had better results. I did clear up a few things though.

First - As noted before not all of the shares are experiencing this issue. Only the first four of eight.

Second - I believe I lost a bit of timeline while going through my troubleshooting previously. The remaining four shares had the correct vfs_objects attached and did not need to be reapplied for them to work. They made the transition without issue.

Third - To the extent of my troubleshooting the end result is the same. Adding zfsacl has thus far been my only successful resolution to getting the shares working.

I did try doing the update to 11.0 via ISO and saw a few warnings that could be normal. They came when the migration scripts were running. Likely nothing but I'll attach them just in case.

Per request I also attempted the update to 11.1 and the issue persists.

#18 Updated by James Thompson almost 2 years ago

  • File debug.11_1.webguiupgrade.tgz added
  • File debug.after11ISOupgrade.tgz added
  • Private changed from No to Yes

I have a couple debugs here that I hope helps.

#19 Updated by James Thompson almost 2 years ago

  • Private changed from Yes to No

#20 Updated by James Thompson almost 2 years ago

  • Private changed from No to Yes

#22 Updated by Dru Lavigne almost 2 years ago

  • Status changed from 15 to Unscreened

#23 Updated by Timur Bakeyev almost 2 years ago

  • Status changed from Unscreened to Screened

#24 Updated by Timur Bakeyev almost 2 years ago

  • File deleted (debug.after11ISOupgrade.tgz)

#25 Updated by Timur Bakeyev almost 2 years ago

  • File deleted (debug.11_1.webguiupgrade.tgz)

#26 Updated by Timur Bakeyev almost 2 years ago

  • Status changed from Screened to 15

Hi, James!

If you could switch back to 9.10(or where it was before 11.0) and take a dump of the config DB, well, at least sharing part - it could be helpful to restore the sequence of actions that lead to such a result. In case you'd like to do so, here is the command:

# sqlite3 /data/freenas-v1.db "select * from sharing_cifs_share" 

Or, maybe even better:

# sqlite3 /data/freenas-v1.db ".dump sharing_cifs_share" 

We are trying to reproduce this issue in out environment, but it could be that there is something special in your particular configuration.

#27 Updated by James Thompson almost 2 years ago

Sure thing. Here is the output both commands generated on 9.10.2-U1

[root@galaxy] ~# sqlite3 /data/freenas-v1.db "select * from sharing_cifs_share" 
/mnt/vol1/documents|1||documents|0|1|0||0|||0||0|0|1|1
/mnt/vol1/backups|1||backups|0|2|0||0|||0||0|0|2|1
/mnt/vol1/media|1||media|0|3|0||0|||0||0|0|3|1
/mnt/vol1/vhd|1||vhd|0|7|0||0|||0||0|0|4|1
/mnt/vol1/backups/lauri|1||lauri|0|2|0||0|||0||0|0|5|1
/mnt/vol1/userdata|1||userdata|0|8|0||0||windows ad datastore|0|aio_pthread,streams_xattr|1|0|6|1
/mnt/vol1/software|1||software|0||0||0|||0|aio_pthread,streams_xattr|0|0|7|1
/mnt/vol1/pictures|1||pictures|0||0||0|||0|aio_pthread,streams_xattr|0|0|8|1

[root@galaxy] ~# sqlite3 /data/freenas-v1.db ".dump sharing_cifs_share" 
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE "sharing_cifs_share" ("cifs_path" varchar(255), "cifs_default_permissions" bool NOT NULL, "cifs_hostsallow" text NOT NULL, "cifs_name" varchar(120) NOT NULL, "cifs_guestok" bool NOT NULL, "cifs_storage_task_id" integer NULL, "cifs_showhiddenfiles" bool NOT NULL, "cifs_hostsdeny" text NOT NULL, "cifs_ro" bool NOT NULL, "cifs_auxsmbconf" text NOT NULL, "cifs_comment" varchar(120) NOT NULL, "cifs_home" bool NOT NULL, "cifs_vfsobjects" varchar(255) NOT NULL, "cifs_recyclebin" bool NOT NULL, "cifs_guestonly" bool NOT NULL, "id" integer PRIMARY KEY, "cifs_browsable" bool NOT NULL);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/documents',1,'','documents',0,1,0,'',0,'','',0,'',0,0,1,1);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/backups',1,'','backups',0,2,0,'',0,'','',0,'',0,0,2,1);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/media',1,'','media',0,3,0,'',0,'','',0,'',0,0,3,1);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/vhd',1,'','vhd',0,7,0,'',0,'','',0,'',0,0,4,1);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/backups/lauri',1,'','lauri',0,2,0,'',0,'','',0,'',0,0,5,1);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/userdata',1,'','userdata',0,8,0,'',0,'','windows ad datastore',0,'aio_pthread,streams_xattr',1,0,6,1);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/software',1,'','software',0,NULL,0,'',0,'','',0,'aio_pthread,streams_xattr',0,0,7,1);
INSERT INTO "sharing_cifs_share" VALUES('/mnt/vol1/pictures',1,'','pictures',0,NULL,0,'',0,'','',0,'aio_pthread,streams_xattr',0,0,8,1);
COMMIT;

#28 Updated by Timur Bakeyev almost 2 years ago

Great, thanks a lot!

/mnt/vol1/documents|1||documents|0|1|0||0|||0||0|0|1|1
/mnt/vol1/backups|1||backups|0|2|0||0|||0||0|0|2|1
/mnt/vol1/media|1||media|0|3|0||0|||0||0|0|3|1
/mnt/vol1/vhd|1||vhd|0|7|0||0|||0||0|0|4|1
/mnt/vol1/backups/lauri|1||lauri|0|2|0||0|||0||0|0|5|1
/mnt/vol1/userdata|1||userdata|0|8|0||0||windows ad datastore|0|aio_pthread,streams_xattr|1|0|6|1
/mnt/vol1/software|1||software|0||0||0|||0|aio_pthread,streams_xattr|0|0|7|1
/mnt/vol1/pictures|1||pictures|0||0||0|||0|aio_pthread,streams_xattr|0|0|8|1

I think that nails it. Shares, which didn't have anything in the vfs_objects field were not upgraded to get zfsacl module.

Thanks a lot, again, for the provided information, that helped us to locate the problem.

At this point, I guess, it's OK to close the ticket?

#29 Updated by James Thompson almost 2 years ago

Just to clarify would this be considered expected behavior?

#30 Updated by Timur Bakeyev almost 2 years ago

I would define it as a bug in the migration process, so it's going to be fixed in 11.1-U1.

#31 Updated by James Thompson almost 2 years ago

Got it thanks. Just to add to this - in the 9.10.2-U1 dump I pulled. Under CIFS it shows zfsacl and zfs_space for vfs objects

Example

[backups]
    path = /mnt/vol1/backups
    printable = no
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    shadow:snapdir = .zfs/snapshot
    shadow:sort = desc
    shadow:localtime = yes
    shadow:format = auto-%Y%m%d.%H%M-1d
    shadow:snapdirseverywhere = yes
    vfs objects = shadow_copy2 zfs_space zfsacl
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare

#32 Updated by Dru Lavigne almost 2 years ago

  • Status changed from 15 to Screened

#33 Updated by Timur Bakeyev almost 2 years ago

  • Status changed from Screened to Fix In Progress

#34 Updated by Timur Bakeyev almost 2 years ago

  • Target version changed from 11.1-U1 to 11.1-U2

#35 Updated by Dru Lavigne almost 2 years ago

  • Status changed from Fix In Progress to Not Started

#36 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Due date set to 02/12/2018

Due date updated to reflect the code freeze for 11.1U2.

#37 Avatar?id=13649&size=24x24 Updated by Ben Gadd almost 2 years ago

  • Severity set to New

#38 Updated by Timur Bakeyev almost 2 years ago

  • Status changed from Not Started to In Progress

#40 Updated by Timur Bakeyev almost 2 years ago

  • Related to Bug #23439: ZFS VFS objects are optional added

#41 Avatar?id=14398&size=24x24 Updated by Kris Moore almost 2 years ago

  • Status changed from In Progress to Done
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#42 Updated by Dru Lavigne almost 2 years ago

  • Subject changed from vfs_objects being dropped while upgrading from 9.10-U1 to 11.0-U4 yielding windows error "The Parameter is incorrect" to Add zfs_space and zfsacl as default modules to VFS objects
  • Private changed from Yes to No

Also available in: Atom PDF