Project

General

Profile

Bug #76971

Run ZFS operations in a process pool

Added by Joshua Sirrine over 1 year ago. Updated over 1 year ago.

Status:
Done
Priority:
Blocks Until Resolved
Assignee:
William Grzybowski
Category:
Middleware
Seen in:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
Yes

Description

Tried to create a periodic snapshot task, and could no create it. After clicking save, the WebGUI rejects the dataset chosen saying "Select a valid choice. tank/Backups/Dropbox is not one of the available choices." This is a bit whimsical since it is a dropdown.

I can reproduce this trivially. If a dev would like to see and debug this on my personal FreeNAS, have them contact me via Ryver.

2019-02-24_00h03_23.jpg (147 KB) 2019-02-24_00h03_23.jpg Joshua Sirrine, 02/23/2019 11:04 PM
2019-03-05_17h15_35.jpg (201 KB) 2019-03-05_17h15_35.jpg Joshua Sirrine, 03/05/2019 04:18 PM
Pasted Image- Mar 21, 2019 - 6-02-59pm.png (46.3 KB) Pasted Image- Mar 21, 2019 - 6-02-59pm.png Screenshot of traceback from autorepl.py Joe Maloney, 03/21/2019 06:23 PM
55795
58764
62331

Related issues

Copied to FreeNAS - Bug #83908: Run ZFS operations in a process pool Done

Associated revisions

Revision 9292732e (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/utils): method to get attributes from a filter list Ticket: #76971

Revision efa67bc4 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/zfs): use zfs(8) to get datasets names Ticket: #76971

Revision ad45c549 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): load all plugins on workers so we can resolve schemas Ticket: #76971

Revision 56321295 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/zfs): run ZFS operations in a process pool py-libzfs is not thread-safe which means we were blocking at one ZFS operation. e.g. say snapshots were listing, no other operations could run. Now we are only limited by the size of the process pool. Ticket: #76971

Revision 6fcbb7ee (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): load all plugins on workers so we can resolve schemas Ticket: #76971

Revision ee8a95ec (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/zfs): run ZFS operations in a process pool py-libzfs is not thread-safe which means we were blocking at one ZFS operation. e.g. say snapshots were listing, no other operations could run. Now we are only limited by the size of the process pool. Ticket: #76971

Revision f07104c1 (diff)
Added by William Grzybowski over 1 year ago

Remove unused modules Ticket: #76971

Revision bd2c56c5 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/utils): method to get attributes from a filter list Ticket: #76971

Revision 086b21c1 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/zfs): use zfs(8) to get datasets names Ticket: #76971

Revision 7b0489f9 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): load all plugins on workers so we can resolve schemas Ticket: #76971

Revision 0d199b56 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/zfs): run ZFS operations in a process pool py-libzfs is not thread-safe which means we were blocking at one ZFS operation. e.g. say snapshots were listing, no other operations could run. Now we are only limited by the size of the process pool. Ticket: #76971

Revision 8c6e7d6e (diff)
Added by William Grzybowski over 1 year ago

Remove unused modules Ticket: #76971

Revision b324a75c (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/utils): method to get attributes from a filter list Ticket: #76971 (cherry picked from commit 9292732e33114e9a4eaf0855aee2f9d711c91c32)

Revision 9120fe70 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/zfs): use zfs(8) to get datasets names Ticket: #76971

Revision 78a9820f (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): load all plugins on workers so we can resolve schemas Ticket: #76971

Revision dbb1c82a (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/zfs): run ZFS operations in a process pool py-libzfs is not thread-safe which means we were blocking at one ZFS operation. e.g. say snapshots were listing, no other operations could run. Now we are only limited by the size of the process pool. Ticket: #76971

Revision af10ec57 (diff)
Added by William Grzybowski over 1 year ago

Remove unused modules Ticket: #76971

History

#1 Updated by Dru Lavigne over 1 year ago

  • Assignee changed from Release Council to Erin Clark
  • Target version changed from 11.2-U3 to 11.2-U4

#3 Updated by Dru Lavigne over 1 year ago

Joshua: Is this reproducible in the legacy UI as well?

#8 Updated by Erin Clark over 1 year ago

That problem also doesn't happen on my box, I can select datasets from both of my pools in old UI

#9 Updated by Joe Maloney over 1 year ago

I've created 1000 snaps, and 20 datasets per pool with 11.2-U3. I have also tried to create "temptank/Backups/Dropbox" and select that with the snapshot task. With that I have not been able to reproduce. I will now go back, and test 11.2-U2 since that was the version reported here to determine if this has already somehow been fixed.

#10 Updated by Joe Maloney over 1 year ago

  • File debug-fn07-20190321164241.tgz added
  • File debug-freenas-20190321164645.tgz added

Also tested 11.2 and cannot reproduce. Attached are debugs from the systems I attempted.

I wrote the following script which did produce a hang in listing operations while the data set, and snapshot creation tasks where underway.

#!/bin/sh

if [ -z $1 ] ; then
  echo "Please provide pool name!" 
  exit
fi

pool=$1

# Create 20 datasets
for i in `seq 1 1000`; do
   zfs create ${pool}/$i
done

# Create 1000 snapshots
for i in `seq 1 10000`; do
    zfs snapshot ${pool}@$i
done

I did crash middleware once on an encrypted volume while trying multiple operations to ddos the machine at the same time. However with single operations to create datasets, and snapshots in batch the UI became responsive again.

I also created a dataset with an extremely long name like this:

``

I was of course given warning about character limit so i shortened to the limits, and I was still able to create the perodic snap task successfully using both django, and angular UI.

#11 Updated by Joe Maloney over 1 year ago

Just to clarify when I said I confirmed this affects new and old GUI I confirmed based on the screenshots Josh Sirrine provided. I need to see the issue in his environment to proceed further.

#12 Updated by Joe Maloney over 1 year ago

62331

I found the issue on a call with Josh today. Let me know if you need a system setup for reproduction. Attached is an image showing autorepl.py trying to run and giving a traceback when a periodic snapshot is created. This is caused when a destination replication server goes offline. As a result of a replication server going offline no new periodic snapshots cannot be created. That is the bug we see here.

To reproduce:
Add a periodic snapshot
Add a replication task to a remote server
Take the remote server offline
Try to add another periodic snapshot task

#13 Updated by William Grzybowski over 1 year ago

Joshua, whats the output of "midclt call pool.filesystem_choices", while presenting the problem?

Also please attach a Save Debug.

#14 Updated by William Grzybowski over 1 year ago

  • Status changed from Unscreened to Blocked
  • Severity changed from New to Medium
  • Reason for Blocked set to Need additional information from Author

#20 Updated by Bug Clerk over 1 year ago

  • Status changed from Blocked to In Progress

#21 Updated by Bug Clerk over 1 year ago

  • Status changed from In Progress to Ready for Testing

#22 Updated by Bug Clerk over 1 year ago

  • Target version changed from 11.2-U4 to 11.3-BETA1

#23 Updated by Bug Clerk over 1 year ago

  • Copied to Bug #83908: Run ZFS operations in a process pool added

#24 Updated by Dru Lavigne over 1 year ago

  • File deleted (debug-fn07-20190321164241.tgz)

#25 Updated by Dru Lavigne over 1 year ago

  • File deleted (debug-freenas-20190321164645.tgz)

#26 Updated by Dru Lavigne over 1 year ago

  • File deleted (debug.txt)

#27 Updated by Dru Lavigne over 1 year ago

  • Subject changed from Periodic Snapshot task cannot be created - Dropdown is identified as "invalid" to Run ZFS operations in a process pool
  • Private changed from Yes to No
  • Reason for Blocked deleted (Need additional information from Author)
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#28 Updated by Dru Lavigne over 1 year ago

  • Status changed from Ready for Testing to Done
  • Target version changed from 11.3-BETA1 to Master - FreeNAS Nightlies
  • Needs QA changed from Yes to No

Also available in: Atom PDF