Bug #26650
Correctly report ZFS dataset quota overflows
Description
When writing to a particular ZFS dataset and that dataset being below quota when the files are initially opened for write, but reaching its quota while the clients are actively writing, eventually something in zfs slows to a crawl.
This is trivially reproducible. Running TrueNAS 11.0-U4
1.zfs create cargo/quota-test
2.zfs set compression=off cargo/quota-test
3.zfs set refquota=50m cargo/quota-test
4.cd /mnt/cargo/quota-test
5.dd if=/dev/zero of=file.out bs=1m
If you perform these steps, you will notice that the "dd" command will continue issuing writes to the dataset but will never stop even though the quota has been met. I have procstat -kk -a | fgrep dd and attached the output to this ticket.
NOTE I have tried this on proper freeBSD 11.0, 11.1, and CURRENT and all produce the same results.
Associated revisions
History
#1
Updated by Alexander Motin about 3 years ago
- Subject changed from zfs refquota deadlock to ZFS dataset quota overflows not reported correctly
- Status changed from Unscreened to Fix In Progress
- Priority changed from Important to Critical
- Target version set to TrueNAS 11.1-U1
Investigation of the problem brought me to FreeBSD-specific change r298105 by avg@ on 2016-04-16. If quota overflow detected during write, the write will fail, but the error status can be lost, falsely reporting partial completion. As result written data are flowing to nowhere and indefinitely, as fast as CPU can handle the loop.
#2
Updated by Alexander Motin about 3 years ago
- Project changed from TrueNAS to FreeNAS
- Category changed from 162 to 129
- Target version changed from TrueNAS 11.1-U1 to 11.1
- Migration Needed deleted (
No) - Support Department Priority deleted (
0)
This is not specific to TrueNAS.
#3
Updated by Alexander Motin about 3 years ago
- Status changed from Fix In Progress to 19
The patch is committed to FreeBSD head. Started FreeNAS test build for the pull request with Jenkins.
#4
Updated by Alexander Motin about 3 years ago
- Status changed from 19 to Ready For Release
#5
Updated by Dru Lavigne about 3 years ago
- Subject changed from ZFS dataset quota overflows not reported correctly to Correctly report ZFS dataset quota overflows
#6
Updated by Dru Lavigne about 3 years ago
- Target version changed from 11.1 to 11.1-RC2
#7
Updated by Nick Wolff about 3 years ago
- Needs QA changed from Yes to No
- QA Status Test Passes FreeNAS added
- QA Status deleted (
Not Tested)
Stress tested 20 minutes of looping 50mb writes with zero hangs. Passing test.
#8
Updated by Dru Lavigne about 3 years ago
- Target version changed from 11.1-RC2 to 11.1-RC3
#9
Updated by Dru Lavigne about 3 years ago
- Status changed from Ready For Release to Resolved