Project

General

Profile

Feature #17923

Add dataset quota email notifications

Added by LT xda about 2 years ago. Updated 4 months ago.

Status:
Done
Priority:
Important
Assignee:
Vladimir Vinogradenko
Category:
Middleware
Target version:
Estimated time:
Sprint:
Severity:
New
Backlog Priority:
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Description

It would be great to have a feature that would notify an admin and user associated with a dataset of a quota issue. E-mail notification for reaching 80% of quota and then another e-mail notification for reaching quota.


Related issues

Copied to FreeNAS - Bug #34684: Fix quota exceeded emails being sent every minuteDone

Associated revisions

Revision f33a4108 (diff)
Added by William Grzybowski about 1 year ago

fix(middlewared/zfs): zvols do not have `quota` property

Ticket: #17923

Revision c0d81025 (diff)
Added by William Grzybowski about 1 year ago

fix(middlewared/zfs): zvols do not have `quota` property

Ticket: #17923

Revision 5d0aaec7 (diff)
Added by Dru Lavigne about 1 year ago

Add zvols to alert condition.
Ticket: #17923

Revision 2d83c750 (diff)
Added by Vladimir Vinogradenko 11 months ago

fix(zfs.quota): Notify refquota excess

Also properly handle legacy dataset mountpoint, not mounted datasets and any generic stat errors

Ticket: #17923

Revision 7de6f3fd (diff)
Added by Vladimir Vinogradenko 11 months ago

fix(zfs.quota): Notify refquota excess

Also properly handle legacy dataset mountpoint, not mounted datasets and any generic stat errors

Ticket: #17923

Revision e91f5174 (diff)
Added by Vladimir Vinogradenko 4 months ago

fix(zfs): Use datastore.delete instead of datastore.sql for HA systems

Ticket: #17923

Revision de1612dd (diff)
Added by Vladimir Vinogradenko 4 months ago

Do not run quota excess checks on BACKUP systems

Ticket: #17923

Revision 58d09893 (diff)
Added by Vladimir Vinogradenko 4 months ago

fix(zfs): Use datastore.delete instead of datastore.sql for HA systems

Ticket: #17923

Revision 5c9b3d62 (diff)
Added by Vladimir Vinogradenko 4 months ago

Do not run quota excess checks on BACKUP systems

Ticket: #17923

Revision 78fd72dd (diff)
Added by Vladimir Vinogradenko 4 months ago

Do not run quota excess checks on BACKUP systems

Ticket: #17923

Revision 3a181e65 (diff)
Added by Vladimir Vinogradenko 4 months ago

Do not run quota excess checks on BACKUP systems

Ticket: #17923

History

#1 Updated by Stefan Schroepfer about 2 years ago

In addition sending a notification if a dataset's value of the USED property reaches AVAIL might be even more useful. In this way a limit imposed by reservations on other datasets in the same pool would be taken into account, too.

#2 Avatar?id=14398&size=24x24 Updated by Kris Moore about 2 years ago

  • Assignee changed from LT xda to Suraj Ravichandran
  • Priority changed from No priority to Nice to have
  • Target version changed from Unspecified to 9.10.3

#3 Avatar?id=14398&size=24x24 Updated by Kris Moore about 2 years ago

  • Priority changed from Nice to have to Important
  • Target version changed from 9.10.3 to 9.10.2

BRB: We would love this feature to land in TrueNAS 9.10.2 as well, so lets see if we can make this a priority.

#4 Updated by Suraj Ravichandran almost 2 years ago

  • Status changed from Unscreened to Screened

#5 Updated by Vaibhav Chauhan almost 2 years ago

suraj can you look at it ?

#6 Updated by Suraj Ravichandran almost 2 years ago

yes after the demo today I will

#7 Updated by Suraj Ravichandran almost 2 years ago

@release engineer and BRB: This is my last 9.10.2 ticket, I will try to get this in today (Monday, the 5th of December) by midnight.

#8 Avatar?id=14398&size=24x24 Updated by Kris Moore almost 2 years ago

  • Target version changed from 9.10.2 to 9.10.3

#9 Updated by Suraj Ravichandran almost 2 years ago

Sorry for not updating this ticket post my last comment.

We discussed this ticket at the time of that release (9.10.2) and decided to punt it forward.

There are some more issues with this (which came up in during the above-mentioned discussion).

I will not be able to get to this till mid-February.

@Kris or @vaibhva please feel free to reassign to if you need it earlier than that.

#10 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Target version changed from 9.10.3 to 9.10.4

#11 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Target version changed from 9.10.4 to 11.1

#12 Updated by Suraj Ravichandran over 1 year ago

Just an update on this ticket, still the same status.

Wanted to put this out there so that people do not think that I forgot about it.

Will try to bring it up in the next meeting

#13 Updated by Dru Lavigne about 1 year ago

  • Assignee changed from Suraj Ravichandran to William Grzybowski

William: do we already have sufficient alerts for this? If not, please load balance between Vladimir and Nikola.

#14 Updated by William Grzybowski about 1 year ago

  • Status changed from Screened to Unscreened
  • Assignee changed from William Grzybowski to Vladimir Vinogradenko

Vladimir, I am thinking we may use devd events from ZFS (in middlewared) to trigger an alert. However I dont know if these events exist yet :). If not we may have to do periodic pulls.

#15 Updated by Vladimir Vinogradenko about 1 year ago

  • Status changed from Unscreened to Screened

#16 Updated by Vladimir Vinogradenko about 1 year ago

You are right, there are no events, doing polling approach

#17 Updated by Vladimir Vinogradenko about 1 year ago

  • Status changed from Screened to Fix In Progress

#18 Updated by Vladimir Vinogradenko about 1 year ago

  • Status changed from Fix In Progress to Needs Developer Review

#19 Updated by Vladimir Vinogradenko about 1 year ago

  • Assignee changed from Vladimir Vinogradenko to William Grzybowski

#20 Updated by William Grzybowski about 1 year ago

  • Status changed from Needs Developer Review to Screened
  • Assignee changed from William Grzybowski to Vladimir Vinogradenko

I have a single concern which I have pointed out in the PR.

Other than that, it looks great!

#21 Updated by William Grzybowski about 1 year ago

  • Status changed from Screened to Reviewed by Developer

#22 Updated by Vladimir Vinogradenko about 1 year ago

  • Status changed from Reviewed by Developer to Ready For Release

#23 Updated by Dru Lavigne about 1 year ago

  • Subject changed from Dataset quota e-mail notificatoins to Add dataset quota email notifications

#24 Updated by Dru Lavigne about 1 year ago

  • Target version changed from 11.1 to 11.1-BETA1

#25 Updated by Dru Lavigne 12 months ago

  • Status changed from Ready For Release to Resolved

#26 Updated by Nick Wolff 11 months ago

How often should this end up being polled? Does this apply to datasets and zvols or just zvols?

Ticket is talking about datasets commits seem focused around zvols though.

#27 Updated by Nick Wolff 11 months ago

  • Status changed from Resolved to Unscreened
  • QA Status Test Fails FreeNAS added
  • QA Status deleted (Not Tested)

Not receiving alert or email on dataset with quota set.

Setting Test Fails

#28 Updated by Nick Wolff 11 months ago

  • Target version changed from 11.1-BETA1 to 11.1

#29 Updated by Dru Lavigne 11 months ago

  • Status changed from Unscreened to 46

Vladimir: please discuss the failed QA test with Nick and William.

#30 Updated by Vladimir Vinogradenko 11 months ago

  • Status changed from 46 to 15

Nick, this only applies to datasets. zvols do not have a quota property in libzfs.

__get_quota_excess code is here: https://github.com/freenas/freenas/pull/296/commits/de40e0dbe8ba5438a5aaf64d5c5ec4f02b792b87#diff-eb0eff492b12244cf46ddc1fa7968b65R385

Quotas are polled every 60 seconds. There is no alert issued for quota excess, only e-mail to root user.

Did you have your e-mail subsystem set up before reaching quota excess?

#31 Updated by Nick Wolff 11 months ago

Sorry about confusion with zvol/dataset the Associated revisions in redmine were a bit confusing. I have verified that email is functional on system.
Not seeing any emails or other indicators
root@fncertified:~ # sqlite3 /data/freenas-v1.db 'select * from storage_quotaexcess;'
root@fncertified:~ #

root@fncertified:~ # zfs get all tank/nwolff/17923-dataset-quota-email
NAME PROPERTY VALUE SOURCE
tank/nwolff/17923-dataset-quota-email type filesystem -
tank/nwolff/17923-dataset-quota-email creation Tue Nov 7 7:51 2017 -
tank/nwolff/17923-dataset-quota-email used 8.79G -
tank/nwolff/17923-dataset-quota-email available 1.21G -
tank/nwolff/17923-dataset-quota-email referenced 8.79G -
tank/nwolff/17923-dataset-quota-email compressratio 1.00x -
tank/nwolff/17923-dataset-quota-email mounted yes -
tank/nwolff/17923-dataset-quota-email quota none local
tank/nwolff/17923-dataset-quota-email reservation none local
tank/nwolff/17923-dataset-quota-email recordsize 128K default
tank/nwolff/17923-dataset-quota-email mountpoint /mnt/tank/nwolff/17923-dataset-quota-email default
tank/nwolff/17923-dataset-quota-email sharenfs off default
tank/nwolff/17923-dataset-quota-email checksum on default
tank/nwolff/17923-dataset-quota-email compression lz4 inherited from tank
tank/nwolff/17923-dataset-quota-email atime on default
tank/nwolff/17923-dataset-quota-email devices on default
tank/nwolff/17923-dataset-quota-email exec on default
tank/nwolff/17923-dataset-quota-email setuid on default
tank/nwolff/17923-dataset-quota-email readonly off default
tank/nwolff/17923-dataset-quota-email jailed off default
tank/nwolff/17923-dataset-quota-email snapdir hidden default
tank/nwolff/17923-dataset-quota-email aclmode passthrough local
tank/nwolff/17923-dataset-quota-email aclinherit passthrough inherited from tank
tank/nwolff/17923-dataset-quota-email canmount on default
tank/nwolff/17923-dataset-quota-email xattr off temporary
tank/nwolff/17923-dataset-quota-email copies 1 default
tank/nwolff/17923-dataset-quota-email version 5 -
tank/nwolff/17923-dataset-quota-email utf8only off -
tank/nwolff/17923-dataset-quota-email normalization none -
tank/nwolff/17923-dataset-quota-email casesensitivity sensitive -
tank/nwolff/17923-dataset-quota-email vscan off default
tank/nwolff/17923-dataset-quota-email nbmand off default
tank/nwolff/17923-dataset-quota-email sharesmb off default
tank/nwolff/17923-dataset-quota-email refquota 10G local
tank/nwolff/17923-dataset-quota-email refreservation none local
tank/nwolff/17923-dataset-quota-email primarycache all default
tank/nwolff/17923-dataset-quota-email secondarycache all default
tank/nwolff/17923-dataset-quota-email usedbysnapshots 0 -
tank/nwolff/17923-dataset-quota-email usedbydataset 8.79G -
tank/nwolff/17923-dataset-quota-email usedbychildren 0 -
tank/nwolff/17923-dataset-quota-email usedbyrefreservation 0 -
tank/nwolff/17923-dataset-quota-email logbias latency default
tank/nwolff/17923-dataset-quota-email dedup off default
tank/nwolff/17923-dataset-quota-email mlslabel -
tank/nwolff/17923-dataset-quota-email sync standard default
tank/nwolff/17923-dataset-quota-email refcompressratio 1.00x -
tank/nwolff/17923-dataset-quota-email written 8.79G -
tank/nwolff/17923-dataset-quota-email logicalused 8.79G -
tank/nwolff/17923-dataset-quota-email logicalreferenced 8.79G -
tank/nwolff/17923-dataset-quota-email volmode default default
tank/nwolff/17923-dataset-quota-email filesystem_limit none default
tank/nwolff/17923-dataset-quota-email snapshot_limit none default
tank/nwolff/17923-dataset-quota-email filesystem_count none default
tank/nwolff/17923-dataset-quota-email snapshot_count none default
tank/nwolff/17923-dataset-quota-email redundant_metadata all default
root@fncertified:~ #

#32 Updated by Vladimir Vinogradenko 11 months ago

Nick,

tank/nwolff/17923-dataset-quota-email quota none local
tank/nwolff/17923-dataset-quota-email refquota 10G local

You have refquota set. We are currently only reading quota.

William/Kris, should we also take refquota into account? (e.g. calculate threshold against min(quota, refquota))

#33 Updated by Dru Lavigne 11 months ago

  • Assignee changed from Vladimir Vinogradenko to William Grzybowski

#34 Updated by William Grzybowski 11 months ago

  • Assignee changed from William Grzybowski to Vladimir Vinogradenko

We should probably take both into account?

quota checks the space of the dataset and descendants
refquota checks the space of just the dataset

Does that make sense?

#35 Updated by Vladimir Vinogradenko 11 months ago

  • Status changed from 15 to Needs Developer Review
  • Assignee changed from Vladimir Vinogradenko to William Grzybowski

#36 Updated by William Grzybowski 11 months ago

  • Status changed from Needs Developer Review to Reviewed by Developer
  • Assignee changed from William Grzybowski to Vladimir Vinogradenko

#37 Updated by Vladimir Vinogradenko 11 months ago

  • Status changed from Reviewed by Developer to Ready For Release

#38 Updated by Bonnie Follweiler 11 months ago

  • Needs QA changed from Yes to No
  • QA Status deleted (Test Fails FreeNAS)

Received this email: Refquota exceed on dataset tank/Photos.
Used 82.36% (43.18 MB of 52.43 MB)

#39 Updated by Dru Lavigne 11 months ago

  • Target version changed from 11.1 to 11.1-RC2

#40 Updated by Dru Lavigne 11 months ago

  • Target version changed from 11.1-RC2 to 11.1-RC3

#41 Updated by Dru Lavigne 11 months ago

  • Status changed from Ready For Release to Resolved

#43 Updated by Joshua Sirrine 4 months ago

  • Status changed from Resolved to In Progress
  • Target version changed from 11.1-RC3 to TrueNAS-11.1-U6.1

#45 Updated by Vladimir Vinogradenko 4 months ago

  • Status changed from In Progress to Blocked
  • Reason for Blocked set to Waiting for feedback

I see debug only from one node. Is there a debug from another node?

It's a bug that both nodes process these alerts, it will be fixed, but nevertheless it's unclear why other node sends alerts over and over again.

#46 Updated by Dru Lavigne 4 months ago

  • Copied to Bug #34684: Fix quota exceeded emails being sent every minute added

#47 Updated by Dru Lavigne 4 months ago

  • Status changed from Blocked to Done
  • Target version changed from TrueNAS-11.1-U6.1 to 11.1-RC3
  • Reason for Blocked deleted (Waiting for feedback)

Also available in: Atom PDF