Project

General

Profile

Bug #41365

Avoid sending dozens of crash reports filling up thread pool

Added by William Grzybowski over 1 year ago. Updated over 1 year ago.

Status:
Done
Priority:
No priority
Assignee:
William Grzybowski
Category:
Middleware
Target version:
Seen in:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

I have a few VLAN interfaces on my FreeNAS, and I think that's causing the rrdtool to go a bit haywire. The graphs just spin, and the logs are constant errors.

[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ["ValueError: rrdtool failed: ERROR: opening '/var/db/collectd/rrd/localhost//interface-interface-tap5/if_octets.rrd': No such file or directory\n", '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ["ValueError: rrdtool failed: ERROR: opening '/var/db/collectd/rrd/localhost//interface-interface-bridge1/if_octets.rrd': No such file or directory\n", '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ["ValueError: rrdtool failed: ERROR: opening '/var/db/collectd/rrd/localhost//interface-interface-tap1/if_octets.rrd': No such file or directory\n", '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:40] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:40] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:41] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ["ValueError: rrdtool failed: ERROR: opening '/var/db/collectd/rrd/localhost//interface-interface-vnet1/if_octets.rrd': No such file or directory\n", '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:41] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:41] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:41] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ["ValueError: rrdtool failed: ERROR: opening '/var/db/collectd/rrd/localhost//interface-interface-vlan250/if_octets.rrd': No such file or directory\n", '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:41] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:41] (DEBUG) middlewared.logger.CrashReporting.report():97 - Sending a crash report...
[2018/08/05 02:55:41] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ['ValueError: rrdtool failed: ERROR: bad CF: rx\n', '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']
[2018/08/05 02:55:41] (ERROR) sentry.errors.uncaught._log_failed_submission():702 - ["ValueError: rrdtool failed: ERROR: opening '/var/db/collectd/rrd/localhost//interface-interface-bridge2/if_octets.rrd': No such file or directory\n", '  File "middlewared/main.py", line 154, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 1013, in call_method', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/main.py", line 953, in _call', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "middlewared/schema.py", line 658, in nf', '  File "asyncio/coroutines.py", line 110, in __next__', '  File "plugins/stats.py", line 113, in get_data']

Related issues

Related to FreeNAS - Bug #43789: Sentry responded with an API error: RateLimited(None)Closed
Has duplicate FreeNAS - Bug #41304: Unable to run alert source 'HasUpdate'Closed
Copied from FreeNAS - Bug #40788: Use CallError on rrdtool failureDone

Associated revisions

Revision d03ce642 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): limit sentry crash reporting Ticket: #41365

Revision 07296720 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): limit sentry reporting avoiding middleware lockup Ticket: #41365

Revision e95f5497 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): put crash reporting into a new coroutine Ticket: #41365

Revision 85173570 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): put crash reporting into a new coroutine Ticket: #41365

Revision e443cb66 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): put crash reporting into a new coroutine Ticket: #41365

Revision fb3e320f (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): limit sentry reporting avoiding middleware lockup Ticket: #41365

Revision 82cebd06 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): put crash reporting into a new coroutine Ticket: #41365

History

#1 Updated by William Grzybowski over 1 year ago

  • Copied from Bug #40788: Use CallError on rrdtool failure added

#2 Updated by Bug Clerk over 1 year ago

  • Status changed from Unscreened to In Progress

#3 Updated by Bug Clerk over 1 year ago

  • Status changed from In Progress to Ready for Testing

#4 Updated by William Grzybowski over 1 year ago

  • Has duplicate Bug #41304: Unable to run alert source 'HasUpdate' added

#5 Updated by Dru Lavigne over 1 year ago

  • Status changed from Ready for Testing to Done
  • Needs QA changed from Yes to No

#7 Updated by William Grzybowski over 1 year ago

  • Related to Bug #43789: Sentry responded with an API error: RateLimited(None) added

Also available in: Atom PDF