Project

General

Profile

Bug #43789

Avatar?id=27674&size=50x50

Sentry responded with an API error: RateLimited(None)

Added by Ryan McCullough about 2 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
No priority
Assignee:
William Grzybowski
Category:
Middleware
Target version:
Seen in:
Severity:
New
Reason for Closing:
Duplicate Issue
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

I started getting errors in the console today while moving some files around on my server. I am attaching a debug.tgz I generated a few moments ago. Another attachment is a screenshot of my IPMI console (I can't copy/paste this as text, sorry).

Here is my setup:
FreeNAS 11.2 (Beta 2)
Server Chassis/ Case: CSE-826A-R1200LPB
Motherboard: X9DRi-LN4F+
Backplane: BPN-SAS-826A I-Path Direct Attached Backplane
2x LSI 9210-8i HBA JBOD FREENAS UNRAID
9x 2TB SAS Hard Drives 7.2K RPM 6GB/S HITACHI GST ULTRASTAR HUS723020ALS640

In addition to the errors I am seeing in the console, I am also getting email alerts:

New alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 352, in __run_source
    alerts = (await alert_source.check()) or []
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 905, in run_in_thread
    return await self.run_in_executor(self.__threadpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 902, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1043, in call_sync
    return fut.result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 967, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 913, in run_in_io_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/www/freenasUI/middleware/notifier.py", line 1582, in get_update_location
    with client as c:
  File "/usr/local/www/freenasUI/middleware/client.py", line 20, in __enter__
    local.client = Client()
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 314, in __init__
    raise e
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 307, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 169, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.6/site-packages/ws4py/client/__init__.py", line 215, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out

Alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 352, in __run_source
    alerts = (await alert_source.check()) or []
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 905, in run_in_thread
    return await self.run_in_executor(self.__threadpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 902, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1043, in call_sync
    return fut.result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 967, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 129, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 913, in run_in_io_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/www/freenasUI/middleware/notifier.py", line 1582, in get_update_location
    with client as c:
  File "/usr/local/www/freenasUI/middleware/client.py", line 20, in __enter__
    local.client = Client()
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 314, in __init__
    raise e
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 307, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 169, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.6/site-packages/ws4py/client/__init__.py", line 215, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out

console errors.jpg (78.6 KB) console errors.jpg Ryan McCullough, 08/30/2018 09:53 AM
27443

Related issues

Related to FreeNAS - Bug #41365: Avoid sending dozens of crash reports filling up thread poolDone
Related to FreeNAS - Bug #43811: Fix deadlock on thread poolDone

History

#1 Updated by Dru Lavigne about 2 years ago

  • Private changed from No to Yes
  • Reason for Blocked set to Need additional information from Author

Ryan: please attach a debug (System -> Advanced -> Save debug) to this ticket.

#2 Avatar?id=27674&size=24x24 Updated by Ryan McCullough about 2 years ago

Dru Lavigne wrote:

Ryan: please attach a debug (System -> Advanced -> Save debug) to this ticket.

Dru, I attached it when I created this issue. Do we need anything different?

#3 Updated by Dru Lavigne about 2 years ago

Duh! Sorry about that :-)

#4 Updated by Dru Lavigne about 2 years ago

  • Category changed from Console to Middleware
  • Assignee changed from Release Council to William Grzybowski
  • Reason for Blocked deleted (Need additional information from Author)

#5 Avatar?id=27674&size=24x24 Updated by Ryan McCullough about 2 years ago

Dru Lavigne wrote:

Duh! Sorry about that :-)

No problem, just wanted to make sure I had attached the right thing. Thanks!

#6 Avatar?id=27674&size=24x24 Updated by Ryan McCullough about 2 years ago

Also, this may be related to #43811.

#7 Updated by William Grzybowski about 2 years ago

  • Status changed from Unscreened to Closed
  • Target version changed from Backlog to N/A
  • Reason for Closing set to Duplicate Issue

This should be fixed in related tickets.

Please report back if it happens on BETA3 once its out.

#8 Updated by William Grzybowski about 2 years ago

  • Related to Bug #41365: Avoid sending dozens of crash reports filling up thread pool added

#9 Updated by William Grzybowski about 2 years ago

  • Related to Bug #43811: Fix deadlock on thread pool added

#10 Updated by Dru Lavigne about 2 years ago

  • File deleted (debug.tgz)

#11 Updated by Dru Lavigne about 2 years ago

  • Private changed from Yes to No

#12 Updated by Alain Dumas almost 2 years ago

  • Subject changed from Console Errors: "Sentry responded with an API error: RateLimited(None)" to Unable to run alert source 'HasUpdate'
  • Seen in changed from 11.2-BETA2 to 11.2-RC1

Got this email alert in RC1. Is this related ?

  • Unable to run alert source 'HasUpdate'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in _run_source
    alerts = (await alert_source.check()) or []
    File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 943, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1073, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.
    _get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 997, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 943, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/www/freenasUI/middleware/notifier.py", line 1603, in get_update_location
    syspath = c.call('systemdataset.config')['path']
    File "/usr/local/www/freenasUI/middleware/notifier.py", line 1603, in get_update_location
    syspath = c.call('systemdataset.config')['path']
    File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 447, in call
    raise CallTimeout("Call timeout")
    middlewared.client.client.CallTimeout: Call timeout

#13 Updated by William Grzybowski almost 2 years ago

  • Subject changed from Unable to run alert source 'HasUpdate' to Sentry responded with an API error: RateLimited(None)
  • Seen in changed from 11.2-RC1 to 11.2-BETA2

Please dont reopen old closed bugs.

If you're facing a similar issue open a new bug attaching a debug.

Also available in: Atom PDF