Project

General

Profile

Bug #67836

Sentry responded with an API error: RateLimited(None)

Added by Richard Fancher almost 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
No priority
Assignee:
William Grzybowski
Category:
Middleware
Target version:
Severity:
New
Reason for Closing:
Duplicate Issue
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No
Tags:

Description

All that I did was power off a VM that was running and I started to get these spamming in my /var/log/messages

Jan 5 03:17:34 FreeNasV2 uwsgi: [sentry.errors:674] Sentry responded with an API error: RateLimited(None)
Jan 5 03:17:34 FreeNasV2 uwsgi: [sentry.errors.uncaught:702] ['timeout: timed out', ' File "django/core/handlers/exception.py", line 42, in inner', ' File "django/core/handlers/base.py", line 244, in legacy_get_response', ' File "freenasUI/freeadmin/middleware.py", line 296, in process_request', ' File "freenasUI/middleware/auth.py", line 8, in authenticate', ' File "freenasUI/middleware/client.py", line 20, in enter', ' File "middlewared/client/client.py", line 320, in init', ' File "middlewared/client/client.py", line 313, in init', ' File "middlewared/client/client.py", line 170, in connect', ' File "ws4py/client/_init__.py", line 215, in connect']

Freenas GUI was unresponsive for about 5 mins and kept disconnecting


Related issues

Is duplicate of FreeNAS - Bug #69339: Add a semaphore for calls within the same connectionDone

History

#1 Updated by Dru Lavigne almost 3 years ago

  • Category changed from GUI (new) to Middleware
  • Assignee changed from Release Council to William Grzybowski
  • Private changed from No to Yes
  • Seen in set to 11.2-RELEASE-U1

#2 Updated by Richard Fancher almost 3 years ago

I would also like to add that I received these from my freenas after it had recovered.

@New alerts:
  • Unable to run alert source 'ZpoolCapacity'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in run_source
    alerts = (await alert_source.check()) or []
    File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/zpool_capacity.py", line 18, in check_sync
    for pool in self.middleware.call_sync("pool.query")
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in _get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1049, in _call
    return await methodobj(*args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
    return await f(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 266, in query
    'datastore.query', self._config.datastore, [], datastore_options
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call
    return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
    select=options.get('select'),
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 90, in __queryset_serialize
    self.middleware, i, extend=extend, field_prefix=field_prefix, select=select
    File "/usr/local/lib/python3.6/site-packages/middlewared/utils/
    _init
    .py", line 58, in django_modelobj_serialize
    data = middleware.call_sync(extend, data)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/asyncio/base_events.py", line 639, in run_in_executor
    return futures.wrap_future(executor.submit(func, *args), loop=self)
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 123, in submit
    self._adjust_thread_count()
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 142, in _adjust_thread_count
    t.start()
    File "/usr/local/lib/python3.6/threading.py", line 846, in start
    _start_new_thread(self._bootstrap, ())
    RuntimeError: can't start new thread
Gone alerts:
  • Unable to run alert source 'LAGGStatus'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
    alerts = (await alert_source.check()) or []
    File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/asyncio/base_events.py", line 639, in run_in_executor
    return futures.wrap_future(executor.submit(func, *args), loop=self)
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 123, in submit
    self._adjust_thread_count()
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 142, in _adjust_thread_count
    t.start()
    File "/usr/local/lib/python3.6/threading.py", line 846, in start
    _start_new_thread(self._bootstrap, ())
    RuntimeError: can't start new thread
  • Unable to run alert source 'VolumeStatus'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in run_source
    alerts = (await alert_source.check()) or []
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/volume_status.py", line 15, in check
    for pool in await self.middleware.call("pool.query"):
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call
    return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1049, in call
    return await methodobj(*args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
    return await f(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 266, in query
    'datastore.query', self._config.datastore, [], datastore_options
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call
    return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
    select=options.get('select'),
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 90, in __queryset_serialize
    self.middleware, i, extend=extend, field_prefix=field_prefix, select=select
    File "/usr/local/lib/python3.6/site-packages/middlewared/utils/
    _init
    .py", line 58, in django_modelobj_serialize
    data = middleware.call_sync(extend, data)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in _get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/pool.py", line 257, in pool_extend
    pool['topology'] = self._topology(zpool['groups'])
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/pool.py", line 236, in _topology
    x[key] = self._topology(x[key], False)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/pool.py", line 239, in _topology
    x[i] = self._topology(x[i], False)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/pool.py", line 236, in _topology
    x[key] = self._topology(x[key], False)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/pool.py", line 239, in _topology
    x[i] = self._topology(x[i], False)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/pool.py", line 230, in _topology
    device = self.middleware.call_sync('disk.label_to_dev', path[5:], geom_scan)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.
    _get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/asyncio/base_events.py", line 639, in run_in_executor
    return futures.wrap_future(executor.submit(func, *args), loop=self)
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 123, in submit
    self._adjust_thread_count()
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 142, in _adjust_thread_count
    t.start()
    File "/usr/local/lib/python3.6/threading.py", line 846, in start
    _start_new_thread(self._bootstrap, ())
    RuntimeError: can't start new thread

@

@Gone alerts:
  • Unable to run alert source 'HasUpdate'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in _run_source
    alerts = (await alert_source.check()) or []
    File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.
    _get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/asyncio/base_events.py", line 639, in run_in_executor
    return futures.wrap_future(executor.submit(func, *args), loop=self)
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 123, in submit
    self._adjust_thread_count()
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 142, in _adjust_thread_count
    t.start()
    File "/usr/local/lib/python3.6/threading.py", line 846, in start
    _start_new_thread(self._bootstrap, ())
    RuntimeError: can't start new thread

@

#3 Updated by William Grzybowski almost 3 years ago

  • Status changed from Unscreened to Blocked
  • Reason for Blocked set to Need additional information from Author

Are you able to reproduce it?

Can you log the exact required steps?

Unfortunately the debug does not have anything useful to pinpoint the exact issue.

#4 Updated by Richard Fancher almost 3 years ago

William Grzybowski wrote:

Are you able to reproduce it?

Can you log the exact required steps?

Unfortunately the debug does not have anything useful to pinpoint the exact issue.

I am not but it was similar to another bug I submitted [FreeNAS - Bug #64539] VM Cloning failed, it did the same thing but it does not happen consistently.

#5 Updated by William Grzybowski almost 3 years ago

  • Status changed from Blocked to Closed
  • Target version changed from Backlog to N/A
  • Reason for Closing set to Duplicate Issue
  • Reason for Blocked deleted (Need additional information from Author)

#6 Updated by William Grzybowski almost 3 years ago

  • Is duplicate of Bug #69339: Add a semaphore for calls within the same connection added

#7 Updated by Dru Lavigne almost 3 years ago

  • File deleted (debug-FreeNasV2-20190105032459.tgz)

#8 Updated by Dru Lavigne almost 3 years ago

  • Private changed from Yes to No

Also available in: Atom PDF