Project

General

Profile

Bug #56778

New alerts: * Unable to run alert source 'HasUpdate'

Added by Richie Crews almost 3 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
No priority
Assignee:
William Grzybowski
Category:
Middleware
Target version:
Seen in:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Got an email with the following error over night.

New alerts:
  • Unable to run alert source 'HasUpdate'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in run_source
    alerts = (await alert_source.check()) or []
    File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 943, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1073, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in _get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 997, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 943, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/www/freenasUI/middleware/notifier.py", line 1602, in get_update_location
    with client as c:
    File "/usr/local/www/freenasUI/middleware/client.py", line 20, in __enter

    local.client = Client()
    File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 320, in init
    raise e
    File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 313, in init
    self._ws.connect()
    File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 170, in connect
    rv = super(WSClient, self).connect()
    File "/usr/local/lib/python3.6/site-packages/ws4py/client/
    _init__.py", line 215, in connect
    bytes = self.sock.recv(128)
    socket.timeout: timed out
Alerts:
  • Unable to run alert source 'HasUpdate'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in run_source
    alerts = (await alert_source.check()) or []
    File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 943, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1073, in call_sync
    return fut.result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
    File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in _get_result
    raise self._exception
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 997, in _call
    return await run_method(methodobj, *args)
    File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 943, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
    File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/www/freenasUI/middleware/notifier.py", line 1602, in get_update_location
    with client as c:
    File "/usr/local/www/freenasUI/middleware/client.py", line 20, in __enter

    local.client = Client()
    File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 320, in init
    raise e
    File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 313, in init
    self._ws.connect()
    File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 170, in connect
    rv = super(WSClient, self).connect()
    File "/usr/local/lib/python3.6/site-packages/ws4py/client/
    _init__.py", line 215, in connect
    bytes = self.sock.recv(128)
    socket.timeout: timed out

Related issues

Related to FreeNAS - Bug #55851: Unable to run alert source 'HasUpdate'Closed

History

#1 Updated by Richie Crews almost 3 years ago

  • File debug-TARDIS-20181110212856.txz added
  • Private changed from No to Yes

#2 Updated by Dru Lavigne almost 3 years ago

  • Related to Bug #55851: Unable to run alert source 'HasUpdate' added

#3 Updated by Dru Lavigne almost 3 years ago

  • Category changed from OS to Middleware
  • Assignee changed from Release Council to William Grzybowski

#5 Updated by William Grzybowski almost 3 years ago

  • Status changed from Unscreened to Blocked
  • Reason for Blocked set to Waiting for feedback

Can you verify if this happens again once RC2 is out, please?

#6 Updated by Dru Lavigne almost 3 years ago

  • File deleted (debug-TARDIS-20181110212856.txz)

#7 Updated by Dru Lavigne almost 3 years ago

  • Status changed from Blocked to Closed
  • Target version changed from Backlog to N/A
  • Private changed from Yes to No

Richie: closing out this ticket for now. Please leave a comment on this ticket if the issue reoccurs.

#8 Updated by Tom Mü-Ko almost 3 years ago

New alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
 File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
   alerts = (await alert_source.check()) or []
 File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
   return await self.middleware.run_in_thread(self.check_sync)
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
 File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
 File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
   path = self.middleware.call_sync("notifier.get_update_location")
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
   return fut.result()
 File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
   return self.__get_result()
 File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
   raise self._exception
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
   return await run_method(methodobj, *args)
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
 File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
 File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
   syspath = c.call('systemdataset.config')['path']
 File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
   syspath = c.call('systemdataset.config')['path']
 File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 447, in call
   raise CallTimeout("Call timeout")
middlewared.client.client.CallTimeout: Call timeout

Alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
 File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
   alerts = (await alert_source.check()) or []
 File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
   return await self.middleware.run_in_thread(self.check_sync)
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
 File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
 File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
   path = self.middleware.call_sync("notifier.get_update_location")
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
   return fut.result()
 File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
   return self.__get_result()
 File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
   raise self._exception
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
   return await run_method(methodobj, *args)
 File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
 File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
 File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
   syspath = c.call('systemdataset.config')['path']
 File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
   syspath = c.call('systemdataset.config')['path']
 File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 447, in call
   raise CallTimeout("Call timeout")
middlewared.client.client.CallTimeout: Call timeout

#9 Updated by gear head almost 3 years ago

I also got this error today. I've attached the bug report.

 CRITICAL: Dec. 22, 2018, 9:05 p.m. - Unable to run alert source 'HasUpdate' Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source alerts = (await alert_source.check()) or [] File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check return await self.middleware.run_in_thread(self.check_sync) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync path = self.middleware.call_sync("notifier.get_update_location") File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync return fut.result() File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call return await run_method(methodobj, *args) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location syspath = c.call('systemdataset.config')['path'] File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location syspath = c.call('systemdataset.config')['path'] File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 447, in call raise CallTimeout("Call timeout") middlewared.client.client.CallTimeout: Call timeout 

#10 Updated by Dru Lavigne almost 3 years ago

  • Status changed from Closed to Unscreened
  • Target version changed from N/A to Backlog
  • Reason for Blocked deleted (Waiting for feedback)

#11 Updated by William Grzybowski almost 3 years ago

  • Status changed from Unscreened to Blocked
  • Reason for Blocked set to Need additional information from Author

How often does it happen?

#12 Updated by gear head almost 3 years ago

it has only happened one time so far

#13 Updated by William Grzybowski almost 3 years ago

gear head wrote:

it has only happened one time so far

Did it happen while freenas-boot was scrubbing?

#14 Updated by gear head almost 3 years ago

It was during or after scrubbing.

#15 Updated by William Grzybowski almost 3 years ago

  • Status changed from Blocked to Screened
  • Target version changed from Backlog to 11.2-U2
  • Severity changed from New to Medium
  • Reason for Blocked deleted (Need additional information from Author)

#16 Updated by William Grzybowski almost 3 years ago

  • Target version changed from 11.2-U2 to 11.2-U3

#17 Updated by IT IGP over 2 years ago

also received 4 mails in total (on the 28th, 2 at 1am, 2 at 4am), probably from two systems. one system upgraded from 11.1u6.3 and another fresh 11.2 install.

also : in the email the freenas host name is not set, so it is not clear which system sent which mail!

New alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
    alerts = (await alert_source.check()) or []
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
    return fut.result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
    syspath = c.call('systemdataset.config')['path']
  File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
    syspath = c.call('systemdataset.config')['path']
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 447, in call
    raise CallTimeout("Call timeout")
middlewared.client.client.CallTimeout: Call timeout

Alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
    alerts = (await alert_source.check()) or []
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
    return await self.middleware.run_in_thread(self.check_sync)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 32, in check_sync
    path = self.middleware.call_sync("notifier.get_update_location")
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
    return fut.result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
    syspath = c.call('systemdataset.config')['path']
  File "/usr/local/www/freenasUI/middleware/notifier.py", line 1609, in get_update_location
    syspath = c.call('systemdataset.config')['path']
  File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 447, in call
    raise CallTimeout("Call timeout")
middlewared.client.client.CallTimeout: Call timeout

#18 Updated by William Grzybowski over 2 years ago

  • Target version changed from 11.2-U3 to 11.2-U4

I am still unsure how to reproduce and tackle this problem. Re-targeting to U4.

#19 Avatar?id=14398&size=24x24 Updated by Kris Moore over 2 years ago

  • Status changed from Screened to Closed

Also available in: Atom PDF