Project

General

Profile

Bug #69517

Middleware timeout when managing SSH service

Added by Sid Bradford 7 days ago. Updated 1 day ago.

Status:
Blocked
Priority:
No priority
Assignee:
Release Council
Category:
Middleware
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Need additional information from Author
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

I recently updated a FreeNAS host from 11.1 U4 to 11.1 U6 and began experiencing middleware timeouts whenever I changed anything relating to the SSH service (stop/start, config change). Config changes do appear to save, but starting and stopping requires manually doing so on the command line.

I have also tested upgrades to 11.1 U6.3 and 11.2 U1 (after reverting back to the 11.1 U4 boot), which had the same issue. The system in question has no internet access, so I am performing the update on the CLI using the `freenas-update` command and the manual update tarball.

Stack trace:

Environment:

Software Version: FreeNAS-11.1-U6 (caffd76fa)
Request Method: POST
Request URL: https://<ip>/admin/services/ssh/edit/1/

Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in wrapper
  210.                 return self._admin.admin_view(view)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  149.                     response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57.         response = view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/site.py" in inner
  142.             return view(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in edit
  571.                     mf.save()
File "./freenasUI/services/forms.py" in save
  484.         started = notifier().reload("ssh")
File "./freenasUI/middleware/notifier.py" in reload
  230.             return c.call('service.reload', what, {'onetime': onetime}, **kwargs)
File "./freenasUI/middleware/notifier.py" in reload
  230.             return c.call('service.reload', what, {'onetime': onetime}, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  429.             raise CallTimeout("Call timeout")

Exception Type: CallTimeout at /admin/services/ssh/edit/1/
Exception Value: Call timeout

History

#1 Updated by Dru Lavigne 6 days ago

  • Status changed from Unscreened to Blocked
  • Reason for Blocked set to Need additional information from Author

Sid: this may be an indication that the boot device is wearing out. As a test, please save your config, perform a fresh install of 11.1-U6 on a new stick and restore your config. Let us know if that resolves the issue or if it persists.

#2 Updated by Sid Bradford 4 days ago

Dru Lavigne wrote:

Sid: this may be an indication that the boot device is wearing out. As a test, please save your config, perform a fresh install of 11.1-U6 on a new stick and restore your config. Let us know if that resolves the issue or if it persists.

Had a chance to perform this today. I did a fresh install of 11.1-U6.3 then restored the config backup. Unfortunately I am running into the same issue as before. Just in case, I have provided the latest stack trace below:

Environment:

Software Version: FreeNAS-11.1-U6.3 (35cbff5a1)
Request Method: POST
Request URL: https://<ip>/admin/services/ssh/edit/1/

Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in wrapper
  210.                 return self._admin.admin_view(view)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  149.                     response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57.         response = view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/site.py" in inner
  142.             return view(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in edit
  571.                     mf.save()
File "./freenasUI/services/forms.py" in save
  484.         started = notifier().reload("ssh")
File "./freenasUI/middleware/notifier.py" in reload
  249.                 **kwargs,
File "./freenasUI/middleware/notifier.py" in reload
  249.                 **kwargs,
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  429.             raise CallTimeout("Call timeout")

Exception Type: CallTimeout at /admin/services/ssh/edit/1/
Exception Value: Call timeout

#3 Updated by Dru Lavigne 4 days ago

Just to confirm, the fresh install was on a new stick?

If so, are you able to create a debug (System -> Advanced -> Save debug) to attach to this ticket so we can start to diagnose the underlying cause?

#4 Updated by Sid Bradford 1 day ago

My apologies. I missed the "new stick" portion in your original comment. So one thing I should mention is that the FreeNAS OS is installed to two drives in a ZFS mirror. Serial numbers of those two drives are fairly close, however.

Some song and dance will be needed to upload that debug output, but I'll upload it once I've got the thumbs up.

Also available in: Atom PDF