Project

General

Profile

Bug #34687

Add timeout for LDAP status calls

Added by Joe Maloney 10 months ago. Updated 7 months ago.

Status:
Done
Priority:
No priority
Assignee:
Andrew Walker
Category:
Services
Target version:
TrueNAS - TrueNAS-11.1-U6.1
Seen in:
TrueNAS - TrueNAS 11.1-U5
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
Yes
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

I have reproduced this on two seperate TrueNAS systems running 11.1-u5-RC6. While ldap01.tn.ixsystems.com is online, and running in a VM I can join, and disjoin ldap. When I power this VM down TrueNAS is unable to disjoin LDAP.

Environment:

Software Version: TrueNAS-11.1-U5-RC7 (6ccf719c5)
Request Method: POST
Request URL: http://10.20.20.32/admin/directoryservice/ldap/edit/1/?inline=true


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in wrapper
  210.                 return self._admin.admin_view(view)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  149.                     response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57.         response = view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/site.py" in inner
  142.             return view(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in edit
  571.                     mf.save()
File "./freenasUI/directoryservice/forms.py" in save
  951.         started = notifier().started("ldap")
File "./freenasUI/middleware/notifier.py" in started
  209.             return c.call('service.started', what, **kwargs)
File "./freenasUI/middleware/notifier.py" in started
  209.             return c.call('service.started', what, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  429.             raise CallTimeout("Call timeout")

Exception Type: CallTimeout at /admin/directoryservice/ldap/edit/1/
Exception Value: Call timeout

Other side effects which I may produce two additional tickets for are that I am unable to collect a debug. I have been waiting over 10 minutes. Also I am unable to list the system advanced page in under 30 seconds for the first time when navigating to that page when the ldap server is offline.

History

#1 Updated by Joe Maloney 10 months ago

  • File debug-x20qa1-20180608124726.tgz added

Attaching debug that took forever to generate with ldap offline.

#2 Updated by Dru Lavigne 10 months ago

  • Category set to OS
  • Assignee set to John Hixson
  • Target version changed from Backlog to TrueNAS-11.1-U6.1

#3 Updated by Joe Maloney 10 months ago

  • Needs Automation changed from No to Yes

This has been broken since at least FreeNAS 11.0-U4, and all of the 11.1 releases. We do not catch this in automation because our automated tests due not attempt to shutdown the ldap server, or break DNS, and then attempt to disjoin the LDAP server.

#4 Updated by John Hixson 8 months ago

  • Assignee changed from John Hixson to Andrew Walker

#5 Updated by John Hixson 8 months ago

  • Category changed from OS to Services

#6 Updated by Bug Clerk 8 months ago

  • Status changed from Unscreened to In Progress

#7 Updated by Andrew Walker 8 months ago

Reproduced problem by configuring LDAP on TrueNAS 11.1-U5.1, then suspending the LDAP server VM. Command

midclt call service.started ldap
timed out. Noted the following code block:

    async def _started_ldap(self, **kwargs):
        if (await self._system('/usr/sbin/service ix-ldap status') != 0):
            return False, []
        return await self.middleware.call('notifier.ldap_status'), []

Issued

service ix-ldap status
and command hung indefinitely. ix-ldap was hanging because we were performing ldapsearch without a timeout set. I added a timeout based on the sysctl timeout value for determining ldap service status so that the timeout value is ultimately user-configurable.

PR for Master - https://github.com/freenas/freenas/pull/1751
PR for Stable - https://github.com/freenas/freenas/pull/1752

#8 Updated by Bug Clerk 8 months ago

  • Status changed from In Progress to Ready for Testing
  • Target version changed from TrueNAS-11.1-U6.1 to TrueNAS 11.1-U6.2

#9 Updated by Dru Lavigne 8 months ago

  • File deleted (debug-x20qa1-20180608124726.tgz)

#10 Updated by Dru Lavigne 8 months ago

  • Project changed from TrueNAS to FreeNAS
  • Category changed from Services to Services
  • Target version changed from TrueNAS 11.1-U6.2 to TrueNAS-11.1-U6.1
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No
  • Migration Needed deleted (No)
  • Hide from ChangeLog deleted (No)
  • Support Department Priority deleted (0)

This will also appear in the Changelog for FN 11.1-U7 and FN 11.2-BETA3.

#11 Updated by Dru Lavigne 8 months ago

  • Subject changed from Cannot disjoin ldap when ldap is offline to Add timeout for LDAP status calls

#13 Updated by Bonnie Follweiler 7 months ago

  • Status changed from Ready for Testing to Passed Testing
  • Needs QA changed from Yes to No

Passed testing in TrueNAS-11.1-U6-INTERNAL11

#14 Updated by Dru Lavigne 7 months ago

  • Status changed from Passed Testing to Done

Also available in: Atom PDF