Project

General

Profile

Bug #82843

FreeNAS_LDAP_Directory[ERROR]: {'desc': 'Size limit exceeded'}

Added by George - over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
No priority
Assignee:
Andrew Walker
Category:
Middleware
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No
Tags:

Description

Trying to turn on LDAP on a system with a big LDAP directory gives me:

ldap.SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'}
Mar 19 19:05:12 nas-backup /cachetool.py: [common.freenasldap:195] FreeNAS_LDAP_Directory[ERROR]: {'desc': 'Size limit exceeded'}

among other errors. FreeNAS tries to import users (I think) for several minutes, then throws error.

A more complete log below.

Mar 19 19:02:21 nas-backup uwsgi: [common.frenascache:307] FreeNAS_LDAP_QueryCache.__init__: enter
Mar 19 19:02:21 nas-backup uwsgi: [common.frenascache:89] FreeNAS_BaseCache._init__: enter
Mar 19 19:02:21 nas-backup uwsgi: [common.frenascache:110] FreeNAS_BaseCache._init__: cachedir = /var/tmp/.cache/.query
Mar 19 19:02:21 nas-backup uwsgi: [common.frenascache:113] FreeNAS_BaseCache._init__: cachefile = /var/tmp/.cache/.query/.cache.db
Mar 19 19:02:21 nas-backup uwsgi: [common.frenascache:115] FreeNAS_BaseCache._init__: leave
Mar 19 19:02:21 nas-backup uwsgi: [common.frenascache:315] FreeNAS_LDAP_QueryCache.__init__: leave
Mar 19 19:02:21 nas-backup uwsgi: [common.freenasldap:351] FreeNAS_LDAP_Directory.open: connection open
Mar 19 19:02:22 nas-backup ldapsearch: DIGEST-MD5 common mech free
Mar 19 19:02:31 nas-backup ldapsearch: DIGEST-MD5 common mech free
Mar 19 19:02:51 nas-backup ldapsearch: DIGEST-MD5 common mech free
Mar 19 19:02:53 nas-backup ldapsearch: DIGEST-MD5 common mech free
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:307] FreeNAS_LDAP_QueryCache.__init__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:89] FreeNAS_BaseCache._init__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:110] FreeNAS_BaseCache._init__: cachedir = /var/tmp/.cache/.query
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:113] FreeNAS_BaseCache._init__: cachefile = /var/tmp/.cache/.query/.cache.db
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:115] FreeNAS_BaseCache._init__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:315] FreeNAS_LDAP_QueryCache.__init__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:508] FreeNAS_UserCache.__new__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:466] FreeNAS_Directory_LocalUserCache.__new__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:229] FreeNAS_LDAP_LocalUserCache.__init__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:89] FreeNAS_BaseCache._init__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:110] FreeNAS_BaseCache._init__: cachedir = /var/tmp/.cache/.ldap/.ldap/.local/.users
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:113] FreeNAS_BaseCache._init__: cachefile = /var/tmp/.cache/.ldap/.ldap/.local/.users/.cache.db
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:115] FreeNAS_BaseCache._init__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:237] FreeNAS_LDAP_LocalUserCache.__init__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:481] FreeNAS_Directory_LocalUserCache.__new__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:519] FreeNAS_UserCache.__new__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:424] FreeNAS_Directory_UserCache.__new__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:203] FreeNAS_LDAP_UserCache.__init__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:89] FreeNAS_BaseCache._init__: enter
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:110] FreeNAS_BaseCache._init__: cachedir = /var/tmp/.cache/.ldap/.ldap/.users
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:113] FreeNAS_BaseCache._init__: cachefile = /var/tmp/.cache/.ldap/.ldap/.users/.cache.db
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:115] FreeNAS_BaseCache._init__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:211] FreeNAS_LDAP_UserCache.__init__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.frenascache:439] FreeNAS_Directory_UserCache.__new__: leave
Mar 19 19:02:58 nas-backup /cachetool.py: [common.freenasldap:2467] FreeNAS_LDAP_Users.__get_users: LDAP users not in cache
Mar 19 19:02:59 nas-backup /cachetool.py: [common.freenasldap:351] FreeNAS_LDAP_Directory.open: connection open
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:307] FreeNAS_LDAP_QueryCache.__init__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:89] FreeNAS_BaseCache._init__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:110] FreeNAS_BaseCache._init__: cachedir = /var/tmp/.cache/.query
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:113] FreeNAS_BaseCache._init__: cachefile = /var/tmp/.cache/.query/.cache.db
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:115] FreeNAS_BaseCache._init__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:315] FreeNAS_LDAP_QueryCache.__init__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:525] FreeNAS_GroupCache.__new__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:487] FreeNAS_Directory_LocalGroupCache.__new__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:242] FreeNAS_LDAP_LocalGroupCache.__init__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:89] FreeNAS_BaseCache._init__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:110] FreeNAS_BaseCache._init__: cachedir = /var/tmp/.cache/.ldap/.ldap/.local/groups
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:113] FreeNAS_BaseCache._init__: cachefile = /var/tmp/.cache/.ldap/.ldap/.local/groups/.cache.db
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:115] FreeNAS_BaseCache._init__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:250] FreeNAS_LDAP_LocalGroupCache.__init__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:502] FreeNAS_Directory_LocalGroupCache.__new__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:536] FreeNAS_GroupCache.__new__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:445] FreeNAS_Directory_GroupCache.__new__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:216] FreeNAS_LDAP_GroupCache.__init__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:89] FreeNAS_BaseCache._init__: enter
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:110] FreeNAS_BaseCache._init__: cachedir = /var/tmp/.cache/.ldap/.ldap/.groups
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:113] FreeNAS_BaseCache._init__: cachefile = /var/tmp/.cache/.ldap/.ldap/.groups/.cache.db
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:115] FreeNAS_BaseCache._init__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:224] FreeNAS_LDAP_GroupCache.__init__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.frenascache:460] FreeNAS_Directory_GroupCache.__new__: leave
Mar 19 19:05:05 nas-backup /cachetool.py: [common.freenasldap:2756] FreeNAS_LDAP_Groups.__get_groups: LDAP groups not in cache
Mar 19 19:05:05 nas-backup /cachetool.py: [common.freenasldap:351] FreeNAS_LDAP_Directory.open: connection open
Mar 19 19:05:12 nas-backup /cachetool.py: [common.freenasldap:193] FreeNAS_LDAP_Directory[ERROR]: An LDAP Exception occured
Traceback (most recent call last):
File "/usr/local/www/freenasUI/common/freenasldap.py", line 474, in _search
type, data = self._handle.result(id, 0)
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 704, in result
resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 708, in result2
resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 715, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 722, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/local/lib/python3.6/site-packages/ldap/lda
Mar 19 19:05:12 nas-backup /cachetool.py: pobject.py", line 309, in _ldap_call
reraise(exc_type, exc_value, exc_traceback)
File "/usr/local/lib/python3.6/site-packages/ldap/compat.py", line 42, in reraise
raise exc_value
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 293, in _ldap_call
result = func(args,*kwargs)
ldap.SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'}
Mar 19 19:05:12 nas-backup /cachetool.py: [common.freenasldap:195] FreeNAS_LDAP_Directory[ERROR]: {'desc': 'Size limit exceeded'}
Traceback (most recent call last):
File "/usr/local/www/freenasUI/common/freenasldap.py", line 474, in _search
type, data = self._handle.result(id, 0)
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 704, in result
resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 708, in result2
resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 715, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 722, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/local/lib/python3.6/site-packages/ld
Mar 19 19:05:12 nas-backup /cachetool.py: ap/ldapobject.py", line 309, in _ldap_call
reraise(exc_type, exc_value, exc_traceback)
File "/usr/local/lib/python3.6/site-packages/ldap/compat.py", line 42, in reraise
raise exc_value
File "/usr/local/lib/python3.6/site-packages/ldap/ldapobject.py", line 293, in _ldap_call
result = func(args,*kwargs)
ldap.SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'}
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] Traceback (most recent call last):
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 251, in dispatch
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 482, in dispatch
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] response = method(request, **kwargs)
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 319, in put_list
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] return self.put_detail(request, **kwargs)
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 332, in put_detail
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] updated_bundle = self.obj_update(bundle=bundle, **self.remove_api_resource_names(kwargs))
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 2236, in obj_update
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] return self.save(bundle, skip_errors=skip_errors)
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 415, in save
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] form.save()
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "./freenasUI/directoryservice/forms.py", line 959, in save
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] started = notifier().restart("ldap", timeout=_fs().directoryservice.ldap.timeout.restart)
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "./freenasUI/middleware/notifier.py", line 230, in restart
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] **kwargs,
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "./freenasUI/middleware/notifier.py", line 230, in restart
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] **kwargs,
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 447, in call
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] raise CallTimeout("Call timeout")
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] middlewared.client.client.CallTimeout: Call timeout
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] Traceback (most recent call last):
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 219, in wrapper
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] response = callback(request, *args, **kwargs)
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 450, in dispatch_list
Mar 19 19:05:23 nas-backup uwsgi: [api.utils:12] return self.dispatch('list', request, **kwargs)

History

#1 Updated by George - over 2 years ago

obviously I meant to say "big" LDAP directory.

#2 Updated by Dru Lavigne over 2 years ago

  • Reason for Blocked set to Need additional information from Author

How big is big?

#3 Updated by Dru Lavigne over 2 years ago

  • Description updated (diff)

#4 Updated by George - over 2 years ago

About 200k entries

#5 Updated by Dru Lavigne over 2 years ago

  • Assignee changed from Release Council to William Grzybowski
  • Reason for Blocked deleted (Need additional information from Author)

#6 Updated by William Grzybowski over 2 years ago

  • Assignee changed from William Grzybowski to Andrew Walker
  • Target version changed from Backlog to 11.2-U4

#9 Updated by Andrew Walker over 2 years ago

Hi George. Sorry it has taken a bit to respond. There are some scaling issues with our caching of LDAP users. Short-term solution will be to introduce a checkbox to disable caching of LDAP users. Long term will be a rewrite of how this caching is handled. I'll try to have this ready for U4.

#10 Updated by George - over 2 years ago

Thanks Andrew...

Actually, for my usage case, I don't want the whole LDAP directory to be imported. We have a big directory, but our group only wants a select few users to be imported, mainly for UID mapping to NFS.

If I can't do it from FreeNAS, I'll probably write a script to extract the users we care about from LDAP. Unsure what I'm going to do after that to map the LDAP uids for NFSv4 access.

#11 Updated by Andrew Walker over 2 years ago

  • Status changed from Unscreened to Screened

George - wrote:

Thanks Andrew...

Actually, for my usage case, I don't want the whole LDAP directory to be imported. We have a big directory, but our group only wants a select few users to be imported, mainly for UID mapping to NFS.

If I can't do it from FreeNAS, I'll probably write a script to extract the users we care about from LDAP. Unsure what I'm going to do after that to map the LDAP uids for NFSv4 access.

The issue is importing your ldap directory into the django cache and having "enum users" enabled in SSSD. The checkbox will basically apply some optimizations for large environments. It should be more stable. I'll send you a patch (probably next week) for testing in your environment.

#12 Updated by Jaron Parsons over 2 years ago

  • Status changed from Screened to Closed

Also available in: Atom PDF