Project

General

Profile

Bug #73930

django.db.utils.OperationalError: database is locked errors

Added by Peter Sprague 6 months ago. Updated 6 months ago.

Status:
Closed
Priority:
No priority
Assignee:
Release Council
Category:
Middleware
Target version:
Seen in:
Severity:
New
Reason for Closing:
Not Applicable
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Recently upgraded from 11.1 --> 11.2
My system has slowed to a crawl with endless traceback errors all linked to sqlite3 locking.

Based on a discussion here: https://stackoverflow.com/questions/3172929/operationalerror-database-is-locked
This seems to be a concurrency issue

Example alert error:
-----------------------------
Gone alerts:
  • Unable to run alert source 'smartd'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
    File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
    execute = self.locked_retry(Database.Cursor.execute, query, params)
    File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
    raise e
    File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
    rv = method(self, *args, **kwargs)
    sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in run_source
alerts = (await alert_source.check()) or []
File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
return await self.middleware.run_in_thread(self.check_sync)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/smartd.py", line 12, in check_sync
("srv_enable", "=", True)]):
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
return fut.result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
return await run_method(methodobj, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
select=options.get('select'),
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize
for i in qs:
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter

self._fetch_all()
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in iter
results = compiler.execute_sql()
File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
execute = self.locked_retry(Database.Cursor.execute, query, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
raise e
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
rv = method(self, *args, **kwargs)
django.db.utils.OperationalError: database is locked

  • Unable to run alert source 'Replication'
    Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
    File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
    execute = self.locked_retry(Database.Cursor.execute, query, params)
    File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
    raise e
    File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
    rv = method(self, *args, **kwargs)
    sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in run_source
alerts = (await alert_source.check()) or []
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/replication.py", line 10, in check
for replication in await self.middleware.call("replication.query", [["enabled", "=", True]]):
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call
return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1049, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 266, in query
'datastore.query', self._config.datastore, [], datastore_options
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call
return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
return await run_method(methodobj, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
select=options.get('select'),
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize
for i in qs:
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter

self._fetch_all()
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in iter
results = compiler.execute_sql()
File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
execute = self.locked_retry(Database.Cursor.execute, query, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
raise e
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
rv = method(self, *args, **kwargs)
django.db.utils.OperationalError: database is locked

History

#1 Updated by Dru Lavigne 6 months ago

  • Status changed from Unscreened to Blocked
  • Reason for Blocked set to Need additional information from Author

Peter: that's a classic sign of a dying boot device. Try this: save your config, perform a fresh install of 11.2 to a different USB stick or SSD, restore your config, and report back after a day or so of running to let us know if that resolves your issue.

#2 Updated by Peter Sprague 6 months ago

Tried to perform fresh install with 2 different pairs of fresh usb flash drives - no go. Moved to a pair of 128gb ssd mirrored with a fresh install. Installed without error messages. FreeNAS 11.2-RELEASE now seems to be operational without the alerts. Will post if something changes. Thanks

#3 Updated by Dru Lavigne 6 months ago

  • Status changed from Blocked to Closed
  • Target version changed from Backlog to N/A
  • Reason for Closing set to Not Applicable
  • Reason for Blocked deleted (Need additional information from Author)

Thanks for the update Peter. Please leave a comment on this ticket if the issue reappears.

Also available in: Atom PDF