Project

General

Profile

Bug #43811

Fix deadlock on thread pool

Added by Waqar Ahmed about 2 years ago. Updated about 2 years ago.

Status:
Done
Priority:
No priority
Assignee:
William Grzybowski
Category:
Middleware
Target version:
Severity:
High
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

This issue has been seen on a machine in TN Office ( realmini ), it seems randomly middlewared becomes totally unresponsive and locks itself out. The issue is resolved for the time being by restarting middlewared but the root cause should be investigated and patched.

Risk
This is already a high risk ticket and is causing total failure of middlewared.

Acceptance Criteria
Running
"seq 1 25|while read i; do midclt call vm.get_available_memory &; done;"

should finish as expected and not lock up middlewared.


Related issues

Related to FreeNAS - Bug #43789: Sentry responded with an API error: RateLimited(None)Closed

Associated revisions

Revision c92650c2 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_io_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision 3d9da0ea (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_io_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision 39fcd1d3 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_io_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision 1a2664f0 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_io_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision 2358dac5 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_io_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision 1f3450c6 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision 274e4555 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision f9f38d0b (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared): deadlock on thread pool Threads to handle websocket connection are gated on `__threadpool`. Any other calls should use `run_in_thread` as that launches its own thread and does not cause deadlock waiting another thread to finish in the pool (which could happen on the stack call, e.g. service.foo calls something in using the thread pool and something also uses the thread pool. If service.foo is called many times before each thread finishes we will have a deadlock) Ticket: #43811

Revision 4a5d8a07 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared/disk): boot.get_disks is no longer an async generator Ticket: #43811

Revision 9fdb26e4 (diff)
Added by William Grzybowski about 2 years ago

fix(middlewared/disk): boot.get_disks is no longer an async generator Ticket: #43811

History

#1 Updated by William Grzybowski about 2 years ago

  • Status changed from Unscreened to Not Started
  • Severity changed from New to High

#2 Updated by Dru Lavigne about 2 years ago

  • Target version changed from Backlog to 11.2-BETA3

#3 Updated by Bug Clerk about 2 years ago

  • Status changed from Not Started to In Progress

#4 Updated by William Grzybowski about 2 years ago

  • Description updated (diff)

#5 Updated by William Grzybowski about 2 years ago

  • Related to Bug #43789: Sentry responded with an API error: RateLimited(None) added

#6 Updated by Bug Clerk about 2 years ago

  • Status changed from In Progress to Ready for Testing

#7 Updated by Dru Lavigne about 2 years ago

  • Subject changed from Middlewared becomes totally unresponsive to Fix deadlock on thread pool
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#8 Updated by Bonnie Follweiler about 2 years ago

  • Status changed from Ready for Testing to Blocked
  • Reason for Blocked set to Waiting for feedback

#11 Updated by Bonnie Follweiler about 2 years ago

  • Status changed from Blocked to Passed Testing
  • Reason for Blocked deleted (Waiting for feedback)
  • Needs QA changed from Yes to No

Test passed in FreeNAS-11.2-MASTER-201808310859

#12 Updated by Dru Lavigne about 2 years ago

  • Status changed from Passed Testing to Done

#13 Updated by Bug Clerk about 2 years ago

  • Status changed from Done to In Progress

#14 Updated by Bug Clerk about 2 years ago

  • Status changed from In Progress to Ready for Testing

#15 Updated by William Grzybowski about 2 years ago

  • Status changed from Ready for Testing to Done

Also available in: Atom PDF