Project

General

Profile

Bug #29785

Add a worker process to use with non thread-safe modules

Added by William Grzybowski over 1 year ago. Updated about 1 year ago.

Status:
Done
Priority:
No priority
Assignee:
William Grzybowski
Category:
Middleware
Target version:
Seen in:
Severity:
Med High
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Reporting this as a bug from: https://forums.freenas.org/index.php?threads/unauthorized-reboot-11-1-release.60260

NOTE that some users in that thread are reporting their systems ARE actually rebooting. Thankfully for me that is NOT the case. For me, it seems the middleware python PID is crashing & dumping core which is in turn touching the /data/sentinels/unauthorized-reboot file which is triggering the reboot email. So far since upgrading various systems from various prior FN releases (9.10 & 10.x I believe) I have seen this on 3 different systems - 2 of which had been running for ~1 year with no such issues/emails and the third had also been running quietly for ~6months.

Copying & pasting my latest forum post with the details:
Mine still aren't actually rebooting, but I received another email from a new system today. Tracked it down to this:

[CODE]Mar 6 03:45:03 hostname1 kernel: pid 262 (python3.6), uid 0: exited on signal 11 (core dumped)[/CODE]
So I checked for python processes started around that time and found this:

[CODE]
root 13531 0.0 0.1 74884 65288 - I 03:45 0:02.68 /usr/local/bin/python /etc/find_alias_for_smtplib.py -FCronDaemon -odi -oem -oi -t (python3.6)
root 13665 0.0 0.2 188560 138304 - S 03:45 0:12.82 python3.6: middlewared (python3.6)[/CODE]
Went back and found the same thing on another system:
[CODE]Feb 28 12:19:08 hostname2 kernel: pid 250 (python3.6), uid 0: exited on signal 11 (core dumped)[/CODE]
And again:
[CODE]root 42507 0.0 0.2 194096 136816 - S Wed12 22:34.33 python3.6: middlewared (python3.6)[/CODE]
hey [USER=21716]@Scharbag[/USER] any updates on your bug? I still can't see it. I'm about ready to log one of mine own too because my hosts thankfully aren't actually rebooting ... they're just emailing me telling me they did erroneously.

The bug referenced above is 28028, but I still get a 403 forbidden error when I try to view it.


Related issues

Copied from FreeNAS - Bug #29133: Fix" Unauthorized Reboot" emails when system did not actually rebootDone2018-03-06

Associated revisions

Revision 1c0d1646 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): allow services to run in another process

Ticket: #29785

Revision df4f777c (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/jail): make jail service run in a different process

Thats because py-libzfs is not thread-safe (and its what iocage uses)

Ticket: #29785

Revision 104464d9 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): do not try to close twice

At that time another fd with same number could be in use.

Ticket: #29785

Revision 993fdda3 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): set LC_ALL for the worker

Ticket: #29785

Revision 16d2b93d (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): pep8

Ticket: #29785

Revision 3b9a7760 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): pep8

Ticket: #29785

Revision 767272e9 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): lets try to use a ProcessPoolExecutor for the worker

Ticket: #29785

Revision 0b76e713 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): minor fixes

Ticket: #29785

Revision 6752473e (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): run init() when worker process is spawned

Ticket: #29785

Revision 52132249 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): calling job in a worker

Ticket: #29785

Revision 1b650862 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): pep8, typos and code duplication fixes

Ticket: #29785

Revision 3451a68e (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): allow services to run in another process

Ticket: #29785

Revision d4da49b0 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared/jail): make jail service run in a different process

Thats because py-libzfs is not thread-safe (and its what iocage uses)

Ticket: #29785

Revision 00f337ac (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): do not try to close twice

At that time another fd with same number could be in use.

Ticket: #29785

Revision 0c0c4c70 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): set LC_ALL for the worker

Ticket: #29785

Revision 217d6f6d (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): pep8

Ticket: #29785

Revision 6d6e2760 (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): lets try to use a ProcessPoolExecutor for the worker

Ticket: #29785

Revision 8f3d7fd6 (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): minor fixes

Ticket: #29785

Revision 61e118dc (diff)
Added by William Grzybowski over 1 year ago

feat(middlewared): run init() when worker process is spawned

Ticket: #29785

Revision c1f0ef3a (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): calling job in a worker

Ticket: #29785

Revision 83e73c5b (diff)
Added by William Grzybowski over 1 year ago

fix(middlewared): pep8, typos and code duplication fixes

Ticket: #29785

History

#1 Updated by William Grzybowski over 1 year ago

  • Copied from Bug #29133: Fix" Unauthorized Reboot" emails when system did not actually reboot added

#2 Updated by William Grzybowski over 1 year ago

T F, I have copied this bug so we can make a proper fix for 11.2.

For U3 we will only be getting rid of the emails when system has not actually rebooted.

#3 Updated by William Grzybowski over 1 year ago

  • Private changed from Yes to No

#4 Updated by William Grzybowski over 1 year ago

  • Status changed from Not Started to In Progress

#5 Updated by William Grzybowski over 1 year ago

  • Reason for Blocked deleted (Waiting for feedback)

#6 Updated by William Grzybowski over 1 year ago

  • Status changed from In Progress to Done
  • Needs Doc changed from Yes to No

#7 Updated by Dru Lavigne over 1 year ago

  • Subject changed from Middlewared process is segfaulting to Add a worker process to use with non thread-safe modules
  • Target version changed from 11.2-RC2 to 11.2-BETA1
  • Needs Merging changed from Yes to No

#8 Updated by Dru Lavigne over 1 year ago

  • File deleted (python3.6.core.gz)

#9 Updated by Dru Lavigne over 1 year ago

  • File deleted (t01-python.core.gz)

#10 Updated by Dru Lavigne over 1 year ago

  • File deleted (t02-python.core.gz)

#11 Updated by Dru Lavigne about 1 year ago

  • Status changed from Done to Ready for Testing

#12 Updated by William Grzybowski about 1 year ago

  • Needs QA changed from Yes to No

#13 Updated by Dru Lavigne about 1 year ago

  • Status changed from Ready for Testing to Done

Also available in: Atom PDF