Project

General

Profile

Bug #56247

Log starting and stopping jails on boot

Added by David Davtian almost 2 years ago. Updated over 1 year ago.

Status:
Done
Priority:
No priority
Assignee:
Brandon Schneider
Category:
Middleware
Target version:
Seen in:
Severity:
Med High
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

I can't seem to be able to get a newly created jail to start on boot. I have done the following on creating an iocage based jail. It creates and starts just fine, but on a system reboot it doesn't come up automatically. I need to manually start them up. Any help is very much appreciated.

Versions

FreeNAS-11.2-RC1
Iocage Version: 1.0 ALPHA 1

Commands

iocage create -n dave -r 11.2-RELEASE ip4_addr="vnet0|192.168.25.100/24" defaultrouter="192.168.25.1" vnet="on" allow_raw_sockets="1" boot="on" 
dave successfully created!
* Starting dave
  + Started OK
  + Configuring VNET OK
  + Starting services OK

I have also added the following rc.conf tunable but had no success.

iocage_enable=YES

I also reported the issue with iocage and was asked to open the issue here instead.

Thanks

Associated revisions

Revision 577dbd66 (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

Revision 8be4f01f (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

Revision b012285b (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

Revision 57beffc3 (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

Revision 2603af31 (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

Revision bb28c8e4 (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

Revision 137e5bce (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

Revision efab07bf (diff)
Added by Brandon Schneider almost 2 years ago

fix(jail/rc): Log starting and stopping jails on boot Requires PR for Ticket #51997 to be merged first. - Return True when done Ticket: #56247

History

#1 Updated by Brandon Schneider almost 2 years ago

  • Category changed from Services to Middleware
  • Status changed from Unscreened to In Progress
  • Assignee changed from Release Council to Brandon Schneider
  • Severity changed from New to Med High

Grabbing this as I instructed David to file this here.

#2 Updated by Brandon Schneider almost 2 years ago

  • Status changed from In Progress to Blocked
  • Reason for Blocked set to Need additional information from Author

David: Can you paste a debug please? I cannot replicate this here as the jail is actually starting on boot for me. The middleware needs to work through things until it hits the jail portion, so it may not be immediate. Remove that tunable as it can cause conflicts with both trying to start the jail.

#3 Updated by Bug Clerk almost 2 years ago

  • Status changed from Blocked to In Progress

#4 Updated by Brandon Schneider almost 2 years ago

  • Status changed from In Progress to Blocked
  • Target version changed from Backlog to 11.2-U2

PR: https://github.com/freenas/freenas/pull/2044
STABLE PR: https://github.com/freenas/freenas/pull/2045

This will log that the start and stop actions were at least run and successful.

These will help future issues like David's, in the meantime, the status remains blocked until we can confirm if there is indeed an issue on David's system or if the tunable was interfering/boot delay as the system warms up.

#5 Updated by David Davtian almost 2 years ago

Brandon, thank you for your quick responses. Your help is appreciated. I tried removing the following tunable using the UI (new UI) and the spinner keeps spinning and I am not able to make the edit. I see it here, but removing it here and rebooting places it right back in. Do you know which file I can edit to remove it permanently?

cat /tmp/rc.conf.freenas | grep iocage
iocage_enable="YES" 
root@e5-freenas:~ #

I should have been a bit more clear when I opened the bug. Some jails are indeed starting on boot, but as mentioned earlier the newly created one from scratch is not. Also if you notice below 2/5 are down. I have searched high and low trying to do a diff between the rc configs and thus far not able to see the difference.

root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 3   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 2   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 1   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+

As for the debug requested, happy to provide it if you can provide me the step(s)

Thank you again

#6 Updated by Brandon Schneider almost 2 years ago

David: That may be a new UI bug, in the meantime try the legacy UI to remove the tunable. It shouldn't exist by default.

root@freenas[~]# cat /tmp/rc.conf.freenas| grep iocage
root@freenas[~]# 

As for the debug, sure! System -> Advanced -> Save debug should allow you to attach it to this ticket.

#7 Updated by David Davtian almost 2 years ago

  • File debug.tgz added

#8 Updated by Brandon Schneider almost 2 years ago

David: Ooft! Ok, what we're going to do is modifying the database. So make a backup of /data/freenas-v1.db first.

Something like:

cp /data/freenas-v1.db /data/freenas-v1-2018_11_07_backup.db

And then we get into the fun part:

root@freenas[~]# sqlite3 /data/freenas-v1.db 'SELECT * FROM system_tunable'
1|YES|rc||1|iocage_enable

As you can see, the iocage_enable tunable is set to ID 1 in the table. Since you may have other tunables you wish to preserve, we're going to delete just that ID. Make sure your ID in the next command matches the output your system gives you above.

Deletion time:

root@freenas[~]# sqlite3 /data/freenas-v1.db 'DELETE FROM system_tunable WHERE id=1'

Since my ID was 1, I specify 1 here. Double check again you are using the correct ID.

Afterwards you can see it is indeed removed :)

root@freenas[~]# sqlite3 /data/freenas-v1.db 'SELECT * FROM system_tunable'         
root@freenas[~]# 

Reboot your system and verify everything is OK.

#9 Updated by David Davtian almost 2 years ago

Brandon,

I am back in business, apologies for putting you through this :-( The docs do in fact say, be careful, you will void your warranty :-)

Let me know if you need additional debugs from me or any other info in regards to the Iocage startup issue.

Thanks again
Dave

#10 Updated by David Davtian almost 2 years ago

  • Description updated (diff)

David Davtian wrote:

I can't seem to be able to get a newly created jail to start on boot. I have done the following on creating an iocage based jail. It creates and starts just fine, but on a system reboot it doesn't come up automatically. I need to manually start them up. Any help is very much appreciated.

Versions

[...]

Commands

[...]

I have also added the following rc.conf tunable but had no success.

[...]

I also reported the issue with iocage and was asked to open the issue here instead.

Thanks

#11 Updated by Brandon Schneider almost 2 years ago

No problem! We got some more information out of this so that’s a plus.

Just to confirm, your issue is now solved?

#12 Updated by David Davtian almost 2 years ago

Brandon,

The issue with me adding the tunable and the new UI spinning endlessly under Tunables and with the same token breaking the legacy UI is resolved per the removal of the item from the DB.

The issue with a newly created iocage and the ones listed below still stands. I can't figure out what is the difference is in these jails, some start and some don't. And again anything new created doesn't auto start either.

root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 3   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 2   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 1   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+

Thanks
Dave

#13 Updated by William Grzybowski almost 2 years ago

Does running:

"midclt call jail.start_on_boot"

from SSH makes the jails start?

If not, would "iocage start --rc" start them?

#14 Updated by David Davtian almost 2 years ago

William,

root@e5-freenas:~ # midclt call jail.start_on_boot
null

Did not bring up any of my Iocage jails which had problems starting up before.

on_boot=on

This command, does indeed bring up the down jails to an up state, but upon reboot they remain down, in my case plexmediaserver and openvpn

iocage start --rc
e5-freenas% iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 3   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 2   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 1   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+

What is so special about the last three as they come up just fine on reboot :-)

#15 Updated by Brandon Schneider almost 2 years ago

David: Sorry it's a bit hard to follow, iocage start --rc does bring up your jails but midclt call jail.start_on_boot doesn't? Or the other way around?

#16 Updated by David Davtian almost 2 years ago

Brandon,

Apologies, I confused with my answers :-) Please see the steps below

root@e5-freenas:~ #
root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 3   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 2   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 1   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+

root@e5-freenas:~ # midclt call jail.start_on_boot
null

root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| 5   | openvpn         | up    | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| 4   | plexmediaserver | up    | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 3   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 2   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 1   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+

Then I reboot the box and list the jails again, top two are down.

Welcome to FreeNAS
e5-freenas% iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 3   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 2   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 1   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+
e5-freenas%

#17 Updated by Brandon Schneider almost 2 years ago

David: Very odd, can you try this:

iocage stop --rc
iocage start --rc

Does anything get printed?

#18 Updated by David Davtian almost 2 years ago

Brandon,

Issuing those commands did stop/start the jails. Although during the startup process first attempt stalled the CLI, I had to Control-C it and try again, at that point all started just fine.

root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 3   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 2   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 1   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+
root@e5-freenas:~ # iocage stop --rc
root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| -   | radarr          | down  | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| -   | rclone          | down  | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| -   | sabnzbd         | down  | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+
root@e5-freenas:~ # iocage start --rc
^C
Aborted!
root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| -   | openvpn         | down  | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| -   | plexmediaserver | down  | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 6   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 5   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 4   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+
root@e5-freenas:~ # iocage start --rc
root@e5-freenas:~ # iocage list
+-----+-----------------+-------+--------------+---------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4      |
+=====+=================+=======+==============+===============+
| 8   | openvpn         | up    | 11.2-RELEASE | 192.168.25.56 |
+-----+-----------------+-------+--------------+---------------+
| 7   | plexmediaserver | up    | 11.2-RELEASE | 192.168.25.50 |
+-----+-----------------+-------+--------------+---------------+
| 6   | radarr          | up    | 11.2-RELEASE | 192.168.25.57 |
+-----+-----------------+-------+--------------+---------------+
| 5   | rclone          | up    | 11.2-RELEASE | 192.168.25.67 |
+-----+-----------------+-------+--------------+---------------+
| 4   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61 |
+-----+-----------------+-------+--------------+---------------+

#19 Updated by Brandon Schneider almost 2 years ago

David: Odd, how long did the stall last? This is all very bizarre behavior!

#20 Updated by David Davtian almost 2 years ago

Brandon, well I waited about 40 seconds or so and then followed through with a Control-C and then I issued

iocage start --rc again which started the jails and prompt returned

#21 Updated by Brandon Schneider almost 2 years ago

  • Status changed from Blocked to In Progress

That's way long enough, thanks for the information! I'll see if I can replicate it with one of your jail configurations.

#22 Updated by Brandon Schneider almost 2 years ago

  • Reason for Blocked deleted (Need additional information from Author)

#23 Updated by David Davtian almost 2 years ago

Brandon, as you are putting in the hard work and trying to reproduce this issue, what can I do from my side? I mean I have a working iocage that starts up on boot and another that doesn't Which files/configs/lines I can look at to do a diff between working vs. none working?

#24 Updated by Brandon Schneider almost 2 years ago

David: Appreciate the offer! Comparing the jails config.json is a good start. Then narrowing down where on each host the settings differ.

#25 Updated by David Davtian almost 2 years ago

Brandon/William,

I believe I have found the issue. These are the steps to reproduce.

1. Create an iocage jail, nothing interesting with this

root@e5-freenas:~ # iocage create -n dave -r 11.2-RELEASE ip4_addr="vnet0|192.168.25.100/24" defaultrouter="192.168.25.1" vnet="on" allow_raw_sockets="1" boot="on" 
dave successfully created!
* Starting dave
  + Started OK
  + Configuring VNET OK
  + Starting services OK

2. At this point if I reboot the system, the jail doesn't auto start
3. Stopped the jail

iocage stop dave

4. The priority by default it's set to

99
cat config.json | grep priority
    "priority": "99",

5. Changed it to a much lower value, i.e

12

6. Reboot the box, jail comes up and making this change for my other none auto starting jails on boot has solved my issues.

e5-freenas% iocage list
+-----+-----------------+-------+--------------+----------------+
| JID |      NAME       | STATE |   RELEASE    |      IP4       |
+=====+=================+=======+==============+================+
| 3   | dave            | up    | 11.2-RELEASE | 192.168.25.100 |
+-----+-----------------+-------+--------------+----------------+
| 2   | openvpn         | up    | 11.2-RELEASE | 192.168.25.56  |
+-----+-----------------+-------+--------------+----------------+
| 1   | plexmediaserver | up    | 11.2-RELEASE | 192.168.25.50  |
+-----+-----------------+-------+--------------+----------------+
| 6   | radarr          | up    | 11.2-RELEASE | 192.168.25.57  |
+-----+-----------------+-------+--------------+----------------+
| 5   | rclone          | up    | 11.2-RELEASE | 192.168.25.67  |
+-----+-----------------+-------+--------------+----------------+
| 4   | sabnzbd         | up    | 11.2-RELEASE | 192.168.25.61  |
+-----+-----------------+-------+--------------+----------------+

Hope this helps narrow down the issue in code.

Thank you both.

#26 Updated by Bug Clerk almost 2 years ago

  • Status changed from In Progress to Ready for Testing

#27 Updated by Bug Clerk almost 2 years ago

  • Target version changed from 11.2-U2 to 11.3

#29 Updated by William Grzybowski almost 2 years ago

  • Status changed from Ready for Testing to In Progress
  • Target version changed from 11.3 to 11.2-RELEASE

#30 Updated by Dru Lavigne almost 2 years ago

  • Subject changed from Iocage won't start on boot to Log starting and stopping jails on boot
  • Status changed from In Progress to Ready for Testing
  • Target version changed from 11.2-RELEASE to 11.2-RC2
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#31 Updated by Dru Lavigne almost 2 years ago

  • File deleted (debug.tgz)

#32 Updated by Joe Maloney almost 2 years ago

  • Private changed from No to Yes

To test ssh into FreeNAS, run this from CLI before starting, and stopping jails:

tail -f /var/log/middlewared.log

If that file is populated with output showing the jail starting, and stopping then it is working.

#33 Updated by William Grzybowski almost 2 years ago

Joe Maloney wrote:

To test ssh into FreeNAS, run this from CLI before starting, and stopping jails:

[...]

If that file is populated with output showing the jail starting, and stopping then it is working.

Thats about starting jails on boot. So it should just says jail started while looking at middlewared.log after the system has booted.

#34 Updated by Bonnie Follweiler almost 2 years ago

  • Status changed from Ready for Testing to Passed Testing
  • Needs QA changed from Yes to No

Test Passed in FreeNAS-11.2-INTERNAL48

#35 Updated by Dru Lavigne almost 2 years ago

  • Status changed from Passed Testing to Done

#36 Updated by Dru Lavigne over 1 year ago

  • Private changed from Yes to No

Also available in: Atom PDF