Project

General

Profile

Bug #52637

Catch any exceptions trying to load freebsd-version during iocage check_config

Added by Disk Didler almost 2 years ago. Updated almost 2 years ago.

Status:
Done
Priority:
No priority
Assignee:
Brandon Schneider
Category:
Middleware
Target version:
Seen in:
Severity:
Med High
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

The Jails section of the UI now has a new thing at the top I've never seen before.

"Activated Pool for Jail Manager: ARRAY" (my main pool name)
In the console, SSH in I get the following

root@freenas:~ # iocage list
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/list.py", line 81, in cli
    dataset_type, header, _long, _sort, plugin=plugins, quick=quick)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1150, in list
    quick=quick
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_list.py", line 75, in list_datasets
    _all = self.list_all(ds)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_list.py", line 153, in list_all
    conf = iocage_lib.ioc_json.IOCJson(mountpoint).json_load()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 407, in json_load
    conf = self.json_check_config(conf)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 910, in json_check_config
    with open(freebsd_version, "r") as r:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/iocage/releases/EMPTY/root/bin/freebsd-version'
root@freenas:~ #

However in a positive note, I can confirm that my jails all appear to be visible in /mnt/iocage/jails/ so I guess I just need to re-initialise this configuration, ideally without breaking them all?

Finally, I did have some major issues upgrading, therefore this fault may be unique to me. I'm unsure, but I did think I saw one other person on Reddit report something similar.

The issue # for my upgrade problems is #52083.

Unfortunately, I have rebooted and the issue persists.
Any suggestions would be nice.

I've taken a debug of the system, just after a reboot, won't upload unless requested.


Related issues

Has duplicate FreeNAS - Bug #56427: Jails stop working after update to 11.2 RC1Closed
Has duplicate FreeNAS - Bug #58131: iocage jails stoped working 11.2 RC1-1Closed

History

#1 Updated by Disk Didler almost 2 years ago

Disk Didler wrote:

The Jails section of the UI now has a new thing at the top I've never seen before.

"Activated Pool for Jail Manager: ARRAY" (my main pool name)

(There's also a config button, sorry I didn't make that clear. I can PROBABLY click this and it'll configure iocage for me, but I'm concerned it might 'wipe the index' or delete and recreate the existing folders already there)

Also, yes obviously I misdiagnosed my system as repaired the other day, I only checked that the WebUI came back with RC1 in it.

#2 Updated by Dru Lavigne almost 2 years ago

  • Reason for Blocked set to Need additional information from Author

I assume you mean the top section shown at http://doc.freenas.org/11.2/_images/jails.png? That has been there for awhile (it replaces the old Jails -> Configuration screen in the legacy UI). You only need to click it if you want to change or verify your jail's root.

If verifying/fixing the jail root does not resolve the issue, go ahead and attach the debug.

#3 Updated by Disk Didler almost 2 years ago

Dru Lavigne wrote:

I assume you mean the top section shown at http://doc.freenas.org/11.2/_images/jails.png? That has been there for awhile (it replaces the old Jails -> Configuration screen in the legacy UI). You only need to click it if you want to change or verify your jail's root.

If verifying/fixing the jail root does not resolve the issue, go ahead and attach the debug.

Yeah that's the one. No idea why though, it just de mapped itself.

I'll try it, I was hesitant.

#4 Updated by Disk Didler almost 2 years ago

Ok so I've clicked it, it comes up wth choose a pool or dataset for storage, the only one in the list is my main pool ("ARRAY")

I chose it in the drop down and hit activate which took 2 minutes.

It then went back to the same view as if nothing happened at all.

Clicked away and back to jail's section to refresh.

Not fixed sadly.

Will get a debug.

#5 Updated by Disk Didler almost 2 years ago

  • File debug-freenas-20181023032858.tgz added

debug

#6 Updated by Dru Lavigne almost 2 years ago

  • Category changed from OS to Middleware
  • Assignee changed from Release Council to William Grzybowski
  • Private changed from No to Yes
  • Seen in set to 11.2-RC1

#7 Updated by Dru Lavigne almost 2 years ago

  • Reason for Blocked deleted (Need additional information from Author)

#8 Updated by William Grzybowski almost 2 years ago

  • Assignee changed from William Grzybowski to Brandon Schneider
  • Target version changed from Backlog to 11.2-RC2
  • Severity changed from New to Med High

#9 Updated by Brandon Schneider almost 2 years ago

  • Status changed from Unscreened to In Progress

Disk: Did you create a manual dataset in releases called EMPTY? That would be the source of the problem

#11 Updated by Brandon Schneider almost 2 years ago

  • Status changed from In Progress to Done
  • Needs Merging changed from Yes to No

Hard to test, requires a very specific setup to trigger. Marking done.

Note to Disk: Remove that dataset ;) This will only trigger a real exception telling us the cause for other potential futures, but for you that's the solution.

#12 Updated by Dru Lavigne almost 2 years ago

  • Subject changed from After troubled upgrade to RC1, iocage jails are broken to Catch any exceptions trying to load freebsd-version during iocage check_config
  • Private changed from Yes to No
  • Needs QA changed from Yes to No
  • Needs Doc changed from Yes to No

#13 Updated by Dru Lavigne almost 2 years ago

  • File deleted (debug-freenas-20181023032858.tgz)

#14 Updated by Disk Didler almost 2 years ago

Brandon Schneider wrote:

Disk: Did you create a manual dataset in releases called EMPTY? That would be the source of the problem

Not that I'm aware of, no.

It's an old install though.

How can I fix and we prevent?

#15 Updated by Brandon Schneider almost 2 years ago

It’s odd because iocage should never create an EMPTY dataset, so just destroy it from the GUI or cli.

If it reappears we can drill down to see what is creating that. But the first report I’ve ever seen of it.

#16 Updated by Disk Didler almost 2 years ago

Brandon Schneider wrote:

It’s odd because iocage should never create an EMPTY dataset, so just destroy it from the GUI or cli.

If it reappears we can drill down to see what is creating that. But the first report I’ve ever seen of it.

sorry guys, doing this remotely, please ignore any spelling mistakes.

where do i find this apparent empty dataset?
i have expanded every single 'triangle' depicting nested data in the pools / storage section of the UI and i dont see a empty dataset?
or anything mentioning empty at all?... am i looking in the wrong spot

#17 Updated by Disk Didler almost 2 years ago

Anyone got any idea? I see no such dataset, looked again last night.

#18 Updated by Jurgen Segaert almost 2 years ago

Disk Didler: Did you upgrade some of your jails from warden, using the migrate_warden.py migration script?
Some versions of the script incorrectly set the release to EMPTY. See https://redmine.ixsystems.com/issues/27236#note-22, item 3

#19 Updated by Disk Didler almost 2 years ago

Jurgen Segaert wrote:

Disk Didler: Did you upgrade some of your jails from warden, using the migrate_warden.py migration script?
Some versions of the script incorrectly set the release to EMPTY. See https://redmine.ixsystems.com/issues/27236#note-22, item 3

Now this makes sense!
Am I crazy or is the choice of term "dataset" incorrect here then?
I thought that was the term for the release build?

So, how can I change this, when the jails are in a limbo state and I can't enter them?

Also, has the migration script been fixed to solve this issue?

#20 Updated by Brandon Schneider almost 2 years ago

While it set the release to EMPTY, it didn't create that dataset, it was creating an empty jail. Which is different, but possible it's linked to this in a different way.

You can use zfs destroy on the CLI Disk, the debug is gone otherwise I'd point to the specific dataset. Feel free to upload another one.

#21 Updated by Jurgen Segaert almost 2 years ago

Hmmm... I don't think there's an actual "empty jail" involved, but rather a converted warden plugin that was incorrectly marked as such.

Disk Didler: For the plugin(s) that you migrated from warden, manually check that jail's config file, typically located in /mnt/iocage/jails/<jail_name>/config.json. Look at the property called "release". If it has the value EMPTY, you're in trouble as that will make iocage look for a file called /mnt/iocage/releases/EMPTY/root/bin/freebsd-version to attempt to check the USERLAND_VERSION and it all goes downhill from there.
Source code: https://github.com/iocage/iocage/blob/master/iocage_lib/ioc_json.py#L894-L917

#22 Updated by Disk Didler almost 2 years ago

Jurgen Segaert wrote:

Hmmm... I don't think there's an actual "empty jail" involved, but rather a converted warden plugin that was incorrectly marked as such.

Disk Didler: For the plugin(s) that you migrated from warden, manually check that jail's config file, typically located in /mnt/iocage/jails/<jail_name>/config.json. Look at the property called "release". If it has the value EMPTY, you're in trouble as that will make iocage look for a file called /mnt/iocage/releases/EMPTY/root/bin/freebsd-version to attempt to check the USERLAND_VERSION and it all goes downhill from there.
Source code: https://github.com/iocage/iocage/blob/master/iocage_lib/ioc_json.py#L894-L917

I think Jurgen is on the money here Brandon, sorry. I distinctly recall people on the forums complaining about the release being set to empty. I didn't understand the implication.

I can totally assure you I do have 3 jails with this empty flag.

What can I do, if anything?

And again, is this issue fixed long term, in the new migration script? I know the job was re opened and work begun to improve it, after I'd finished with it.

Thanks!

#23 Updated by Disk Didler almost 2 years ago

(I'm even ok, reluctant, but ok with moving the 3 bad jails out of the iocage/jails/ folder, rebooting and re mounting, therefore losing them. As long as I can still pick and choose config files out of them.)

But as it stands I'm not just down 3 "bad" jails, I'm down all jails.
Luckily I'm on holiday another 3 days and don't hugely care!

#24 Updated by Brandon Schneider almost 2 years ago

He may be! Jurgen often finds my mistakes. In that case, as long as they aren't important, destroy them and remigrate. The migration script should no longer have any issues regarding that.

#25 Updated by Brandon Schneider almost 2 years ago

  • Status changed from Done to In Progress

We should still not do this on EMPTY :D Great catch as always Jurgen

#26 Updated by Brandon Schneider almost 2 years ago

  • Needs Merging changed from No to Yes

#27 Updated by Brandon Schneider almost 2 years ago

Disk Didler wrote:

(I'm even ok, reluctant, but ok with moving the 3 bad jails out of the iocage/jails/ folder, rebooting and re mounting, therefore losing them. As long as I can still pick and choose config files out of them.)

But as it stands I'm not just down 3 "bad" jails, I'm down all jails.
Luckily I'm on holiday another 3 days and don't hugely care!

If you can wait for 11.2's RELEASE, you won't be out any jails. But if they're not super important, I do recommend remigrating to have an upgrade path for 11.3-RELEASE worlds with these jails if you ever so choose.

#28 Updated by Disk Didler almost 2 years ago

Brandon Schneider wrote:

He may be! Jurgen often finds my mistakes. In that case, as long as they aren't important, destroy them and remigrate. The migration script should no longer have any issues regarding that.

Help is appreciated regardless!

I'd prefer a solution that doesn't involve destruction as a first try.

Glad the script won't do this again at least for others.

#29 Updated by Disk Didler almost 2 years ago

Brandon Schneider wrote:

Disk Didler wrote:

(I'm even ok, reluctant, but ok with moving the 3 bad jails out of the iocage/jails/ folder, rebooting and re mounting, therefore losing them. As long as I can still pick and choose config files out of them.)

But as it stands I'm not just down 3 "bad" jails, I'm down all jails.
Luckily I'm on holiday another 3 days and don't hugely care!

If you can wait for 11.2's RELEASE, you won't be out any jails. But if they're not super important, I do recommend remigrating to have an upgrade path for 11.3-RELEASE worlds with these jails if you ever so choose.

The original warden jails have been destroyed, since I've been using iocage successfully for near 3 months I would say.

So the existing 3 flagged as "empty" I need to somehow convert or rescue the config out of them.

#30 Updated by Brandon Schneider almost 2 years ago

You can manually (backup first!) change each configurations release and cloned_release to the release of the actual jail by hand.

If you don't know it offhand, run file /mnt/POOL/iocage/jails/JAIL/root/bin/sh for each and add a -RELEASE to the number. If it's just 11, make sure it's 11.0-RELEASE, and so on.

An example:
root@freenas[~]# file /mnt/foo/iocage/jails/foobar/root/bin/sh
/mnt/foo/iocage/jails/foobar/root/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD
11.1, FreeBSD-style, stripped

This jail is actually 11.1-RELEASE, so setting those two keys manually in the jails (foobar) config.json to 11.1-RELEASE will solve your problem and let you upgrade them in the future.

#31 Updated by Disk Didler almost 2 years ago

Brandon Schneider wrote:

Disk: Did you create a manual dataset in releases called EMPTY? That would be the source of the problem

Brandon, can you do me a favour and identify the 3 jails in the logs which you originally found earlier in the thread. With the empty flag.

Out of 10 plus jails, I don't recall which 3 were the migrations

#32 Updated by Jurgen Segaert almost 2 years ago

The below command may be useful to find the EMPTY releases. You may have to change the /mnt/tank/ part, based on where iocage is mounted.

grep EMPTY `find /mnt/tank/iocage/jails/*/config.json`

#33 Updated by Disk Didler almost 2 years ago

Jurgen Segaert wrote:

The below command may be useful to find the EMPTY releases. You may have to change the /mnt/tank/ part, based on where iocage is mounted.
[...]

I'm wondering if I'm right something wrong.

The console spits back "find: no match"
However, it didn't drop the console back to the cli, as if it was searching a long time.

Wonder if the find command is missing from my system or I'm just too impatient.

#34 Updated by Brandon Schneider almost 2 years ago

grep EMPTY `find /mnt/iocage/jails/*/config.json` should show you which jails. That looks to be what your mount point is.

#35 Updated by Dru Lavigne almost 2 years ago

  • Status changed from In Progress to Done
  • Needs Merging changed from Yes to No

#36 Updated by Disk Didler almost 2 years ago

  • File debug-freenas-20181025165642.tgz added

Thank you guys for your help, I finally understand stuff.

I ran the command, identified the 3 json files.
I edited them, in 2 places, 2 of which were 11.0-RELEASE but the sickrage one, oddly enough, 10.3-RELEASE (?? I've never used 10, I skipped it)

I'm baffled.

I rebooted and tried again to re-initialise the iocage root (same original problem) - no dice!

debug is attached, hopefully revealing more info.

appreciate help!

Sorry for hassling.

#37 Updated by Disk Didler almost 2 years ago

Disk Didler wrote:

Thank you guys for your help, I finally understand stuff.

I ran the command, identified the 3 json files.
I edited them, in 2 places, 2 of which were 11.0-RELEASE but the sickrage one, oddly enough, 10.3-RELEASE (?? I've never used 10, I skipped it)

I'm baffled.

I rebooted and tried again to re-initialise the iocage root (same original problem) - no dice!

debug is attached, hopefully revealing more info.

appreciate help!

Sorry for hassling.

If anyone can take a crack in the next 24 hours I'd appreciate it, I'll be off the plane then and back home

#38 Updated by Brandon Schneider almost 2 years ago

The debug looks incomplete. Can you try running iocage debug on your cli and see if it shows some exception?

Also run this to list which jails might still be an issue:

find /mnt/iocage/jails -name 'config.json' -exec grep -l '"release": "EMPTY"' {} \;

#39 Updated by Disk Didler almost 2 years ago

Brandon Schneider wrote:

The debug looks incomplete. Can you try running iocage debug on your cli and see if it shows some exception?

Also run this to list which jails might still be an issue:
[...]

No idea how to run iocage debug in cli?

Just trying to reconfigure the jail root path / store.

I did run the command though, got this

"root@freenas:~ # find /mnt/iocage/jails -name 'config.json' -exec grep -l '"release": "EMPTY"' {} \;
grep: /mnt/iocage/jails/couchpotato_1/root/usr/local/share/couchpotato/CouchPotatoServer/couchpotato/static/fonts/config.json: No such file or directory
root@freenas:~ #"

As for the debug, trying now

#40 Updated by Disk Didler almost 2 years ago

  • File debug (2).tgz added

ok

rebooted it

attempted iocage reconfigure
failed

new debug for you.

#41 Updated by Disk Didler almost 2 years ago

Anyone?

#42 Updated by Disk Didler almost 2 years ago

  • File debug (3).tgz added

To update, when I do the activation it even says down the bottom of the new UI

"Successfully activate pool ARRAY"
(note the lack of a D on activateD)

Yet, click away, then back to jails, nope it still needs pool activation.
This is 25 minutes after a reboot.

Note, my data is still (seemingly) undamaged in /mnt/iocage/jails

Debug is again attached.

#43 Updated by Disk Didler almost 2 years ago

Is there anything left I can to try and fix this please?

#44 Updated by Disk Didler almost 2 years ago

For anyone who finds this thread, I was helped by 8 bit Yoda on the forums.

https://forums.freenas.org/index.php?threads/installed-rc1-iocage-jail-root-forgotten-cant-set-it-back-no-more-jails.70913/#post-490366

Once I downloaded 2 different releases, then rebooted, finally my jails are back.
Thanks to all of you who attempted to fix this, hopefully we don't see it going forwrd for other users.

Please close, this already closed job ,...

#45 Updated by William Grzybowski almost 2 years ago

Disk Didler wrote:

Is there anything left I can to try and fix this please?

Sorry we couldnt get to this earlier. Brandon is on a time off.

#46 Updated by Disk Didler almost 2 years ago

No worries, more importantly, this issue is fixed for all users going forward and only occurred, I believe, due to me testing the early migration script. So it's improved the end product. I'm happy.

Thanks again. Enjoy holiday Brandon.

#47 Updated by Dru Lavigne almost 2 years ago

  • Has duplicate Bug #56427: Jails stop working after update to 11.2 RC1 added

#48 Updated by Dru Lavigne almost 2 years ago

  • Has duplicate Bug #58131: iocage jails stoped working 11.2 RC1-1 added

#49 Updated by Dru Lavigne almost 2 years ago

  • File deleted (debug-freenas-20181025165642.tgz)

#50 Updated by Dru Lavigne almost 2 years ago

  • File deleted (debug (2).tgz)

#51 Updated by Dru Lavigne almost 2 years ago

  • File deleted (debug (3).tgz)

Also available in: Atom PDF