Project

General

Profile

Bug #27236

Improve warden to iocage migration script

Added by Timothy Moore II almost 3 years ago. Updated almost 2 years ago.

Status:
Done
Priority:
Nice to have
Assignee:
Brandon Schneider
Category:
Middleware
Target version:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

As per my conversation with Brandon, I'm opening a ticket reminder to continue testing and bug-fixing the utility created to migrate jails from Warden to Iocage.

QA may want to look into adding some automated testing of the utility too.


Related issues

Related to FreeNAS - Bug #46927: Fix errant jail RELEASE types in warden migration scriptDone
Related to FreeNAS - Bug #48688: Jail migration: IPv6 autofconfigureClosed
Related to FreeNAS - Bug #48691: warden jail configuration unexpected behavior: Closed
Related to FreeNAS - Bug #48702: Jail migration: DHCP errorsClosed
Has duplicate FreeNAS - Bug #38140: Warden to iocage migration script has some minor bugs with networkingClosed
Has duplicate FreeNAS - Bug #42689: Save not clickable in Jail Edit after warden_migrate created migration.Closed

Associated revisions

Revision eaaa7f1e (diff)
Added by Brandon Schneider about 2 years ago

fix(migrate_warden): Fixes lots of behavior in the migration - Don't use EMPTY for jail RELEASE - Fix destinations in fstab and create destination directories if they don't exist - Migrate defaultrouter for ipv4 and ipv6 - Create proper mac addresses for migration and correctly iterate onto the next one for the second address that iocage uses - Do not add vnet0|none for VNET IP variants that don't have an IP address - Optimize setting boot=on Requires iocage ticket #46245 as migrated shared IP jails without an interface currently will not have working network. Ticket: #27236

Revision f95d1f29 (diff)
Added by Brandon Schneider about 2 years ago

fix(migrate_warden): Fixes lots of behavior in the migration - Don't use EMPTY for jail RELEASE - Fix destinations in fstab and create destination directories if they don't exist - Migrate defaultrouter for ipv4 and ipv6 - Create proper mac addresses for migration and correctly iterate onto the next one for the second address that iocage uses - Do not add vnet0|none for VNET IP variants that don't have an IP address - Optimize setting boot=on Requires iocage ticket #46245 as migrated shared IP jails without an interface currently will not have working network. Ticket: #27236

Revision 66c5f732 (diff)
Added by Brandon Schneider about 2 years ago

fix(migrate_warden): Fixes lots of behavior in the migration (#1788) - Don't use EMPTY for jail RELEASE - Fix destinations in fstab and create destination directories if they don't exist - Migrate defaultrouter for ipv4 and ipv6 - Create proper mac addresses for migration and correctly iterate onto the next one for the second address that iocage uses - Do not add vnet0|none for VNET IP variants that don't have an IP address - Optimize setting boot=on Requires iocage ticket #46245 as migrated shared IP jails without an interface currently will not have working network. Ticket: #27236

History

#1 Updated by Dru Lavigne almost 3 years ago

  • Status changed from Untriaged to Unscreened

#2 Updated by Brandon Schneider almost 3 years ago

  • Status changed from Unscreened to Screened

#3 Avatar?id=14398&size=24x24 Updated by Kris Moore almost 3 years ago

  • Target version changed from 11.2-BETA1 to 11.3

#4 Updated by Brandon Schneider over 2 years ago

  • Status changed from Screened to Unscreened
  • Assignee changed from Brandon Schneider to Joe Maloney

Tested it still does the right thing here, passing to Joe to distribute it to QA so we can ensure it continues to do so until we remove Warden from base.

#5 Updated by Dru Lavigne over 2 years ago

  • Status changed from Unscreened to 47
  • Priority changed from No priority to Nice to have

#6 Updated by Dru Lavigne over 2 years ago

  • Target version changed from 11.3 to 11.2-BETA1

#7 Updated by Dru Lavigne over 2 years ago

  • Status changed from 47 to In Progress
  • Reason for Blocked set to Need verification

#8 Updated by Joe Maloney over 2 years ago

Are there any instructions on how to use the utility? Should this work with jails, and plugins? Or only jails?

#9 Updated by Dru Lavigne over 2 years ago

No since it's a super simple utility. Just run the program to get the arguments, if you don't supply a jail and only a zpool it will migrate all jails.

#10 Updated by Joe Maloney over 2 years ago

  • QA Status Test Fails FreeNAS added
  • QA Status deleted (Not Tested)

Steps taken:

  1. Started with 11.0-U4.
  2. Created FreeBSD 11.0 jail from GUI with DHCP option named joe-jenkins.
  3. Used warden console to connect to jail, and install packages.
  4. Installed jenkins from pkg, avahi-app, nss_mdns.
  5. Edit /etc/nsswitch for mdns.
  6. Enabled jenkins, and avahi services with bsdconfig.
  7. Upgraded to latest nightly.
  8. Stopped all warden jails (only one in this case), and verified jails were stopped with warden list.
  9. Ran iocage activate tank.
  10. Ran migration script.
root@joe-mini:~ # cat /etc/version 
FreeNAS-11-MASTER-201802050538 (1e57c18)
root@joe-mini:~ # migrate_warden.py -j joe-jenkins -p tank
-- Migrating: joe-jenkins --
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage/cli/create.py", line 143, in cli
    basejail=basejail, empty=empty)
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/iocage.py", line 643, in create
    exit_on_error=self.exit_on_error).create_jail()
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/ioc_create.py", line 86, in create_jail
    return self._create_jail(jail_uuid, location)
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/ioc_create.py", line 348, in _create_jail
    with open(f"{location}/root/etc/hosts", "r") as _etc_hosts:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/iocage/jails/joe-jenkins/root/etc/hosts'
Traceback (most recent call last):
  File "/usr/local/sbin/migrate_warden.py", line 386, in <module>
    loop.run_until_complete(main(sys.argv[1:], loop))
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "/usr/local/sbin/migrate_warden.py", line 376, in main
    await Migrate(jail, _dir, iocage_pool, verbose, loop).migrate_jail()
  File "/usr/local/sbin/migrate_warden.py", line 210, in migrate_jail
    self.create_jail(props)
  File "/usr/local/sbin/migrate_warden.py", line 303, in create_jail
    su.check_call(cmd, stdout=su.PIPE)
  File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['iocage', 'create', '-n', 'joe-jenkins', '-e', 'notes=warden_id=1', 'allow_raw_sockets=1', 'vnet=on', 'ip4_addr=vnet0|none', 'ip6_addr=vnet0|none', 'vnet0_mac=62:17:fb:9f:84:ad,62:17:fb:9f:84:ad']' returned non-zero exit status 1.
root@joe-mini:~ # 
root@joe-mini:~ # iocage list
+-----+-------------+-------+---------+------+
| JID |    NAME     | STATE | RELEASE | IP4  |
+=====+=============+=======+=========+======+
| -   | joe-jenkins | down  | EMPTY   | none |
+-----+-------------+-------+---------+------+
root@joe-mini:~ #
root@joe-mini:~ # iocage start joe-jenkins
* Starting joe-jenkins
  + Start FAILED
jail: mount.devfs: /mnt/iocage/jails/joe-jenkins/root/dev: No such file or directory

#11 Updated by Dru Lavigne over 2 years ago

  • Category changed from Middleware to OS
  • Status changed from In Progress to Blocked
  • Assignee changed from Joe Maloney to Brandon Schneider

#12 Updated by Dru Lavigne over 2 years ago

  • Status changed from Blocked to Broken
  • Reason for Blocked deleted (Need verification)

#13 Updated by Brandon Schneider over 2 years ago

Joe can you show me both mount and zfs list please?

#14 Updated by Brandon Schneider over 2 years ago

  • Status changed from Broken to In Progress

#15 Updated by Brandon Schneider over 2 years ago

  • Status changed from In Progress to Blocked
  • Reason for Blocked set to Waiting for feedback

Joe: I believe this is because of an older version of iocage there.

Please try with a recent nightly and let me know

#16 Updated by Joe Maloney over 2 years ago

  • Assignee changed from Brandon Schneider to Joe Maloney

#17 Updated by Joe Maloney over 2 years ago

  • Reason for Blocked changed from Waiting for feedback to Need verification

#18 Updated by Joe Maloney over 2 years ago

  • Status changed from Blocked to In Progress
  • Reason for Blocked deleted (Need verification)

#19 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 2 years ago

  • Due date set to 03/09/2018

#20 Updated by Dru Lavigne over 2 years ago

  • Target version changed from 11.2-BETA1 to 11.2-RC2

#21 Updated by Dru Lavigne over 2 years ago

  • Subject changed from Test & Bugfix Warden -> Iocage jail migration utility to Test warden to iocage jail migration utility
  • Status changed from In Progress to Ready for Testing
  • Target version changed from 11.2-RC2 to 11.2-BETA1
  • Severity set to New

#22 Updated by Jurgen Segaert over 2 years ago

I tried to migrate a few jails with the script. These are my initial findings.

root@FreenasVBox:~ # cat /etc/version
FreeNAS-11.1-U5 (8e2a858a1)

root@FreenasVBox:~ # warden list
ID                       AUTOSTART    STATUS       TYPE
---------------------------------------------------------------------------
mineos_10_3              Enabled      Stopped      standard-10.3-RELEASE
plex_11_0                Enabled      Stopped      standard
xmrig_1                  Enabled      Stopped      pluginjail

root@FreenasVBox:~ # migrate_warden.py -j mineos_10_3 -p tank
-- Migrating: mineos_10_3 --

root@FreenasVBox:/ # iocage list -l
+-----+-------------+------+-------+------+---------+------------------+-----+----------+
| JID |    NAME     | BOOT | STATE | TYPE | RELEASE |       IP4        | IP6 | TEMPLATE |
+=====+=============+======+=======+======+=========+==================+=====+==========+
| -   | mineos_10_3 | on   | down  | jail | EMPTY   | 192.168.0.221/24 | -   | -        |
+-----+-------------+------+-------+------+---------+------------------+-----+----------+

So far, so good. Below are the issues:

1) fstab entry wasn't migrated properly; I expected the data to be mounted in /mnt/iocage/jails/mineos_10_3/root/var/games/minecraft

root@FreenasVBox:/ # iocage fstab -l mineos
+-------+-----------------------------------------------------------------------------------------+
| INDEX |                                       FSTAB ENTRY                                       |
+=======+=========================================================================================+
| 0     | /mnt/tank/data/minecraft /mnt/tank/jails/mineos_10_3//var/games/minecraft nullfs rw 0 0 |
+-------+-----------------------------------------------------------------------------------------+

2) The jail started fine, but it didn't get an IP address. (uses shared IP, not VNET) because the interface was missing; I expected ip4_addr="vnet|192.168.0.221/24"

root@FreenasVBox:/ # iocage get all mineos | grep ip
allow_sysvipc:0
ip4:new
ip4_addr:192.168.0.221/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1

3) The release is set to EMPTY, so the migrated jail cannot be upgraded. I expected it to be 10.3-RELEASE

root@FreenasVBox:/ # iocage exec mineos freebsd-version
10.3-RELEASE

root@FreenasVBox:/ # iocage get all mineos | grep release
cloned_release:EMPTY
release:EMPTY

#23 Updated by Dru Lavigne over 2 years ago

  • Status changed from Ready for Testing to Done

Jurgen: thank you for taking the time to test this!

As a safety measure, we will be recommending that users recreate their iocage jails (as we know that no script can catch every configuration scenario). However, we will look at pull requests if you would like to add your recommendations to the script.

#24 Updated by Dru Lavigne over 2 years ago

  • Target version changed from 11.2-BETA1 to Master - FreeNAS Nightlies
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#25 Updated by Dru Lavigne about 2 years ago

  • Has duplicate Bug #38140: Warden to iocage migration script has some minor bugs with networking added

#26 Updated by Disk Didler about 2 years ago

Whoever fixes this, please take a peek at my closed off duplicate log, I have drilled down and identified some mac address issues, ipv6 issues, ipv4 gateway issues also.

Hopefully it helps fixing faster, thanks.

#27 Updated by Disk Didler about 2 years ago

I have just run this script again on my system for some other jails.

I can confirm

the ipv6 really should be set to "none"
setting an ipv4 route / gateway, would be handy but not critical.

MOST importantly however.

The damn mac address, it's being imported in, incorrectly with this : : : : business in it and the macs not being different (change it by one digit, like other jails, as per my notes in the other job)

Finally.

I just migrated a jail which mounted points like this

Source (host mount)
/mnt/ARRAY/media
Destination (iocage mount)
/mnt/ARRAY/jails/couchpotato_1//media

That's a no no, isn't it?

Shouldn't it be

Source (host mount)
/mnt/ARRAY/media
Destination (iocage mount)
/mnt/*iocage/jails/couchpotato_1/root/media*

It's mounting into the old jail?
Please note! Thank you for the script, it's still good work and with some fiddling it works fine, but a couple of adjusted lines in it and it should be basically flawless.

In the very least if this stuff won't be fixed can we get a sticky forum post, outlining the stuff it doesn't do, so people aren't frustrated?

#28 Updated by Disk Didler about 2 years ago

(I don't think this should be set to done?)

#29 Updated by Chiraq Tech about 2 years ago

Agreed, this ticket should not be marked as done. There are still the same bugs in the script as of 11.2-BETA2 regarding the mac address. Further I am unable to click save on any edits to the jail in the new GUI after running the migration script (see Bug #42689).

#30 Updated by Brandon Schneider about 2 years ago

  • Subject changed from Test warden to iocage jail migration utility to Fix up remaining issues with warden migration script
  • Category changed from OS to Middleware
  • Status changed from Done to Not Started
  • Assignee changed from Joe Maloney to Brandon Schneider
  • Severity changed from New to Medium
  • Needs Merging changed from No to Yes

Agreed, not sure why this rotted so long.

#31 Updated by Brandon Schneider about 2 years ago

  • Status changed from Not Started to In Progress

#32 Updated by Dru Lavigne about 2 years ago

  • Target version changed from Master - FreeNAS Nightlies to 11.2-RC1
  • Needs Doc changed from No to Yes

#33 Updated by Disk Didler about 2 years ago

#34 Updated by Brandon Schneider about 2 years ago

PR: https://github.com/freenas/freenas/pull/1788
DESC: Fix up all the warts on the migration
RISK: Low
ACCEPTANCE: Create a Warden jail, migrate it, profit.

#35 Updated by Bug Clerk about 2 years ago

  • Status changed from In Progress to Ready for Testing

#36 Updated by Brandon Schneider about 2 years ago

  • Needs Merging changed from Yes to No

Alright guys, this is all done. Thanks for the great feedback Jurgen and Disk :)

#37 Updated by Disk Didler about 2 years ago

Happy to help, hope it was useful. Thanks.

#38 Updated by Jurgen Segaert about 2 years ago

Way to go Brandon! A lot of users will appreciate this.

#39 Updated by Dru Lavigne about 2 years ago

  • Subject changed from Fix up remaining issues with warden migration script to Improve warden to iocage migration script

#40 Updated by Dru Lavigne about 2 years ago

  • Has duplicate Bug #42689: Save not clickable in Jail Edit after warden_migrate created migration. added

#41 Updated by Brandon Schneider about 2 years ago

  • Related to Bug #46927: Fix errant jail RELEASE types in warden migration script added

#42 Updated by Timothy Moore II about 2 years ago

  • Status changed from Ready for Testing to Blocked
  • Reason for Blocked set to Dependent on a related task to be completed

Testing on hold until 46927 is resolved.

#43 Updated by Dru Lavigne about 2 years ago

  • Status changed from Blocked to Ready for Testing

#44 Updated by Timothy Moore II almost 2 years ago

  • Reason for Blocked deleted (Dependent on a related task to be completed)
  • Needs QA changed from Yes to No

Testing with FreeNAS VM installed with 11.1-U5 then updated to FreeNAS-11.2-MASTER-201809270855:

This ticket is too generic to cover some of the specific issues seen post-migration, so I've retested it multiple times with a focus on the actual success of the migration script. I'll open new issues for all other post-jail migration identified bugs.

Create VM installed with FreeNAS 11.1-U5. Configure a pool and global configurations for warden. Create multiple test jails: some simple with no network settings, some with DHCP & IPv6 autoconfig enabled, and some based on preconfigured warden templates. Update FreeNAS to FreeNAS-11.2-MASTER-201809270855 and test the migrate_warden.py command. Jail migration works and an iocage jail is created, but there are multiple settings mismatches and the jail is largely uneditable in the new UI. Discover error with warden where manually configured jail settings do not override the generic defaults, appearing to lead to problems with migration. Rebuild test environment/pool and adjust warden global config to have as few settings as possible. Remake all test jails, update FreeNAS, and test migration again. Simple and template warden jails are migrated with no complaints from the cli, but must be edited with functional network settings before the jail can be started.

Recommendations:
- Document or provide warnings about current issues with jail migration: the warden global config problem, necessity of editing the jail configuration post-migration to reconfigure networking, and to avoid all IPv6 autoconfiguration settings for a warden jail.
- Open new issues to track and improve individual parts of jail migration (IPv6 autoconfigure for example).

#45 Updated by Timothy Moore II almost 2 years ago

  • Related to Bug #48688: Jail migration: IPv6 autofconfigure added

#46 Updated by Timothy Moore II almost 2 years ago

  • Related to Bug #48691: warden jail configuration unexpected behavior: added

#47 Updated by Timothy Moore II almost 2 years ago

  • Related to Bug #48702: Jail migration: DHCP errors added

#48 Updated by Dru Lavigne almost 2 years ago

  • Status changed from Ready for Testing to Done
  • Needs Doc changed from Yes to No

Also available in: Atom PDF