Project

General

Profile

Feature #8657

Installer should figure out how to trampoline updates

Added by Jordan Hubbard over 4 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Nice to have
Assignee:
Sean Fagan
Category:
OS
Target version:
Estimated time:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Description

It seems there are a lot of people too poor to afford multiple USB devices, or are more likely not too poor, but simply didn't think about it when they were at Fry's and now we are telling them "you can't install from the device you booted from" and this makes them sad, because now they have to get back in the car and drive for 45 minutes to by ANOTHER goddamned USB device.

It would be a lot simpler if our installer was smart enough to simply say "Hey, they have RAM! I'll just copy the bits of myself I need into that RAM, make sure everything went OK with that, and then blast the device I just booted from!" (If they don't have enough RAM then screw it, then we can tell them they'll have to drive back to Fry's for both another USB stick AND some more RAM, because their HW sucks and they should be ashamed of themselves).

This is very frequently requested and we frankly look a little lame for not supporting such an obvious installation mode. We should fix that and not be lame.

Associated revisions

Revision aa938617 (diff)
Added by Sean Fagan about 3 years ago

Check for update scripts stashed by the updater previously, and run them. Ticket: #8657

Revision 7bd34501 (diff)
Added by Sean Fagan about 3 years ago

When doing an install, stash the install/update scripts in the new root. This will allow the new version to run them on reboot. Ticket: #8657

Revision 44804dfa (diff)
Added by Sean Fagan about 3 years ago

Wrong path! Ticket: #8657

Revision 9d15cb0b (diff)
Added by Sean Fagan about 3 years ago

Wrong path! Ticket: #8657

Revision 395cf684 (diff)
Added by Sean Fagan about 3 years ago

Squashed commit of the following: commit 9d15cb0b37d34a6bdb0d4a957c8918fad67ad866 Author: Sean Eric Fagan <sef@ixsystems.com> Date: Wed Nov 23 13:06:18 2016 -0800 Wrong path! Ticket: #8657

Revision 6655a355
Added by Sean Fagan about 3 years ago

Merge branch 'FIX-8657' This is a possibly risky change, that will not show up until after two updates. Ticket: #8657

Revision 44150360 (diff)
Added by Sean Fagan about 3 years ago

Fix fresh installs. Ticket: #8657

History

#1 Updated by Jordan Hubbard over 4 years ago

  • Description updated (diff)

#2 Updated by Josh Paetzel over 4 years ago

Linux has a thing called pivot_root that makes this REALLY easy.

FreeBSD has a thing called init_chroot that can be used to do this but it's really hard.

The hilarious thing is the installer used to do this but we never used it because we had an image you could dd to a usb stick.

#3 Updated by Sean Fagan over 4 years ago

  • Status changed from Unscreened to Screened
  • Target version changed from Unspecified to 49

In FreeBSD, it's not really something you can decide after the fact: you have to do it from the beginning, or else you have a reference on the root device that will never go away.

We do this for the manual upgrade, and it is a major pain. It's difficult to debug, and we're terrified of making any changes to it. (Now, some of that is because it's completely non-interactive, so for the installer, that does make it a bit easier.)

This is major work, for benefit of people who are too cheap to spend another US$20 for a second thumb drive.

I find that argument uncompelling.

#4 Updated by Sean Fagan over 4 years ago

Note that "init_chroot" doesn't help here, since any chroot still leaves a reference to the real root.

You need to set up loader.conf on the boot device to specify a ram disk image, and the limit on that is too small to contain the package files, which means you also have to be able to find the boot device, and the filesystem on it, and then create another ram disk and copy the packages to it.

All doable, but not at all trivial.

#5 Updated by Jordan Hubbard over 4 years ago

The argument really isn't "people who are cheap" (even though I cited that scenario in jest). The argument for it is simplicity and ease of use. "Buy a device, use a device." Very easy to document and described if there are no dangling additional dependencies like we have now. I also think it's an argument for FINALLY bringing pivot_root() into FreeBSD. How long has Linux had it - 10 years? More??

#6 Updated by Cpu Roast over 4 years ago

I guess you are being more receptive then when I suggested it months ago in:

https://bugs.freenas.org/issues/7578

#7 Updated by Jordan Hubbard over 4 years ago

Well, it'll probably never happen so my degree of "receptivity" is probably irrelevant in any case. :)

#8 Updated by Sean Fagan over 3 years ago

Note that we got rid of the old GUI trampoline boot. This is a fairly significant amount of work (see the trampoline code in the GUI upgrade script).

#10 Updated by Sean Fagan about 3 years ago

As a note, I'm not sure this is actually possible any longer. The old trampoline code was able to work by using the other partition to boot from; we don't have that any longer.

Hm. Could create the new BE, install into it but not run post-upgrade/post-install scripts, instead write them as files to a directory in the new BE. Then activate new BE, boot into it, and as part of startup could check for that special directory, and run the scripts, remove them, and then reboot again.

Too bad we can't retroactively change the update code.

#11 Updated by Sean Fagan about 3 years ago

  • Status changed from Screened to Investigation

I'm trying to figure out what would break if I did that. It's going to be a much less pleasant update experience (since it'd require two reboots, which can take a fairly long time on some systems).

pkgng packages could not be used with this change.

Pre-upgrade / pre-install scripts could be run as they are now, since they're expected to run before any changes are made.

Would this only need to be done with packages requiring a reboot? If base-os requires a reboot, but middleware doesn't, can middleware's post-install/post-upgrade scripts still be run, or do they all need to be postponed to after reboot?

#12 Updated by Jordan Hubbard about 3 years ago

I think it might only be necessary for major upgrades - between 9 and 10 (this is something Suraj needs because the new 10 environment is so alien to the 9 environment) or between any build we deem "ABI breaking" with any previous build, e.g. it's like a reboot=yes update but MOAR SO. More like omg_the_world_has_changed_and_zombies_are_in_the_whitehouse=yes flag material.

#13 Updated by Sean Fagan about 3 years ago

9.10 -> 9.10 is going to have this problem. I think the next stable release, in fact.

(In this case, I don't believe it will affect the update, just as it didn't in 10. I could be wrong, however.)

#14 Updated by Sean Fagan about 3 years ago

I've got three changes: one for freenas-pkgtools, one for fn9, and one for fn10. (The latter two being for startup.)

The freenas-pkgtools diff below causes it to stash the various scripts in a directory, which are then executed by the modified startup. Tested on fn10 so far.

diff --git a/lib/Installer.py b/lib/Installer.py
index 32ff7cd..b139ffe 100644
--- a/lib/Installer.py
+++ b/lib/Installer.py
@ -296,16 +296,27 @ def RunPkgScript(scripts, type, root=None, **kwargs):
log.debug("No %s script to run" % type)
return

- scriptName = "/%d-%s" % (os.getpid(), type)
+ trampoline = kwargs.pop("trampoline", True)
+
+ scriptName = "%s-%s" % (kwargs.pop("pkgName", str(os.getpid())), type)
if root and root != "":
- scriptPath = "%s%s" % (root, scriptName)
+ scriptPath = os.path.join(root, "update-scripts", scriptName)
+ if trampoline:
+ MakeDirs(os.path.join(root, "update-scripts"))
+ with open(os.path.join(root, "update-scripts", "order"), "a") as f:
+ print(scriptName, file=f)
else: # Writing to root isn't ideal, so this should be re-examined
- scriptName = "/tmp" + scriptName
+ scriptName = os.path.join("/tmp", scriptName)
scriptPath = scriptName
-
+ trampoline = False
+
with open(scriptPath, "w") as f:
f.write(scripts[type])
+
+ if trampoline:
+ return
+
args = ["/bin/sh", "-x", scriptName]
if "SCRIPT_ARG" in kwargs and kwargs["SCRIPT_ARG"] is not None:
args.append(kwargs["SCRIPT_ARG"])
@ -730,14 +741,16 @ def install_file(pkgfile, dest): # upgrade scripts from the old version.
if upgrade_aware:
RunPkgScript(
- old_scripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_PRE_UPGRADE, dest, PKG_PREFIX=prefix
+ old_scripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_PRE_UPGRADE, dest, PKG_PREFIX=prefix,
+ pkgName=pkgName,
)
RunPkgScript(
old_scripts,
PKG_SCRIPT_TYPES.PKG_SCRIPT_UPGRADE,
dest,
PKG_PREFIX=prefix,
- SCRIPT_ARG="PRE-UPGRADE"
+ SCRIPT_ARG="PRE-UPGRADE",
+ pkgName=pkgName,
)

  1. If the new version is a delta package, we do things differently
    @ -749,7 +762,8 @ def install_file(pkgfile, dest): # Run a pre-delta script if any, but only if dest is none
    if dest is None:
    RunPkgScript(
    - pkgScripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_PRE_DELTA, dest, PKG_PREFIX=prefix
    + pkgScripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_PRE_DELTA, dest, PKG_PREFIX=prefix,
    + pkgName=pkgName,
    )
  1. Next step for a delta package is to remove any removed files and directories.
    @ -783,14 +797,16 @ def install_file(pkgfile, dest):
    old_scripts,
    PKG_SCRIPT_TYPES.PKG_SCRIPT_PRE_DEINSTALL,
    dest,
    - PKG_PREFIX=prefix
    + PKG_PREFIX=prefix,
    + pkgName=pkgName,
    )
    RunPkgScript(
    old_scripts,
    PKG_SCRIPT_TYPES.PKG_SCRIPT_DEINSTALL,
    dest,
    PKG_PREFIX=prefix,
    - SCRIPT_ARG="DEINSTALL"
    + SCRIPT_ARG="DEINSTALL",
    + pkgName=pkgName,
    )
if pkgdb.RemovePackageFiles(pkgName) == False:
@ -813,14 +829,16 @ def install_file(pkgfile, dest):
old_scripts,
PKG_SCRIPT_TYPES.PKG_SCRIPT_POST_DEINSTALL,
dest,
- PKG_PREFIX=prefix
+ PKG_PREFIX=prefix,
+ pkgName=pkgName,
)
RunPkgScript(
old_scripts,
PKG_SCRIPT_TYPES.PKG_SCRIPT_INSTALL,
dest,
PKG_PREFIX=prefix,
- SCRIPT_ARG="POST-DEINSTALL"
+ SCRIPT_ARG="POST-DEINSTALL",
+ pkgName=pkgName,
)
if pkgDeltaVersion is not None:
@ -835,9 +853,11 @ def install_file(pkgfile, dest):
  1. Is this correct behaviour for delta packages?
    if upgrade_aware is False:
    RunPkgScript(pkgScripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_PRE_INSTALL,
    - dest, PKG_PREFIX=prefix)
    + dest, PKG_PREFIX=prefix, pkgName=pkgName)
    RunPkgScript(pkgScripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_INSTALL,
    - dest, PKG_PREFIX=prefix, SCRIPT_ARG="PRE-INSTALL")
    + dest, PKG_PREFIX=prefix, SCRIPT_ARG="PRE-INSTALL",
    + pkgName=pkgName,
    + )
  1. Go through the tarfile, looking for entries in the manifest list.
    pkgFiles = []
    @ -881,29 +901,41 @ def install_file(pkgfile, dest):
    pkgdb.AddFilesBulk(pkgFiles)
if upgrade_aware:
- RunPkgScript(pkgScripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_POST_UPGRADE, dest, PKG_PREFIX=prefix)
+ RunPkgScript(pkgScripts,
+ PKG_SCRIPT_TYPES.PKG_SCRIPT_POST_UPGRADE,
+ dest, PKG_PREFIX=prefix,
+ pkgName=pkgName,
+ )
RunPkgScript(
pkgScripts,
PKG_SCRIPT_TYPES.PKG_SCRIPT_UPGRADE,
dest,
PKG_PREFIX=prefix,
- SCRIPT_ARG="POST-UPGRADE"
+ SCRIPT_ARG="POST-UPGRADE",
+ pkgName=pkgName,
)
if dest is None and pkgDeltaVersion is not None:
RunPkgScript(
pkgScripts,
PKG_SCRIPT_TYPES.PKG_SCRIPT_POST_DELTA,
dest,
- PKG_PREFIX=prefix
+ PKG_PREFIX=prefix,
+ pkgName=pkgName,
)
else:
- RunPkgScript(pkgScripts, PKG_SCRIPT_TYPES.PKG_SCRIPT_POST_INSTALL, dest, PKG_PREFIX=prefix)
+ RunPkgScript(pkgScripts,
+ PKG_SCRIPT_TYPES.PKG_SCRIPT_POST_INSTALL,
+ dest,
+ PKG_PREFIX=prefix,
+ pkgName=pkgName,
+ )
RunPkgScript(
pkgScripts,
PKG_SCRIPT_TYPES.PKG_SCRIPT_INSTALL,
dest,
PKG_PREFIX=prefix,
- SCRIPT_ARG="POST-INSTALL"
+ SCRIPT_ARG="POST-INSTALL",
+ pkgName=pkgName
)
return True

#15 Updated by Sean Fagan about 3 years ago

aa938617115c3808ec7ccd72a8de7fdde35f9a3c in the freenas repo, and branch FIX-8657 in the same.

#16 Updated by Sean Fagan about 3 years ago

  • Status changed from Investigation to 19
  • Assignee changed from Sean Fagan to Vaibhav Chauhan
  • Target version changed from 49 to 9.10.2

This is actually in master. VB, 9.10.2 is going to come from master, right? So this can be marked as resolved if so.

#17 Updated by Vaibhav Chauhan about 3 years ago

  • Assignee changed from Vaibhav Chauhan to Sean Fagan

sef: yes 9.10.2 will be branched off from freenas-pkgtools:master branch.

#18 Updated by Sean Fagan about 3 years ago

commit:6655a355c68fb881ce5f087ccfa337258e0330ff has been merged into freenas-pkgtools master. This is mildly risky, and will take two updates to show up.

#19 Updated by Sean Fagan about 3 years ago

  • Subject changed from Installer should figure out how to use a RAMDISK trampoline to allow single device installs to Installer should figure out how to trampoline updates

N.B. Changing the title since we can't do ramdisk updates any longer (well..... I think, maybe I can think of a way to do that using grub), but we do want to have trampolined updates because of kernel ABI changes.

#20 Updated by Vaibhav Chauhan about 3 years ago

  • Status changed from 19 to Ready For Release

code is already in so marking RFR

#21 Updated by Dru Lavigne about 2 years ago

  • Status changed from Ready For Release to Resolved

Also available in: Atom PDF