Project

General

Profile

Bug #23488

FreeNAS 9.10.2 offline tar updates fail to apply

Added by Cpu Roast over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
No priority
Assignee:
Sean Fagan
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

This is the 2nd time I try to apply an offline update using the .tar file and it always fails.

Going from 9.10.2-U1 to 9.10.2-U2 using the tar file failed, so I used the iso upgrade method.

And today again, I try to upgrade to U3 from U2 using the tar file and again it fails with this error, which if I recall is the same error I had the 1st time I tried with the U2 upgrade.

Apr 21 11:54:33 cda-freenas manage.py: [middleware.exceptions:37] [MiddlewareError: Failed to apply update Command '['/usr/local/bin/freenas-update', '-C', u'/var/tmp/firmware', 'update']' returned non-zero exit status 1: ]
Apr 21 11:54:33 cda-freenas manage.py: [rollbar:1097] Exception while posting item

Every time, I've used a pool a temp location for the update storage.

Manifest.py (22.7 KB) Manifest.py Sean Fagan, 04/21/2017 09:39 AM

Associated revisions

Revision d4e4a8f4 (diff)
Added by Sean Fagan over 4 years ago

Looks like, when I rewrote the verifier, I ended up not handling the case of no networking again, and an offline update would fail because it could not get the CRL. This fixes that. Ticket: #23488

History

#1 Updated by Sean Fagan over 4 years ago

  • Status changed from Unscreened to 15
Can you try running the command by hand:
  1. Copy the tar file somewhere on the FreeNAS system (e.g., /tmp/update.tar)
  2. Run "freenas-update -v /tmp/update.tar" and include its output.

#2 Updated by Cpu Roast over 4 years ago

Here is the output:

/tmp# freenas-update -v /tmp/FreeNAS-9.10.2-U3.tar
Extracting ./base-os-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Done extracting ./base-os-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Extracting ./docs-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Done extracting ./docs-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Extracting ./freebsd-pkgdb-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Done extracting ./freebsd-pkgdb-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Extracting ./freenas-pkg-tools-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Done extracting ./freenas-pkg-tools-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Extracting ./FreeNASUI-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Done extracting ./FreeNASUI-9.10.2-U3-0449fefcfe064b30b82ac037398dc224.tgz
Extracting ./ValidateUpdate
Done extracting ./ValidateUpdate
Extracting ./ReleaseNotes
Done extracting ./ReleaseNotes
Extracting ./ChangeLog
Done extracting ./ChangeLog
Extracting ./ChangeLog.txt
Done extracting ./ChangeLog.txt
Extracting ./MANIFEST
Done extracting ./MANIFEST
[freenasOS.Configuration:637] TryGetNetworkFile(['http://update-master.ixsystems.com/updates/ix_crl.pem'])
[freenasOS.Configuration:713] Unable to load http://update-master.ixsystems.com/updates/ix_crl.pem: <urlopen error [Errno 51] Network is unreachable>
[freenasOS.Configuration:727] Unable to load ['http://update-master.ixsystems.com/updates/ix_crl.pem']: <urlopen error [Errno 51] Network is unreachable>
[freenasOS.Update:1554] Could not load cached manifest file: <urlopen error [Errno 51] Network is unreachable>
[freenasOS.Update:1114]
[freenas-update:247] Unable to update:

#3 Updated by Sean Fagan over 4 years ago

That's strange. Not being able to access the CRL shouldn't be fatal, so hopefully that's not it. Can you run "manifest_util version"? Thanks.

#4 Updated by Cpu Roast over 4 years ago

/tmp# manifest_util version
FreeNAS-9.10.2-U2

#5 Updated by Sean Fagan over 4 years ago

... unless I screwed up, which may be possible. Hrm. It may need networking for updating, annoyingly enough.

#6 Updated by Cpu Roast over 4 years ago

If it does require internet connectivity for updating, then the whole reason for the offline tar update sadly goes out the door.

#7 Updated by Sean Fagan over 4 years ago

Tell me about it.

I've attached a replacement file -- place it in /usr/local/lib/freenasOS on the FreeNAS box. I've verified it works when I cannot access update-master.ixsystems.com (I used a firewall rule).

#8 Updated by Sean Fagan over 4 years ago

  • Status changed from 15 to 19

commit:d4e4a8f4c910ec2309f2b1abff15fe562ec4b1f4 in freenas-pkgtools.

#9 Updated by Cpu Roast over 4 years ago

That new Manifest .py file seems to have fixed it:

[freenasOS.Update:260] RunCommand(/usr/local/sbin/beadm, ['activate', u'9.10.2-U3'])
[freenasOS.Update:105] GRUB configuration updated successfully
[freenasOS.Update:105] Activated successfully
System should be rebooted now

System is rebooting, we'll see if U3 comes up normally.

#10 Updated by Cpu Roast over 4 years ago

Well, FreeNAS-9.10.2-U3 (e1497f269) up and running, so it's now fixed.

#11 Updated by Sean Fagan over 4 years ago

Well, that means the fix works. It'll be a couple of builds until it can actually get working without manual intervention. My apologies.

#12 Updated by Dru Lavigne about 4 years ago

What is the target version?

#13 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Target version set to 11.0

#14 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

Sean, this looks resolved. Should we mark it as such, or do you have other testing to do?

#15 Updated by Sean Fagan about 4 years ago

  • Status changed from 19 to Resolved

If it's in a build it should be resolved, so marking.

#17 Updated by Vaibhav Chauhan about 4 years ago

  • Target version changed from 11.0 to 11.0-RC

#18 Updated by Cpu Roast about 4 years ago

Can't this fix be backported to 9.10?
This is a production system which isn't going to be upgraded to 11 for quite some time, which means I'll get hit with this manual update bug every single time, for 9.10.2 updates.

It's broken in 9.10, asking someone to upgrade to 11 for a fix to functionality which is available but broken in 9.10 is wrong.

#19 Updated by Sean Fagan about 4 years ago

Backporting to something that can't update won't work, because the update won't work.

You can manually apply the change, however.

#20 Updated by Cpu Roast about 4 years ago

It's also what I did again for U5, then proceeded to apply

http://download.freenas.org/9.10/STABLE/9.10.2-U5/FreeNAS-9.10.2-U5-manual-update.tar

from the webUI, it rebooted once, did the db migration, then rebooted again, and now I'm greeted with:

/var/tmp/rc.conf.freenas: 23: Syntax error: Unterminated quoted string
Alarm clock
Enter full pathname of shell or RETURN for /bin/sh.
/var/tmp/rc.conf.freenas: 23: Syntax error: Unterminated quoted string

Also, once it's in some version of 9.10.2, all subsequent updates wouldn't have this issue.
Meaning, I wouldn't have to apply the patch all the time.

#21 Updated by Cpu Roast about 4 years ago

If I pressed enter, it dropped me to a single-user shell indicating that the upgrade didn't exactly go as planned.

I then rebooted, chose the previous U3 boot environment, it booted, indicated something about the db's schema, then rebooted, I picked U3 again from the boot menu, it then booted all the way.
I then proceeded to activate the U3 BE and delete the failed U5 BE.

#22 Updated by Cpu Roast about 4 years ago

#23 Updated by Sean Fagan about 4 years ago

What do you mean by "is actually good"?

The problems with updating are with the currently running system.

#24 Updated by Cpu Roast about 4 years ago

Because after applying the manifest.py patch, the U5 offline update applied through the WebUI didn't end up with a functional FreeNAS system using:
http://download.freenas.org/9.10/STABLE/9.10.2-U5/FreeNAS-9.10.2-U5-manual-update.tar

I was greeted by this upon boot.

-----
var/tmp/rc.conf.freenas: 23: Syntax error: Unterminated quoted string
Alarm clock
Enter full pathname of shell or RETURN for /bin/sh.
/var/tmp/rc.conf.freenas: 23: Syntax error: Unterminated quoted string
-----

If I pressed enter, it dropped me to a single-user shell indicating that the upgrade didn't exactly go as planned.

I had to flip back to the U3 BE because of it.

#25 Updated by Sean Fagan about 4 years ago

You should be able to go from U2 to U5 using the manual updater, but you'll require a network connection I think (as per many statements above). I would try doing:

freenas-update -v /path/to/update.tar

and pasting in the the output before rebooting.

But you can also always do an ISO install, and at this point that sounds like less effort.

#26 Updated by Cpu Roast about 4 years ago

As long as I patch manifest.py, wouldn't "Freenas-update -v /path/to/update.tar" be the same as using the WebUI's "Manual Update" functionality.

If so, I'm just trying to say that "FreeNAS-9.10.2-U5-manual-update.tar" specifically could be broken.

#27 Updated by Sean Fagan about 4 years ago

Multiple people have used it, and without error, so I think that's unlikely.

Now, there are a bunch of things that could go wrong -- using nightlies and then switching back to stable, or having the update silently fail (this happens when using the GUI, sometimes), or various other things.

#28 Updated by Cpu Roast about 4 years ago

Not saying that U5 applied on an internet connected FreeNAS via the WebUI is broken, after all it applied just fine on my other FreeNAS box which has Internet access.

I'm saying that the manual update tar file specifically as in:

http://download.freenas.org/9.10/STABLE/9.10.2-U5/FreeNAS-9.10.2-U5-manual-update.tar

would be broken, and I really doubt that many people apart from me have even tried to use it.

The system which resulted in a failed update has always been on stable.

#29 Updated by Sean Fagan about 4 years ago

Again, seems unlikely: it's the same files as the normal update and install use, it's cryptographically signed, the checksums are correct, etc.

#30 Updated by Cpu Roast about 4 years ago

OK,

The WebUI's manual update procedure isn't reliable, even with a patched manifest.py.

I downloaded a fresh copy of the U5 tar file, uploaded it to the FreeNAS system in /tmp.

Then while in /tmp, ran freenas-update -v FreeNAS-9.10.2-U5-manual-update.tar.

It installed successfully, I rebooted the system, it booted, did its DB maintenance and it rebooted again.
This time it booted all the way.

Also available in: Atom PDF