Project

General

Profile

Bug #8910

freenas update process should do a better job of reporting the out-of-space condition

Added by Cy Mike over 5 years ago. Updated about 3 years ago.

Status:
Closed: Duplicate
Priority:
Nice to have
Assignee:
Sean Fagan
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

FreeNAS-9.3-STABLE-201503071634

ran:

# freenas-update update

ended like so:

GRUB configuration updated successfully
Created successfully
RunCommand(/sbin/zfs, ['snapshot', u'freenas-boot/grub@Pre-Upgrade-FreeNAS-9.3-STABLE-201503200528'])
RunCommand(/usr/local/sbin/beadm, ['mount', u'FreeNAS-9.3-STABLE-201503200528', '/tmp/tmpzDjOCl'])
Mounted successfully on '/tmp/tmpzDjOCl'
RunCommand(/sbin/umount, ['-f', '/boot/grub'])
RunCommand(/sbin/mount, ['-t', 'devfs', 'devfs', '/tmp/tmpzDjOCl/dev'])
RunCommand(/sbin/mount, ['-t', 'tmpfs', 'tmpfs', '/tmp/tmpzDjOCl/var/tmp'])
RunCommand(/sbin/mount, ['-t', 'zfs', 'freenas-boot/grub', '/tmp/tmpzDjOCl/boot/grub'])
Searching for base-os-9.3-STABLE-b2b14ab-01034a4-d191868.tgz
Found package file /var/db/system/UpdateUpdate-m4DAlb/base-os-9.3-STABLE-b2b14ab-01034a4-d191868.tgz
Searching for FreeNASUI-9.3-STABLE-b2b14ab-01034a4-d191868.tgz
Found package file /var/db/system/UpdateUpdate-m4DAlb/FreeNASUI-9.3-STABLE-b2b14ab-01034a4-d191868.tgz
Searching for freenas-pkg-tools-9.3-STABLE-e9293b2.tgz
Searching for freenas-pkg-tools-9.3-STABLE-d58c0c8-9.3-STABLE-e9293b2.tgz
Found package file /var/db/system/UpdateUpdate-m4DAlb/freenas-pkg-tools-9.3-STABLE-d58c0c8-9.3-STABLE-e9293b2.tgz
Installer got packages [{u'base-os': <open file u'/var/db/system/UpdateUpdate-m4DAlb/base-os-9.3-STABLE-b2b14ab-01034a4-d191868.tgz', mode 'r' at 0x8061f09c0>}, {u'FreeNASUI': <open file u'/var/db/system/UpdateUpdate-m4DAlb/FreeNASUI-9.3-STABLE-b2b14ab-01034a4-d191868.tgz', mode 'r' at 0x806ca5b70>}, {u'freenas-pkg-tools': <open file u'/var/db/system/UpdateUpdate-m4DAlb/freenas-pkg-tools-9.3-STABLE-d58c0c8-9.3-STABLE-e9293b2.tgz', mode 'r' at 0x806ca5f60>}]
Installing package {u'base-os': <open file u'/var/db/system/UpdateUpdate-m4DAlb/base-os-9.3-STABLE-b2b14ab-01034a4-d191868.tgz', mode 'r' at 0x8061f09c0>}
base-os-9.3-STABLE-b2b14ab-01034a4-d191868
upgrade_aware = True
Update got exception during update: [Errno 28] No space left on device: '/tmp/tmpzDjOCl/usr/local/lib/python2.7/site-packages/django/contrib/formtools/locale/ne/LC_MESSAGES/django.mo'
RunCommand(/sbin/umount, ['-f', '/tmp/tmpzDjOCl/boot/grub'])
RunCommand(/sbin/mount, ['/boot/grub'])
RunCommand(/sbin/umount, ['-f', '/tmp/tmpzDjOCl/dev'])
RunCommand(/sbin/umount, ['-f', '/tmp/tmpzDjOCl/var/tmp'])
RunCommand(/usr/local/sbin/beadm, ['unmount', '-f', u'FreeNAS-9.3-STABLE-201503200528'])
Unmounted successfully
RunCommand(/usr/local/sbin/beadm, ['destroy', '-F', u'FreeNAS-9.3-STABLE-201503200528'])
GRUB configuration updated successfully
Destroyed successfully
Traceback (most recent call last):
  File "/usr/local/bin/freenas-update", line 168, in <module>
    sys.exit(main())
  File "/usr/local/bin/freenas-update", line 153, in main
    rv = Update.ApplyUpdate(download_dir)
  File "/usr/local/lib/freenasOS/Update.py", line 746, in ApplyUpdate
    raise e
OSError: [Errno 28] No space left on device: '/tmp/tmpzDjOCl/usr/local/lib/python2.7/site-packages/django/contrib/formtools/locale/ne/LC_MESSAGES/django.mo'

Running freenas-update update again right now. This is a system I generally only access remote by ssh.


Related issues

Is duplicate of FreeNAS - Bug #11029: Actually prune old boot environments when space is running low and need to create new oneClosed: Not To Be Fixed2015-07-292016-09-19

History

#1 Updated by Jordan Hubbard over 5 years ago

  • Category set to 1
  • Assignee set to Sean Fagan
  • Target version set to Unspecified

#2 Updated by Sean Fagan over 5 years ago

  • Status changed from Unscreened to 15
  • Assignee changed from Sean Fagan to Cy Mike

No space left. Clean up some older boot environments.

See how nicely it cleaned up after itself, though?

Let me know if you're not actually low on space in the boot pool.

#3 Updated by Cy Mike over 5 years ago

After getting that message and running the update command again, it has been "stuck" here for about 45+ minutes now:

Searching for freenas-pkg-tools-9.3-STABLE-d58c0c8-9.3-STABLE-896d06d.tgz
Found package file /var/db/system/UpdateUpdate-cPRzBT/freenas-pkg-tools-9.3-STABLE-d58c0c8-9.3-STABLE-896d06d.tgz
Installer got packages [{u'base-os': <open file u'/var/db/system/UpdateUpdate-cPRzBT/base-os-9.3-STABLE-896d06d-e006356-d191868.tgz', mode 'r' at 0x8061f09c0>}, {u'FreeNASUI': <open file u'/var/db/system/UpdateUpdate-cPRzBT/FreeNASUI-9.3-STABLE-896d06d-e006356-d191868.tgz', mode 'r' at 0x806ca4b70>}, {u'freenas-pkg-tools': <open file u'/var/db/system/UpdateUpdate-cPRzBT/freenas-pkg-tools-9.3-STABLE-d58c0c8-9.3-STABLE-896d06d.tgz', mode 'r' at 0x806ca4f60>}]
Installing package {u'base-os': <open file u'/var/db/system/UpdateUpdate-cPRzBT/base-os-9.3-STABLE-896d06d-e006356-d191868.tgz', mode 'r' at 0x8061f09c0>}
base-os-9.3-STABLE-896d06d-e006356-d191868
upgrade_aware = True

load: 0.28  cmd: python2.7 10274 [tx->tx_quiesce_done_cv] 4294.44r 21.13u 7.06s 0% 90164k
load: 0.26  cmd: python2.7 10274 [tx->tx_quiesce_done_cv] 4298.94r 21.13u 7.06s 0% 90164k

When I log in to another ssh session to the box I grabbed the following:

[root@freenasmini-raptor] ~# cat /etc/version
FreeNAS-9.3-STABLE-201503071634
[root@freenasmini-raptor] ~# ps -auxwwww|grep update
root       10274   0.0  0.5 179424  90232  0  D+    1:14PM      0:28.22 /usr/local/bin/python /usr/local/bin/freenas-update update (python2.7)
root       13066   0.0  0.0  16276   1936  1  S+    2:31PM      0:00.00 grep update
[root@freenasmini-raptor] ~# beadm list
BE                              Active Mountpoint          Space Created
default                         -      -                   10.2M 2015-01-18 17:01
FreeNAS-9.3-STABLE-201501162230 -      -                    5.4M 2015-01-18 18:05
FreeNAS-9.3-STABLE-201501241715 -      -                   87.0M 2015-01-24 12:45
FreeNAS-9.3-STABLE-201502142001 -      -                  933.1M 2015-02-15 19:03
FreeNAS-9.3-STABLE-201502271818 -      -                  276.1M 2015-02-27 16:08
FreeNAS-9.3-STABLE-201503071634 NR     /                    3.1G 2015-03-09 11:33
FreeNAS-9.3-STABLE-201503270027 -      /var/tmp/tmph9Pm_I 573.0M 2015-03-27 13:39
[root@freenasmini-raptor] ~# 

#4 Updated by Sean Fagan over 5 years ago

ZFS behaves badly when it's very low on space.

How much free space is on root? My guess is that it's pretty low.

#5 Updated by Cy Mike over 5 years ago

[root@freenasmini-raptor] ~# df -h
Filesystem                                              Size    Used   Avail Capacity  Mounted on
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201503071634       921M    916M    5.1M    99%    /

[root@freenasmini-raptor] ~# zpool status freenas-boot
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Mon Feb 23 03:46:22 2015
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

[root@freenasmini-raptor] ~# zpool list
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  3.81G  3.69G   125M         -      -    96%  1.00x  ONLINE  -
tank          5.44T  1.14T  4.30T         -    11%    20%  1.00x  ONLINE  /mnt

#6 Updated by Cy Mike over 5 years ago

Finally ended second command attempt with:

Update got exception during update: [Errno 2] No such file or directory: u'/tmp/tmph9Pm_I/usr/local/lib/python2.7/site-packages/django/contrib/admindocs/locale/ar/LC_MESSAGES/django.mo.new'
RunCommand(/sbin/umount, ['-f', '/tmp/tmph9Pm_I/boot/grub'])
RunCommand(/sbin/mount, ['/boot/grub'])
RunCommand(/sbin/umount, ['-f', '/tmp/tmph9Pm_I/dev'])
RunCommand(/sbin/umount, ['-f', '/tmp/tmph9Pm_I/var/tmp'])
RunCommand(/usr/local/sbin/beadm, ['unmount', '-f', u'FreeNAS-9.3-STABLE-201503270027'])
Unmounted successfully
RunCommand(/usr/local/sbin/beadm, ['destroy', '-F', u'FreeNAS-9.3-STABLE-201503270027'])
GRUB configuration updated successfully
Destroyed successfully
Traceback (most recent call last):
  File "/usr/local/bin/freenas-update", line 168, in <module>
    sys.exit(main())
  File "/usr/local/bin/freenas-update", line 153, in main
    rv = Update.ApplyUpdate(download_dir)
  File "/usr/local/lib/freenasOS/Update.py", line 746, in ApplyUpdate
    raise e
IOError: [Errno 2] No such file or directory: u'/tmp/tmph9Pm_I/usr/local/lib/python2.7/site-packages/django/contrib/admindocs/locale/ar/LC_MESSAGES/django.mo.new

Destroying older BEs and trying again:

[root@freenasmini-raptor] ~# beadm list
BE                              Active Mountpoint  Space Created
default                         -      -           10.2M 2015-01-18 17:01
FreeNAS-9.3-STABLE-201501162230 -      -            5.4M 2015-01-18 18:05
FreeNAS-9.3-STABLE-201501241715 -      -           87.0M 2015-01-24 12:45
FreeNAS-9.3-STABLE-201502142001 -      -          933.1M 2015-02-15 19:03
FreeNAS-9.3-STABLE-201502271818 -      -          276.1M 2015-02-27 16:08
FreeNAS-9.3-STABLE-201503071634 NR     /            3.1G 2015-03-09 11:33
[root@freenasmini-raptor] ~# beadm destroy FreeNAS-9.3-STABLE-201501162230
Are you sure you want to destroy 'FreeNAS-9.3-STABLE-201501162230'?
This action cannot be undone (y/[n]): y
Boot environment 'FreeNAS-9.3-STABLE-201501162230' was created from existing snapshot
Destroy 'FreeNAS-9.3-STABLE-201503071634@2015-01-24-12:45:18' snapshot? (y/[n]): y
GRUB configuration updated successfully
Destroyed successfully
[root@freenasmini-raptor] ~# beadm destroy FreeNAS-9.3-STABLE-201501241715
Are you sure you want to destroy 'FreeNAS-9.3-STABLE-201501241715'?
This action cannot be undone (y/[n]): y
Boot environment 'FreeNAS-9.3-STABLE-201501241715' was created from existing snapshot
Destroy 'FreeNAS-9.3-STABLE-201503071634@2015-02-15-19:03:38' snapshot? (y/[n]): y
GRUB configuration updated successfully
Destroyed successfully
[root@freenasmini-raptor] ~# beadm destroy FreeNAS-9.3-STABLE-201502142001
Are you sure you want to destroy 'FreeNAS-9.3-STABLE-201502142001'?
This action cannot be undone (y/[n]): y
Boot environment 'FreeNAS-9.3-STABLE-201502142001' was created from existing snapshot
Destroy 'FreeNAS-9.3-STABLE-201503071634@2015-02-27-16:08:36' snapshot? (y/[n]): y
GRUB configuration updated successfully
Destroyed successfully
[root@freenasmini-raptor] ~# beadm destroy FreeNAS-9.3-STABLE-201502271818
Are you sure you want to destroy 'FreeNAS-9.3-STABLE-201502271818'?
This action cannot be undone (y/[n]): y
Boot environment 'FreeNAS-9.3-STABLE-201502271818' was created from existing snapshot
Destroy 'FreeNAS-9.3-STABLE-201503071634@2015-03-09-11:33:49' snapshot? (y/[n]): y
GRUB configuration updated successfully
Destroyed successfully
[root@freenasmini-raptor] ~# zpool list freenas-boot
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  3.81G  1.85G  1.96G         -      -    48%  1.00x  ONLINE  -
[root@freenasmini-raptor] ~# df -h
Filesystem                                              Size    Used   Avail Capacity  Mounted on
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201503071634       2.8G    916M    1.9G    33%    /

...

[root@freenasmini-raptor] ~# freenas-update update
TryGetNetworkFile(http://update.freenas.org/FreeNAS/FreeNAS-9.3-STABLE/LATEST)
TryGetNetworkFile(http://update.freenas.org/FreeNAS/FreeNAS-9.3-STABLE/LATEST):  Read 5092 bytes total
TryGetNetworkFile(https://web.ixsystems.com/updates/ix_crl.pem)
TryGetNetworkFile(https://web.ixsystems.com/updates/ix_crl.pem):  Read 1028 bytes total
Verify command = ['/usr/local/libexec/verify_signature', '-K', '/usr/local/share/certs/freenas-update.pem', '-C', '/usr/local/share/certs/iX-CA.pem', '-S', u'dw/KAbu9yqmMvfNxl7TyXl/d/0i5S02u+6zhbwzDi1sVEeLGuYLE5aLKUzw8J+Nbsx6MkoZ8Vm0ttVOV7QfM2ClV3gO5ck95HFpI1o4/fJdO1j2NOVvIMvohPjTxEvwZagw6wdMwpMO5dUwQWBunu18jjwgjZPYu4cC3Ej7BJ2iHy6WXccabU3rJdjE0qJ2RZWV3dmWj3oSKiDx2Fz2QwmEW/gkRxy7c2vOM/emmrZfRrhcBGoRMQij9n7kN/Optmp/6EJXvZssP3+7KKsMXjoj5FIUzyZ7GNth4cBw4oG6UpMVfkdsg1/2IJPYdKb0SH5oon6wd+zhcJ9EOKvle3Q==', '-R', '/tmp/tmpttkHdH.pem']
Signature check succeeded
Removing invalid or incomplete cached update
DownloadUpdate:  Will upgrade package base-os
DownloadUpdate:  Will upgrade package FreeNASUI
DownloadUpdate:  Will upgrade package freenas-pkg-tools
Searching for base-os-9.3-STABLE-896d06d-e006356-d191868.tgz
-snip-
+ /usr/local/bin/python /usr/local/www/freenasUI/manage.py migrate --all --merge --delete-ghost-migrations
yes: stdout: Broken pipe
+ rm -f /data/update.failed
Installing package {u'freenas-pkg-tools': <open file u'/var/db/system/UpdateUpdate-YXnvVw/freenas-pkg-tools-9.3-STABLE-d58c0c8-9.3-STABLE-896d06d.tgz', mode 'r' at 0x806ca4f60>}
freenas-pkg-tools-9.3-STABLE-896d06d
upgrade_aware = False
Updated package freenas-pkg-tools from 9.3-STABLE-d58c0c8 to 9.3-STABLE-896d06d in database
RunCommand(/sbin/umount, ['-f', '/tmp/tmp9ngYFR/boot/grub'])
RunCommand(/sbin/mount, ['/boot/grub'])
RunCommand(/sbin/umount, ['-f', '/tmp/tmp9ngYFR/dev'])
RunCommand(/sbin/umount, ['-f', '/tmp/tmp9ngYFR/var/tmp'])
RunCommand(/usr/local/sbin/beadm, ['unmount', '-f', u'FreeNAS-9.3-STABLE-201503270027'])
Unmounted successfully
RunCommand(/usr/local/sbin/beadm, ['activate', u'FreeNAS-9.3-STABLE-201503270027'])
GRUB configuration updated successfully
Activated successfully
System should be rebooted now

I imagine the broken pipe message can be ignored considering this says the update and activation of the new BE was successful.

#7 Updated by Sean Fagan over 5 years ago

  • Status changed from 15 to Investigation

Yeah, the broken pipe complaint is because the 'yes | whatever' has the python script exit, meaning yes is writing to a broken pipe. Correct behaviour there.

Keeping to investigate changes I can make to have the no-space issue handled better.

#8 Updated by Cy Mike over 5 years ago

  • Assignee changed from Cy Mike to Sean Fagan

#9 Updated by Jordan Hubbard over 5 years ago

  • Subject changed from traceback on update command to freenas update process should do a better job of reporting the out-of-space condition

Changing Subject to more accurately match the objective of this ticket.

#10 Updated by Cy Mike almost 5 years ago

Any chance of this making it into the next FN9 SU or should it be punted to be a future FN10 SU ?

#11 Updated by Jordan Hubbard over 4 years ago

  • Is duplicate of Bug #11029: Actually prune old boot environments when space is running low and need to create new one added

#12 Updated by Jordan Hubbard over 4 years ago

  • Status changed from Investigation to Closed: Duplicate

#13 Avatar?id=14398&size=24x24 Updated by Kris Moore about 3 years ago

  • Target version changed from Unspecified to N/A

Also available in: Atom PDF