Project

General

Profile

Bug #28133

Fix error that prevented proper deletion of boot environment

Added by Caleb St. John about 1 year ago. Updated about 1 year ago.

Status:
Done
Priority:
Expected
Assignee:
William Grzybowski
Category:
Middleware
Target version:
Seen in:
TrueNAS - TrueNAS 11.1-U2
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Related projects 1 project

Description

If you browse to System->Boot and delete a boot environment, it will almost instantaneously state "Failed to delete boot environment". However, the webUI updates and no longer shows the boot environment.

If you then ssh to the node and issue a "beadm list" it will show you the boot environment that was never deleted.

So the webUI is removing the entry, however, the entry still exists on the file system.

Attaching debug to this ticket from the affected system.

Create BE traceback.png (969 KB) Create BE traceback.png Joe Maloney, 02/13/2018 12:36 PM
14356

Related issues

Related to FreeNAS - Bug #28476: Boot environments deleted after upgrade......Closed2018-02-13
Related to FreeNAS - Bug #28594: renaming boot environment throws tracebackClosed2018-02-20
Related to FreeNAS - Bug #28595: cloning a boot environment throws a tracebackClosed2018-02-20

Associated revisions

Revision d4cfed01 (diff)
Added by William Grzybowski about 1 year ago

fix(pkg): generate python cache files and add to pkg-plist automatically

Ticket: #28133

Revision c1abfc06 (diff)
Added by William Grzybowski about 1 year ago

fix(pkg): generate python cache files and add to pkg-plist automatically

Ticket: #28133
(cherry picked from commit d4cfed01ceac9e89bba0f0ffbf876f791392f172)

Revision b182cca9 (diff)
Added by William Grzybowski about 1 year ago

fix(pkg): make freenas-pkgtools depend on python

Ticket: #28133

Revision ab3de1fd (diff)
Added by William Grzybowski about 1 year ago

fix(pkg): make freenas-pkgtools depend on python

Ticket: #28133
(cherry picked from commit b182cca91398bc94f5e22abfe4aa28d1bdd9d09d)

Revision 86b6e69e (diff)
Added by William Grzybowski about 1 year ago

fix(pkg): generate python cache files and add to pkg-plist automatically

Ticket: #28133

Revision 1c352a38 (diff)
Added by William Grzybowski about 1 year ago

fix(pkg): make freenas-pkgtools depend on python

Ticket: #28133

Revision 410a9de1 (diff)
Added by William Grzybowski about 1 year ago

fix(middlewared): make sure to close fd when using reserved_ports client

Ticket: #28133

Revision 4fa9ca3b (diff)
Added by William Grzybowski about 1 year ago

fix(middlewared): make sure to close fd when using reserved_ports client

Ticket: #28133
(cherry picked from commit 410a9de1c32ed8e6388fc7469c4a94ae009ce3a9)

Revision 4b104d95 (diff)
Added by William Grzybowski about 1 year ago

fix(middlewared): make sure to close fd when using reserved_ports client

Ticket: #28133

History

#1 Updated by Dru Lavigne about 1 year ago

  • Category set to Middleware
  • Assignee changed from Ash Gokhale to Vladimir Vinogradenko
  • Priority changed from Critical to Expected
  • Target version changed from N/A to TrueNAS 11.1-U2
  • Severity set to Medium

#2 Avatar?id=13649&size=24x24 Updated by Ben Gadd about 1 year ago

  • Assignee changed from Vladimir Vinogradenko to Brandon Schneider

#5 Updated by Dru Lavigne about 1 year ago

  • Assignee changed from Caleb St. John to William Grzybowski

#6 Updated by William Grzybowski about 1 year ago

  • Assignee changed from William Grzybowski to Brandon Schneider

Misassigned.

#7 Updated by Brandon Schneider about 1 year ago

  • Status changed from Not Started to In Progress

#8 Updated by Brandon Schneider about 1 year ago

  • Status changed from In Progress to Blocked
  • Assignee changed from Brandon Schneider to Joe Maloney
  • Reason for Blocked set to Need additional information

I cannot replicate this on a fresh FreeNAS or TrueNAS install.

Looking at the logs, it would seem to be a trashed database

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 150, i
n call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 128, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 908, i
n call_method
    return await self._call(message['method'], serviceobj, methodobj, params, ap
p=app)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 128, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 886, i
n _call
    return await self.run_in_thread(methodobj, *args)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 128, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 849, in run_in_thread
    return await self.run_in_thread_pool(self.__threadpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 128, in throw
    return self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 845, in run_in_thread_pool
    await task
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 259, in sql
    cursor.executelocal(query, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 434, in executelocal
    return self.locked_retry(Database.Cursor.execute, query, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
    rv = method(self, *args, **kwargs)
sqlite3.OperationalError: no such column: disk_enabled

Marking as blocked so Joe can confirm on a different machine as well.

#9 Avatar?id=14398&size=24x24 Updated by Kris Moore about 1 year ago

Possibly a bad migration?

#10 Updated by Brandon Schneider about 1 year ago

Not sure, funny enough restarting middlewared and everything now creates and destroys correctly.

#11 Avatar?id=13649&size=24x24 Updated by Ben Gadd about 1 year ago

  • 1 added project (FreeNAS)
  • Due date set to 02/14/2018

#12 Updated by Dru Lavigne about 1 year ago

  • Target version changed from TrueNAS 11.1-U2 to 11.1-U2

#13 Updated by Dru Lavigne about 1 year ago

  • Related to Bug #28476: Boot environments deleted after upgrade...... added

#14 Updated by Joe Maloney about 1 year ago

14356

It is not just deleting a boot environment that fails. The following are broken after upgrade to latest nightlies:

  • Creating throws a traceback as shown in the attached image
  • Deleting "Failed to delete boot environment"
  • Activating "Failed to activate boot environment"

From what I can tell activate, and create do work when I look at the output of beadm list. Restarting middlewared also makes the errors go away entirely until reboot. Also I cannot reproduce this on an HA Z series, only this X10.

#15 Updated by Joe Maloney about 1 year ago

  • Assignee changed from Joe Maloney to Brandon Schneider

William asked to reassign back to Brandon.

#16 Avatar?id=14398&size=24x24 Updated by Kris Moore about 1 year ago

  • Target version changed from 11.1-U2 to TrueNAS 11.1-U2

#17 Updated by Brandon Schneider about 1 year ago

  • Status changed from Blocked to Closed
  • Reason for Closing set to Cannot Reproduce
  • Reason for Blocked deleted (Need additional information)

Closing as I believe I found the culprit: stale .pyc files.

I was able to reproduce the same thing Joe was seeing (reboot the machines, traceback occurs until a service restart), but after removing the bootenv pyc files and freenasOS/Update.py's pyc's from both nodes, I can no longer recreate this traceback. I've tried multiple reboots of the machines (not the middlewared service which did solve the issue before as noted)

I've also tried on the CLI on both machines independently before launching the UI, and then to make sure on the UI.

So I suppose we can consider this solved.

#18 Updated by William Grzybowski about 1 year ago

  • Status changed from Closed to Not Started
  • Reason for Closing deleted (Cannot Reproduce)

I will take a crack at it. I think saying its pyc does not cut it.

#19 Updated by William Grzybowski about 1 year ago

  • Assignee changed from Brandon Schneider to William Grzybowski

#20 Updated by Joe Maloney about 1 year ago

I am also going to confirm whether or not this is an issue on a second X10 that was racked, and installed fresh yesterday. I should have an update later this morning.

#22 Updated by William Grzybowski about 1 year ago

  • Status changed from Not Started to In Progress

#23 Updated by William Grzybowski about 1 year ago

  • Status changed from In Progress to Done

#24 Updated by Dru Lavigne about 1 year ago

  • Subject changed from deleting boot environment failing to Fix error that prevented proper deletion of boot environment
  • Target version changed from TrueNAS 11.1-U2 to 11.2-RC2

#25 Updated by Dru Lavigne about 1 year ago

  • Target version changed from 11.2-RC2 to 11.1-U2

#27 Updated by Dru Lavigne about 1 year ago

  • Related to Bug #28594: renaming boot environment throws traceback added

#28 Updated by Dru Lavigne about 1 year ago

  • Related to Bug #28595: cloning a boot environment throws a traceback added

#30 Updated by Dru Lavigne about 1 year ago

  • Status changed from Not Started to Done
  • QA Status Test Passes FreeNAS added

This works in FreeNAS. QA will let us know if the test fails on TrueNAS. For now, marking back as DONE.

#31 Updated by William Grzybowski about 1 year ago

  • Status changed from Done to Not Started

They have shown the issue on U2-RC7 in TrueNAS, there is nothing needed to show as far as I am concerned.

#33 Updated by Dru Lavigne about 1 year ago

  • Target version changed from 11.1-U2 to TrueNAS 11.1-U2

#34 Updated by William Grzybowski about 1 year ago

  • Status changed from Not Started to Done

#35 Updated by Joe Maloney about 1 year ago

  • Needs QA changed from Yes to No
  • QA Status Test Passes TrueNAS added
  • QA Status deleted (Not Tested)

#36 Updated by Dru Lavigne about 1 year ago

  • File deleted (debug-truenas03a-20180201112051.tgz)

#37 Updated by Dru Lavigne about 1 year ago

  • Target version changed from TrueNAS 11.1-U2 to TrueNAS 11.1-U3

#38 Updated by Dru Lavigne about 1 year ago

  • Hide from ChangeLog changed from No to Yes

#39 Updated by Dru Lavigne about 1 year ago

  • 1 deleted project (FreeNAS)
  • Project changed from TrueNAS to FreeNAS
  • Description updated (diff)
  • Category changed from Middleware to Middleware
  • Target version changed from TrueNAS 11.1-U3 to 11.1-U3
  • Migration Needed deleted (No)
  • Hide from ChangeLog deleted (Yes)
  • Support Department Priority deleted (0)

Also available in: Atom PDF