Project

General

Profile

Bug #26899

Improve VM shutdown and add Power Off button

Added by Patrick M. Hausen almost 3 years ago. Updated over 2 years ago.

Status:
Done
Priority:
No priority
Assignee:
Marcelo Araujo
Category:
Middleware
Target version:
Seen in:
Severity:
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Subject says it, essentially. Boot e.g. FreeBSD based VM, click on "Stop".
At next "Start" watch console - filesystem will be dirty due to hard power off.

Config of VM is shown in the attachments. The two devices are the only ones.

Kind regards,
Patrick

device-disk.png (46.1 KB) device-disk.png Config of virtual disk device Patrick M. Hausen, 11/28/2017 10:42 AM
device-nic.png (53.3 KB) device-nic.png Config of virtual NIC Patrick M. Hausen, 11/28/2017 10:42 AM
settings-vm.png (69 KB) settings-vm.png Config of VM Patrick M. Hausen, 11/28/2017 10:42 AM
13226
13227
13228

Related issues

Related to FreeNAS - Bug #28211: bhyve crashing on VM shutdownClosed

Associated revisions

Revision 68778ff5 (diff)
Added by Marcelo Araujo over 2 years ago

- Change the behaviour of 'Stop', now instead of a force-poweroff we do a SIGTERM, if the guest machine supports 'ACPI shutdown' it - Add a 'Power off' button in the old UI. - With the 'Power off' button we perform a true 'power off'. Ticket: #26899

Revision cda8af4e (diff)
Added by Marcelo Araujo over 2 years ago

- Change the behaviour of 'Stop', now instead of a force-poweroff we do a SIGTERM, if the guest machine supports 'ACPI shutdown' it will shutdown graceful. - Add a 'Power off' button in the old UI. - With the 'Power off' button we perform a true 'power off'. Ticket: #26899

Revision d25ae085 (diff)
Added by Marcelo Araujo over 2 years ago

- Change the behaviour of 'Stop', now instead of a force-poweroff we do a SIGTERM, if the guest machine supports 'ACPI shutdown' it will shutdown graceful. - Add a 'Power off' button in the old UI. - With the 'Power off' button we perform a true 'power off'. Ticket: #26899

Revision 2c4aecc7 (diff)
Added by Marcelo Araujo over 2 years ago

[STABLE] - Change the behaviour of 'Stop', now instead of a force-poweroff we … (#633) * - Change the behaviour of 'Stop', now instead of a force-poweroff we do a SIGTERM, if the guest machine supports 'ACPI shutdown' it will shutdown graceful. - Add a 'Power off' button in the old UI. - With the 'Power off' button we perform a true 'power off'. Ticket: #26899 * - Fix wrong signal, it must be 0 as we are only testing if the proc is running. Spotted by: william-gr

Revision 447d022e (diff)
Added by Marcelo Araujo over 2 years ago

[MASTER] - Change the behaviour of 'Stop', now instead of a force-poweroff we do a SIGTERM, if the guest machine supports 'ACPI shutdown' it will shutdown graceful. (#632) * - Change the behaviour of 'Stop', now instead of a force-poweroff we do a SIGTERM, if the guest machine supports 'ACPI shutdown' it - Add a 'Power off' button in the old UI. - With the 'Power off' button we perform a true 'power off'. Ticket: #26899 * - Change the behaviour of 'Stop', now instead of a force-poweroff we do a SIGTERM, if the guest machine supports 'ACPI shutdown' it will shutdown graceful. - Add a 'Power off' button in the old UI. - With the 'Power off' button we perform a true 'power off'. Ticket: #26899 * - Fix wrong signal, it must be 0 as we are only testing if the proc is running. Spotted by: william-gr * - Remove a blank extra line.

Revision e65871eb (diff)
Added by Warren Block over 2 years ago

Update VM docs Ticket: #23795 Ticket: #25529 Ticket: #26899

History

#1 Updated by Dru Lavigne almost 3 years ago

  • Assignee changed from Release Council to Marcelo Araujo

#2 Updated by Marcelo Araujo almost 3 years ago

  • Status changed from Unscreened to Screened

#3 Updated by Dru Lavigne almost 3 years ago

  • Target version set to 11.1-U1

#4 Updated by Marcelo Araujo almost 3 years ago

  • Status changed from Screened to Closed: Behaves correctly

Patrick M. Hausen wrote:

Subject says it, essentially. Boot e.g. FreeBSD based VM, click on "Stop".
At next "Start" watch console - filesystem will be dirty due to hard power off.

Config of VM is shown in the attachments. The two devices are the only ones.

Kind regards,
Patrick

The "STOP" option behave like you press the power button of your machine, and "RESTART" is like you push the reset button of your machine.
For clean STOP and RESTART, you need to do it from inside the GUEST using the operation system, such like: reboot, halt -p and etc.

Best,

#5 Updated by Patrick M. Hausen almost 3 years ago

Sorry, but I disagree. I can accept your argument about the buttons, but if I shutdown my FreeNAS system I expect all VMs and jails to be cleanly (!) shut down first.

Kind regards
Patrick

#6 Updated by Marcelo Araujo almost 3 years ago

Patrick M. Hausen wrote:

Sorry, but I disagree. I can accept your argument about the buttons, but if I shutdown my FreeNAS system I expect all VMs and jails to be cleanly (!) shut down first.

Kind regards
Patrick

It is hard to achieve it with hypervisor level 2 without have intrusive code inside your guest.
To make a clean shutdown inside your guest before FreeNAS finish its shutdown, I need a way to connect into your guest, nowadays bhyve doesn't provide it.

Best,

#7 Updated by Patrick M. Hausen almost 3 years ago

While we have to do without guest additions - BTW there was a side project in Corral ... waitaminute ... https://github.com/freenas/freenas-vm-tools - what about sending ACPI power off and waiting a configurable time before a hard power off?

Kind regards
Patrick

#8 Updated by Marcelo Araujo almost 3 years ago

  • Status changed from Closed: Behaves correctly to Investigation
  • Target version changed from 11.1-U1 to 11.2-BETA1

#9 Updated by Marcelo Araujo almost 3 years ago

Patrick M. Hausen wrote:

While we have to do without guest additions - BTW there was a side project in Corral ... waitaminute ... https://github.com/freenas/freenas-vm-tools - what about sending APCI power off and waiting a configurable time before a hard power off?

Kind regads
Patrick

I checked it while ago, let me do it again and see the constraints to have it running on FreeBSD guests at least. As I recall, it is an option for Linux, but it is not on top of my mind about FreeBSD guests.

I bump the target version for FreeNAS 11.2, in case I can figure out it before, I can include it on 11.1-U1.

Thanks to point it out.

#10 Updated by Marcelo Araujo almost 3 years ago

Patrick M. Hausen wrote:

While we have to do without guest additions - BTW there was a side project in Corral ... waitaminute ... https://github.com/freenas/freenas-vm-tools - what about sending ACPI power off and waiting a configurable time before a hard power off?

Kind regards
Patrick

Just to give a bit more of information about the ACPI power off, we could use a kill -SIGTERM, if guest supports ACPI shutdown, the guest will shutdown cleanly and the corresponding bhyve process will terminate, the vmm entry will still be there and can be cleaned up after. However there is no mechanism nowadays to guarantee it and FreeNAS reboot can get stuck.

I still believe, have a tool such like freenas-vm-tools inside the guest is the proper way to do it, and it will solve other problems too, such like get information about cpu, memory and ifaces from inside the guest.

Thanks again to point that out, I definitely will take a look on it in the coming days.

#11 Updated by Patrick M. Hausen almost 3 years ago

They work on HEAD as guest:

https://forums.freenas.org/index.php?threads/guest-health-light-colors.50591/
https://forums.freenas.org/index.php?threads/bhyve-guest-tools.49908/

They need named ports in virtio_console(4), I don't know right now if that has been MFC'd to RELENG_11.
Edit: just checked, nope. The code has not been MFC'd.

Unfortunately all the Redmine issues for Corral are not accessible, anymore - otherwise you would even have a FreeBSD port to start with:
https://bugs.freenas.org/issues/21516

Patrick

#12 Updated by Joshua Senkevech almost 3 years ago

My 2 cents

In 11-U4 the stop button would attempt a clean shutdown. It worked fine for most OS.

The issue people wanted resolved was a button for a forced stop. It seems whoever did the feature request made the stop button a force stop instead of just adding another button.

I personally would like three buttons, stop, restart, force stop. I think that is how a lot of people would like to handle the VM issues that pop up instead of a kill -9 command.

#13 Updated by Patrick M. Hausen almost 3 years ago

I support Joshua's request. Providing both options would be a good thing [tm].

Patrick

#14 Updated by Marcelo Araujo over 2 years ago

  • Status changed from Investigation to Needs Developer Review
  • Assignee changed from Marcelo Araujo to William Grzybowski
  • Target version changed from 11.2-BETA1 to 11.1-U1

Patrick M. Hausen wrote:

I support Joshua's request. Providing both options would be a good thing [tm].

Patrick

I added a new button called 'Power off' where it will poweroff the guest and also I changed the 'stop' button to send a SIGTERM to the guest instead of do the poweroff.

NOTE: For the vm-tools, I will open another ticket.

Best,

#15 Updated by Joshua Senkevech over 2 years ago

Not to be too picky, but it seems to me that the "Power off" should be the SIGTERM and the "stop" button should be the halt or forced shutdown.

-JS

#16 Updated by Patrick M. Hausen over 2 years ago

ROFL :)

May I suggest "Power Off" and "Shut Down Guest"?
Or "Power Off" and "ACPI Shutdown"?

Patrick

#17 Updated by William Grzybowski over 2 years ago

  • Status changed from Needs Developer Review to Reviewed by Developer
  • Assignee changed from William Grzybowski to Marcelo Araujo

#18 Updated by Dru Lavigne over 2 years ago

  • Subject changed from "Stop" or "Restart" of a VM from the UI causes a hard power off instead of a clean shutdown of the VM to Improve VM shutdown and add Power Off button

#19 Updated by Dru Lavigne over 2 years ago

  • Status changed from Reviewed by Developer to Ready For Release

#20 Updated by Dru Lavigne over 2 years ago

  • Status changed from Ready For Release to Resolved

#21 Updated by Patrick M. Hausen over 2 years ago

Hey, guys,

in 11.1-U1 the "Stop" button still does a hard power off of the VM. Provable by connecting to the serial console before stopping and watching the messages of UFS journal replay during the next start up.

Kind regards
Patrick

#22 Updated by Dru Lavigne over 2 years ago

  • Status changed from Resolved to Done

#23 Updated by Dru Lavigne over 2 years ago

  • Status changed from Done to Blocked
  • Target version changed from 11.1-U1 to 11.1-U2
  • Reason for Blocked set to Need verification

#24 Updated by Dru Lavigne over 2 years ago

  • Seen in changed from 11.1-RC1 to 11.1-U1

#25 Updated by Marcelo Araujo over 2 years ago

Patrick M. Hausen wrote:

Hey, guys,

in 11.1-U1 the "Stop" button still does a hard power off of the VM. Provable by connecting to the serial console before stopping and watching the messages of UFS journal replay during the next start up.

Kind regards
Patrick

Hi Patrick,

The implementation is right, it send a SIGTERM to bhyve process and if the guest supports ACPI, it should shutdown properly.
An easy test is, login into your guest VM and using the old GUI, press "Stop", you will see the guest shutdown process.

Does your guest supports ACPI?

#26 Updated by Patrick M. Hausen over 2 years ago

Hi Marcelo,

I did precisely that - logged into the VM (FreeBSD 11.1), pressed "Stop" --> immediate power off.

I just tried once more - "Stop" and "Power Off" seem to do the same thing, namely power off the machine. On next boot:

Trying to mount root from ufs:/dev/gpt/root [rw]...
WARNING: / was not properly dismounted

Contrast:

root@freenas-pmh:~ # ps awwux | grep bhyve
root       82202  11.9  3.3 5307652 1111560  -  S    Sun17     347:48.95 bhyve: 5_Windows (bhyve)
root       84676   2.7  4.8 5278500 1604344  -  S    Sun18     136:22.06 bhyve: 3_Wiki (bhyve)
root       85500   1.4  2.6 5278500  876096  -  S    Sun18      36:20.02 bhyve: 4_Cloud (bhyve)
root       34180   0.5  0.2 1082020   53952  -  S    08:06       0:14.15 bhyve: 1_DHCP (bhyve)
root       89628   0.1  0.9 5278500  312664  -  S    Sun19      12:54.82 bhyve: 6_Project (bhyve)
root       35115   0.0  0.0    6696    2564  1  S+   08:11       0:00.00 grep bhyve
root@freenas-pmh:~ # kill -TERM 34180

leads to:

FreeBSD/amd64 (dhcp) (ttyu0)

login: Stopping cron.
Stopping apache24.
Waiting for PIDS: 620.
Stopping sshd.
Stopping ntpd.
Stopping dhcpd.
Stopping named.
Stopping devd.
Writing entropy file:.
Writing early boot entropy file:.
Terminated
.
Feb  6 08:11:33 dhcp syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 0 0 done
All buffers synced.
Uptime: 4m33s
arp: 217.29.46.111 moved from c8:69:cd:65:e2:a8 to 60:f4:45:e6:15:0f on vtnet0
acpi0: Powering system off

So the buttons seem to do something different ;)

Patrick

#27 Updated by Marcelo Araujo over 2 years ago

Patrick M. Hausen wrote:

Hi Marcelo,

I did precisely that - logged into the VM (FreeBSD 11.1), pressed "Stop" --> immediate power off.

Patrick

I just created a FreeBSD(HEAD) guest with UFS and I made the tests using "Stop" and booting again the VM; and there is no replay.
Also during the "Stop" process, I can see the vnodes syncing and services such like sshd and moused shutting down properly.

Can you do this process and attach the log /var/log/middlewared.log ?

When you use "Stop" it should log something like:
===> Soft Stop VM: FreeBSD12UFS ID: 135 BHYVE_CODE: None

Best,

#28 Updated by Patrick M. Hausen over 2 years ago

middleware.log:

[2018/02/06 07:16:52] (DEBUG) VMService.stop():361 - ===> Soft Stop VM: DHCP ID: 1 BHYVE_CODE: None
[2018/02/06 07:16:52] (WARNING) VMService.destroy_vm():275 - ===> Destroying VM: DHCP ID: 1 BHYVE_CODE: None
[2018/02/06 07:16:52] (DEBUG) VMService.run():250 - DHCP: Assertion failed: (error == 0), function emulate_inout, file /freenas-11-releng/freenas/_BE/os/usr.sbin/bhyve/inout.c, line 230.

[2018/02/06 07:16:52] (INFO) VMService.run():271 - ===> Error VM: DHCP ID: 1 BHYVE_CODE: -6
[2018/02/06 07:16:52] (WARNING) VMService.destroy_vm():275 - ===> Destroying VM: DHCP ID: 1 BHYVE_CODE: -6

BTW: all my VMs are powered off hard - FreeBSD RELENG_11, Ubuntu 16.04, Windows 10 ... should all support ACPI shutdown.

HTH,
Patrick

#29 Updated by Marcelo Araujo over 2 years ago

Patrick M. Hausen wrote:

middleware.log:
[...]

HTH,
Patrick

OK, so your bhyve is crashing as we can see in that "Assertion failed".

There is a FreeBSD ticket related with similar issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221712

Instead to be connected via console, can you connect via VNC and try the "stop" again and check the middlewared.log to see if that "Assertion" still happens?

Thanks,

#30 Updated by Marcelo Araujo over 2 years ago

  • Status changed from Blocked to Closed
  • Reason for Blocked changed from Need verification to Other: make note in comments

Patrick M. Hausen wrote:

middleware.log:
[...]

BTW: all my VMs are powered off hard - FreeBSD RELENG_11, Ubuntu 16.04, Windows 10 ... should all support ACPI shutdown.

HTH,
Patrick

So you are having a different issue, bhyve is crashing, it needs further investigation, but the problem is not related with this ticket.
I suggest open another ticket related with that "ASSERT ISSUE". Also I need information about your hardware and full log of FreeNAS that you can obtain at: system->advanced->"Save Debug".

Thank you Patrick.

#31 Updated by Patrick M. Hausen over 2 years ago

  • Reason for Blocked changed from Other: make note in comments to Need verification

I don't have VNC devices for my FreeBSD and Linux VMs - what for? ;) All serial consoles ...

I connected via VNC to my Windows 10 VM, then pressed "Stop" in the old UI. Result:

[2018/02/06 07:38:43] (DEBUG) VMService.stop():361 - ===> Soft Stop VM: Windows ID: 5 BHYVE_CODE: None
[2018/02/06 07:38:43] (WARNING) VMService.destroy_vm():275 - ===> Destroying VM: Windows ID: 5 BHYVE_CODE: None
[2018/02/06 07:38:43] (DEBUG) VMService.kill_bhyve_web():340 - ==> Killing WEBVNC: 82205
[2018/02/06 07:38:43] (DEBUG) VMService.run():250 - Windows: Assertion failed: (error == 0), function emulate_inout, file /freenas-11-releng/freenas/_BE/os/usr.sbin/bhyve/inout.c, line 230.

[2018/02/06 07:38:43] (DEBUG) VMService.run():250 - Windows: fbuf frame buffer base: 0x942e00000 [sz 16777216]

[2018/02/06 07:38:44] (INFO) VMService.run():271 - ===> Error VM: Windows ID: 5 BHYVE_CODE: -6
[2018/02/06 07:38:44] (WARNING) VMService.destroy_vm():275 - ===> Destroying VM: Windows ID: 5 BHYVE_CODE: -6
[2018/02/06 07:38:45] (DEBUG) VMService.kill_bhyve_web():340 - ==> Killing WEBVNC: 82205

VNC viewer on my Mac terminated immediately.

Patrick

#32 Updated by Patrick M. Hausen over 2 years ago

#33 Updated by Dru Lavigne over 2 years ago

  • Status changed from Closed to Done
  • Target version changed from 11.1-U2 to 11.1-U1
  • Reason for Blocked deleted (Need verification)

#34 Updated by Dru Lavigne over 2 years ago

  • Related to Bug #28211: bhyve crashing on VM shutdown added

Also available in: Atom PDF