Project

General

Profile

Bug #12135

boot crash

Added by David Potterveld almost 5 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
No priority
Assignee:
Alexander Motin
Category:
OS
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Supermicro 846E 192GB 2x Xeon

ChangeLog Required:
No

Description

After applying the October 28 update (FreeNAS-9.3-STABLE-201510290351) my system is unable to boot. It gets as far as starting the ntpd daemon, which then page faults while in kernel mode. I haven't been able to recover a crash dump to analyze yet, but will continue to attempt to do so. The info I have comes from the console monitor, which I had to video in slo-mo to see:

Fatal trap 12: page fault while in kernel mode
Fault code= supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ad18e6
stack pointer = 0x28:0xfffffe025109d230
frame pointer = 0x28:0xfffffe825189d290
...
current process = 5138 (ntpd)

Booting the prior version (FreeNAS-9.3-STABLE-201509282017) works normally. I know there has been a recent security update to ntp. I don't know if this has made it to FreeNAS yet, but I suspect it has and is causing problems (at least for me).


Related issues

Blocks FreeNAS - Bug #12120: After update to FreeNAS-9.3-STABLE-201510290351, NFS is not workingClosed2015-10-29

History

#1 Updated by Jakub Klama almost 5 years ago

  • Status changed from Unscreened to 15
  • Assignee changed from Jakub Klama to Alexander Motin

I'm afraid we can't really tell anything without a crash dump, please provide one :) ntp update itself shouldn't generate kernel panic (unless there is a separate bug in kernel which got revealed by that change).

#2 Updated by David Potterveld almost 5 years ago

  • File textdump.tar.2 added

I've attached a text dump (default settings) from one of the crashes. I finally figured out how to create a minidump, which is 27 MB compressed, too big to attach here. Let me know how to send that to you if you want to look at that.

#3 Updated by Sean Fagan almost 5 years ago

  • Related to Bug #12120: After update to FreeNAS-9.3-STABLE-201510290351, NFS is not working added

#4 Updated by Sean Fagan almost 5 years ago

This sounds somewhat like my panic (see related ticket).

#5 Updated by Sean Fagan almost 5 years ago

How did you get it to do a dump after panic?

#6 Updated by Sean Fagan almost 5 years ago

  • Related to deleted (Bug #12120: After update to FreeNAS-9.3-STABLE-201510290351, NFS is not working)

#7 Updated by Sean Fagan almost 5 years ago

  • Blocks Bug #12120: After update to FreeNAS-9.3-STABLE-201510290351, NFS is not working added

#8 Updated by Sean Choquette almost 5 years ago

David Potterveld wrote:

After applying the October 28 update (FreeNAS-9.3-STABLE-201510290351) my system is unable to boot. It gets as far as starting the ntpd daemon, which then page faults while in kernel mode. I haven't been able to recover a crash dump to analyze yet, but will continue to attempt to do so. The info I have comes from the console monitor, which I had to video in slo-mo to see:

Fatal trap 12: page fault while in kernel mode
Fault code= supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ad18e6
stack pointer = 0x28:0xfffffe025109d230
frame pointer = 0x28:0xfffffe825189d290
...
current process = 5138 (ntpd)

Booting the prior version (FreeNAS-9.3-STABLE-201509282017) works normally. I know there has been a recent security update to ntp. I don't know if this has made it to FreeNAS yet, but I suspect it has and is causing problems (at least for me).

I Have the same problem on my freenas mini. Crash when I get to ntpd. go back to FreeNAS-9.3-STABLE-201509282017 everything is fine. tried it on 3 different usb's same result. my SuperMicro works fine with the latest update.

#9 Updated by David Potterveld almost 5 years ago

Sean Fagan wrote:

How did you get it to do a dump after panic?

It took me a while to figure that out, as I am not familiar with FreeBSD (I have much more experience with old Redhat systems). The standard config for FreeNAS will do a text dump on kernel panic automatically; the dump is written to the first swap device, and in theory, detected and extracted to /data/crash on reboot. For some reason, that last step wasn't working reliably on my system. I have a console display and keyboard, so on reboot after panic I could use the grub menu to select a working previous version, boot to single user mode, remount "/" as writeable, and use the savecore command to manually pull the dump out of the swap device and save to a file. That's when I found it was a text dump.

To get a binary minidump, I found I could still boot to single user mode in the crashing version. I remounted "/" writeable, and edited /etc/rc.conf. I changed "ix_textdump_enable" to "NO" and added a line: ddb_enable="NO".

Lastly, I used sysctl to set a few variables:
sysctl debug.ddb.textdump/pending=0
sysctl debug.debugger_on_panic=0

Then, exit out of single user mode, so the system continues the boot process. When it panics, a binary minidump will be written to a swap device, which can be recovered
manually as above.

The sysctl commands may make the rc.conf changes superfluous, but I don't know the order in which things are processed, so I did both, and it worked. You can also use sysctl to set up a full coredump, if you want, (sysctl debug.minidump=0).

#10 Updated by Sean Fagan almost 5 years ago

The only swap devices on my system are GELI-encrypted (default for freenas).

However: Jordan rolled a new update last night, with some minimal changes but a clean build. My system behaves normally with that update.

#11 Updated by David Potterveld almost 5 years ago

Sean Fagan wrote:

The only swap devices on my system are GELI-encrypted (default for freenas).

However: Jordan rolled a new update last night, with some minimal changes but a clean build. My system behaves normally with that update.

No such luck here :(
My system still crashes in the same way (ntpd, page fault while in kernel mode) with last night's update.

I have the default encrypted swap devices too, but the dump writing and reading seems to bypass that.

Also, I made a typo in my last post... the sysctl variable names are actually:
debug.ddb.textdump.pending
debug.debugger_on_panic

#12 Updated by David Potterveld almost 5 years ago

Additional information: If I disable the ntpd service, then the system boots normally. If I then log in and start ntpd via "service ntpd onestart" it crashes immediately as above.

I see the status is still "Waiting for feedback". What feedback from me do you need? If it's a binary coredump, I can provide that privately; I don't wish to post such to a public forum.

#13 Updated by Kim Culhan almost 5 years ago

  • File session_05_notcrash.log added
  • File FreeBSD 9.3-RELEASE-p29_102615.log added
  • File FreeBSD 9.3-RELEASE-p29_110115.log added
  • File FreeBSD 9.3-REL_P29_110115_take2.log added
  • File FreeBSD 9.3-RELEASE-p25_090215_isp.log added

I have been seeing what appears to be a similar situation, attached some console outputs.

crash: Filename FreeBSD 9.3-RELEASE-p29_102615.log is FreeBSD 9.3-RELEASE-p29 #0 r288272+c4ad350: Mon Oct 26 18:12:58 PDT 2015
crash: Filename FreeBSD 9.3-RELEASE-p29_110115.log has FreeBSD 9.3-RELEASE-p29 #0 r288272+dc0354b: Sun Nov 1 18:57:19 PST 2015
crash: Filename FreeBSD 9.3-REL_P29_110115_take2.log is from a different boot attempt, same freenas update as above, demonstrates very similar characteristics.

Late isp-rel crash: Filename FreeBSD 9.3-RELEASE-p25_090215_isp.log has FreeBSD 9.3-RELEASE-p25 #0 r281084+d3a5bf7: Wed Sep 2 15:00:10 PDT 2015 After boot and starting cron a fiber cable is attached to the HBA, this causes logging of
some lines with saX:

Folling this and not content with no-crash, a tape archiving application is started, resulting in fatal trap involving isp0.

Please let me know if you would like any particular tests, I'm setup to do this at any time.

thanks
-kim

#14 Updated by Kim Culhan almost 5 years ago

  • Seen in changed from 9.3.1-STABLE-201510290351 to 9.3.1-STABLE-201511020249
  • Hardware Configuration updated (diff)

RE. earlier posting about crashes on boot, I have also found that removing NTPD from starting up stops the crashes
at boot, which had been occurring when NTPD startup is printed to the console.

#15 Updated by Sean Fagan almost 5 years ago

Interesting. If you let it boot all the way, and then run ntpd, does it panic?

Before you try doing that, actually, do

ddb unscript kdb.enter.default

and if it panics, "bt" to get a stack backtrace would be helpful, I think.

#16 Updated by Kim Culhan almost 5 years ago

  • Seen in changed from 9.3.1-STABLE-201511020249 to 9.3-RELEASE

Started ntpd after boot finish and no crash.

Seen in:

FreeBSD 9.3-RELEASE-p29 #0 r288272+dc0354b: Sun Nov 1 18:57:19 PST 2015

/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.

thanks
-kim

#17 Updated by Jordan Hubbard almost 5 years ago

Another update has been released with the ntpd changes backed out. This should address this.

#18 Updated by David Potterveld almost 5 years ago

Jordan Hubbard wrote:

Another update has been released with the ntpd changes backed out. This should address this.

The new update boots successfully and runs properly for me. Thank you very much!!

#19 Updated by Jordan Hubbard almost 5 years ago

  • Status changed from 15 to Resolved

#20 Updated by Dru Lavigne almost 3 years ago

  • File deleted (textdump.tar.2)

#21 Updated by Dru Lavigne almost 3 years ago

  • File deleted (session_05_notcrash.log)

#22 Updated by Dru Lavigne almost 3 years ago

  • File deleted (FreeBSD 9.3-RELEASE-p29_102615.log)

#23 Updated by Dru Lavigne almost 3 years ago

  • File deleted (FreeBSD 9.3-RELEASE-p29_110115.log)

#24 Updated by Dru Lavigne almost 3 years ago

  • File deleted (FreeBSD 9.3-REL_P29_110115_take2.log)

#25 Updated by Dru Lavigne almost 3 years ago

  • File deleted (FreeBSD 9.3-RELEASE-p25_090215_isp.log)

#26 Updated by Dru Lavigne almost 3 years ago

  • Target version set to Master - FreeNAS Nightlies

Also available in: Atom PDF