Project

General

Profile

Bug #25237

Fix ichwd(4) shutdown crash

Added by Peter Oruba over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Important
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Intel Xeon E5-2603v4
Asus X99-M WS/SE Intel X99 So.2011-3
32 GB RAM ECC

ChangeLog Required:
No

Description

I am seeing a crash each time I issue a reset or shutdown of my machine. Already did several reinstallations.

Attaching stacktrace and verbose mode boot messages.

Associated revisions

Revision 2e4f4801 (diff)
Added by Alexander Motin over 1 year ago

Attach ichwd(4) only to ISA bus of the LPC bridge.

Resource allocation for parent device does not look good by itself, but
attempt to allocate them for unrelated device just does not end up good.
On Asus X99-E WS/USB3.1 system reporting ISA bridge via both PCI and ACPI
this reported to cause kernel panic on shutdown due to messed resources:
https://bugs.freenas.org/issues/25237.

(cherry picked from commit b63a1a3473c00d3ec6d6fd3693fd05c0b37a658e)

Ticket: #25237

Revision 2e4f4801 (diff)
Added by Alexander Motin over 1 year ago

Attach ichwd(4) only to ISA bus of the LPC bridge.

Resource allocation for parent device does not look good by itself, but
attempt to allocate them for unrelated device just does not end up good.
On Asus X99-E WS/USB3.1 system reporting ISA bridge via both PCI and ACPI
this reported to cause kernel panic on shutdown due to messed resources:
https://bugs.freenas.org/issues/25237.

(cherry picked from commit b63a1a3473c00d3ec6d6fd3693fd05c0b37a658e)

Ticket: #25237

History

#1 Updated by Dru Lavigne over 1 year ago

  • Assignee changed from Release Council to Alexander Motin

#2 Updated by Peter Oruba over 1 year ago

I counterchecked by installing FreeBSD-11.0-RELEASE-amd64 and Ubuntu Server 17.04 and both properly poweroff my machine.

FreeNAS 9.10 leads into a crash, too, when resetting or powering off, but instead of bringing me to the kernel debugger's console it gets stuck in a loop.

#3 Updated by Alexander Motin over 1 year ago

  • Status changed from Unscreened to Screened
  • Priority changed from No priority to Important
  • Target version set to 11.1

This looks like a possible bug in ichwd driver, may be because it calls ichwd_detach() for shutdown event, that, as I suspect, somehow calls it twice. It is surprising that this problem is not reported more widely.

But in your dmesg I also see that you are running debug kernel. Was it enabled intentionally? Does this crash happen for non-debug kernel also?

#4 Updated by Peter Oruba over 1 year ago

I selected the debug kernel to get more insight. The crash happens regardless of the kernel (debug / non-debug).

#5 Updated by Alexander Motin over 1 year ago

Could you attach full debug information archive? Or at very least `devinfo -vr`.

#6 Updated by Alexander Motin over 1 year ago

  • Status changed from Screened to 15

#7 Updated by Peter Oruba over 1 year ago

  • File debug-freenas-20170724103850.tgz added

Attaching full debug archive

#8 Updated by Michael de Silva over 1 year ago

Hi all -

Just to give you some context, this is my hardware config

  • Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  • 32GB Crucial RDIMMs ECC RAM (2x16GB)
  • Asus X99-E WS/USB3.1

I've setup a new FreeNAS 11 box and during shutdowns I'm also experiencing the panic issue. Output from `devinfo -vr` may be found here: https://gist.github.com/bsodmike/0a97281e359111e6c7ad2e2a47344e41

Here's a snap of the panic: https://photos.app.goo.gl/UpAhKn1METXTIPE52

Thanks.

#9 Updated by Alexander Motin over 1 year ago

  • Subject changed from Crash at shutdown / reset to ichwd(4) crash at shutdown / reset
  • Status changed from 15 to Fix In Progress

I see the problem now. Unlike all systems I have access to, your system reports ISA bridge in two ways same time: as PCI device, and as ACPI device. The second path is handled first by the OS device scan process, while the second quietly fails. It would be OK if not the terrible horror of ichwd(4) driver resource allocation. It more or less works in case of ISA bridge attached to PCI device, but doesn't in case of ACPI.

I've made a patch to make ichwd(4) driver attach only to ISA attached to PCI bridge, that supposedly should fix the crash.

As alternative workaround I guess setting such loader tunable may help: debug.acpi.disabled="isa". Make sure to create boot environment for recovery before adding this via UI in case I am wrong.

#10 Updated by Alexander Motin over 1 year ago

  • Status changed from Fix In Progress to 19

Committed the patch to nightly train.

#11 Updated by Peter Oruba over 1 year ago

Confirming setting loader tunable debug.acpi.disabled="isa" solves it. However, I wonder about the side-effects. Will verify the nightly when available.

#12 Updated by Peter Oruba over 1 year ago

I tried FreeNAS-11-MASTER-201707310506-1f71b47 and the issue still persists.

#13 Updated by Alexander Motin over 1 year ago

That is odd. I suspect the version you tried was built before my change. Please try some later one.

#14 Updated by Peter Oruba over 1 year ago

Tried FreeNAS-11-MASTER-201708010408-1f71b47 which is today's nightly containing your fix and confirming the problem is solved. Thanks!

#15 Updated by Alexander Motin over 1 year ago

  • Status changed from 19 to Ready For Release

#16 Updated by Dru Lavigne over 1 year ago

  • Subject changed from ichwd(4) crash at shutdown / reset to Fix ichwd(4) shutdown crash

#17 Updated by Dru Lavigne about 1 year ago

  • Target version changed from 11.1 to 11.1-BETA1

#18 Updated by Dru Lavigne about 1 year ago

  • Status changed from Ready For Release to Resolved

#19 Updated by Joe Maloney about 1 year ago

  • Needs QA changed from Yes to No
  • QA Status deleted (Not Tested)

#20 Updated by Dru Lavigne about 1 year ago

  • File deleted (debug-freenas-20170724103850.tgz)

#21 Updated by Dru Lavigne about 1 year ago

  • File deleted (verbosebootlog)

#22 Updated by Dru Lavigne about 1 year ago

  • File deleted (stacktrace)

Also available in: Atom PDF