Project

General

Profile

Bug #20927

Avatar?id=14398&size=22x22

[regression] After upgrade 9.10.1-U4 > 9.10.2 or 9.10.2-U1 stuck on "loading kernel modules"

Added by David Djerf over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Regression
Assignee:
Kris Moore
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

GIGABYTE GA-Z170N-WIFI (rev. 1.0);
DIMM DDR4 (2400) 16Gb ECC Kingston KVR24E17D8/16, CL17, 2Rx8, 1.2V, Retail;
Жесткий диск 4.0Tb Seagate NAS HDD ST4000VN000 SATA 6 Gb/s, 64 MB Cache, 5900 RPM -2 pieces;
CPU Intel Socket 1151 Pentium G4400 (3.30Ghz/3Mb) Box

Boot disk: Corsair Flash Voyager Slider X2 16GB USB 3.0
system dataset on mirrored Intel 320 SSD (120/160GB)

I also have an unknown model AMD Radeon graphics card that I had left over from work. Its a low power passive cooling "office" model. If its important to know the model I can try to get the name (it was not marked on the card).

ChangeLog Required:
No

Description

I run 9.10.1-U4 without any problems. Upgraded to latest and it hangs on "loading kernel modules".
Its a new server that I also have had the same problems with fresh installation of 9.10.2 and 9.10.2-U1, thats why I tried 9.10.1-U4.
I had the same problem in my virtual test computer (VMware Workstation), but after like 10 tries, i managed to upgrade from 9.10.1-U4 to 9.10.2-U1, so I thought the problems had been resolved.

2017-02-07 21.34.41.jpg (2.72 MB) 2017-02-07 21.34.41.jpg David Djerf, 02/07/2017 12:46 PM
2017-02-25 13.09.14.jpg (3.02 MB) 2017-02-25 13.09.14.jpg David Djerf, 02/25/2017 04:15 AM
2017-02-25 17.40.19.jpg (2.74 MB) 2017-02-25 17.40.19.jpg David Djerf, 02/25/2017 08:54 AM
2017-02-25 18.21.54.jpg (2.62 MB) 2017-02-25 18.21.54.jpg David Djerf, 02/25/2017 09:23 AM
8910
9300
9304
9307

Related issues

Related to FreeNAS - Bug #23604: FreeNAS 9.10.2-U3 hangs at boot "Loading kernel modules:"Resolved2017-04-27
Has duplicate FreeNAS - Bug #21124: 9.10.2 updateClosed: Duplicate2017-02-13
Has duplicate FreeNAS - Bug #21401: Loading kernel modulesClosed: Duplicate2017-02-23
Has duplicate FreeNAS - Bug #20656: Kaby Lake SupportClosed: Duplicate2017-01-29

History

#1 Updated by Sean Fagan over 4 years ago

  • Status changed from Unscreened to 15
  • Assignee set to Sean Fagan

What does it show if you type control-T when it is seemingly hung?

It'd be best if you did that a couple of times, waiting at least a few seconds between tries.

Thanks

#2 Updated by David Djerf over 4 years ago

Sean Fagan wrote:

What does it show if you type control-T when it is seemingly hung?

It'd be best if you did that a couple of times, waiting at least a few seconds between tries.

Thanks

It says:

load: 0.57 cmd: kldload 818 [acmtx] 3.41r 0.00u 0.00s 0% 1784k
load: 0.22 cmd: kldload 818 [acmtx] 61.77r 0.00u 0.00s 0% 1784k
load: 0.19 cmd: kldload 818 [acmtx] 68.65r 0.00u 0.00s 0% 1784k
load: 0.19 cmd: kldload 818 [acmtx] 71.89r 0.00u 0.00s 0% 1784k
load: 0.25 cmd: kldload 818 [acmtx] 75.33r 0.00u 0.00s 0% 1784k

By accident I pressed Ctrl+C and it continued to load... then it got stuck on "mdnsd" and I did the ctrl+c again.
Then I got the the console menu where I can see that it does not have network anymore.
Maybe it fails to load the kernel module for network? (Im a total noob at freebsd so Im just guessing)

#3 Updated by Sean Fagan over 4 years ago

  • Category changed from 1 to 137
  • Status changed from 15 to Unscreened
  • Assignee changed from Sean Fagan to Chris Torek

Over to Torek. ACPI stuff.

#4 Updated by Chris Torek over 4 years ago

  • Status changed from Unscreened to Investigation

For what it's worth, this is the third instance of ACPI hanging we've seen. Two are on these Gigabyte GA-X150M-Pro ECC boards, one is on an unrelated one.

We'll need to turn ACPI debugging on in the debug kernel, to track this down. It seems to be a generic FreeBSD ACPI issue that only comes up on a few BIOSes...

#5 Updated by David Djerf over 4 years ago

I have the same kind of hanging on my VMware Workstation test server.
Anything I can do to provide more information you need?

I have rolled back to 9.10.1-U4 for the moment.

#6 Updated by tom valenti over 4 years ago

i am having the same exact issue. didnt notice this until after i submitted a bug. refer to Issue #21124 for stats on my system.

#7 Updated by Sean Fagan over 4 years ago

#8 Updated by Bjorn Ostby over 4 years ago

Same issue for me with Supermicro X11SAE-M Bios Ver.2 and KabyLake i3-7300T processor.
Had to roll back to 9.10.1-U4.
Tried both to upgrade via GUI and to do a clean install of 9.10.2-U1.
Installation works ok and with first boot hangs forever at "loading kernel modules".

#9 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Status changed from Investigation to Closed: Duplicate

#10 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Status changed from Closed: Duplicate to Unscreened
  • Priority changed from No priority to Regression
  • Target version set to 9.10.3

#11 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Status changed from Unscreened to Investigation

Sorry, screwed up some of the states on this when marking duplicates ;)

#12 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Has duplicate Bug #21401: Loading kernel modules added

#13 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

Torek: Have you had any time to take a look at this? The duplicate tickets are starting to pile up.

#14 Updated by Chris Torek over 4 years ago

Unfortunately, no... I can say that now that we have the ACPI debugging built in to the debug kernel, we should be able to boot the debug kernel with various debugs turned on, as per https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/acpi-overview.html (section 11.13.4). I'm not sure which debug items we need, but probably start with everything and whittle down.

There is this slightly alarming note:

In practice, a serial console should be used to log the output so it is not lost as the console message buffer flushes.

that suggests that verbose debug of everything could be painfully slow and overwhelming.

#15 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Subject changed from After upgrade 9.10.1-U4 > 9.10.2 or 9.10.2-U1 stuck on "loading kernel modules" to [regression] After upgrade 9.10.1-U4 > 9.10.2 or 9.10.2-U1 stuck on "loading kernel modules"

David (Or anybody else with this issue) could you at the boot-loader screen, select your boot-environment with 9.10.2-U1 and try booting it in verbose mode and attach that screenshot when it hangs? Curious if any more information is revealed there. Please also attach your /boot/grub/grub.cfg so we can see if any other modules are being loaded.

Torek: FWIW, the only module we appear to be loading via GRUB is ispfw.ko. Aside from that I thought the kernels on the installer and live system were identical.

#16 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

Also - Our nightlies have switched to FreeBSD 11-STABLE now. Could you test updating to that and see if it keeps failing in the same spot?

#17 Updated by David Djerf over 4 years ago

OK, I'll make this my Saturday project and get back to you!

#18 Updated by David Djerf over 4 years ago

9300

After upgrade to 9.10.2-U1 and selected verbose boot I got stuck at:

Trying to mount root from zfs:freenas-boot/ROOT/9.10.2-U1 []...
start_init: trying /sbin/init
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)

Ill now try to use the latest nightly build.

#19 Updated by David Djerf over 4 years ago

I tried this nightly build: FreeNAS-10-MASTER-201702250123-573adeda2.iso
Is there a nightly for 9-series?

After upgrade the system boots up just fine, but I cannot login using any of my accounts anymore, web or SSH.

update:
I managed to set a password for the root account and got in the new web GUI.
No settings from my 9.10.1-U4 had been migrated, no storage pools or anything.
Maybe that is expected for the 10 nightly version.

I'm going back to 9.10.1-U4 again, but let me know if there are more things you need to fix this.

#20 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

You'll want to grab an ISO image from here (These are the 9.10 nightlies using FreeBSD 11)

http://download.freenas.org/9.10/MASTER/latest/x64/

#21 Updated by David Djerf over 4 years ago

Thanks! I tried to find 9.10 nightlies but failed.
Ill try this asap and get back to you.

#22 Updated by David Djerf over 4 years ago

9304

I have now tried to install FreeNAS-9.10-MASTER-201702241333-51472a1.iso as a new install (plan was to make new install first, and if successful, I go back and try an upgrade).
I get pass 1 of 5 100%, but there it hangs. The LED on my USB boot drive keeps flashing for at least 20 minutes, so it seems that something is going on.
I aborted the first attempt after 20 min. Ill let it run for a bit longer this time and see what happens.
But since all other installs I've made of freeNAS take 5-10 minutes, I would say something isn't working as expected.

#23 Updated by David Djerf over 4 years ago

9307

After ca 45 minutes, it said installation completed. I rebooted got stuck at "Loading kernel modules" again.

Btw, I tried a "normal" and a verbose boot, and it looks the same when it hangs.
I'm new to freeNAS, maybe I didn't do it right. I selected Verbose boot in grub, is there another way I should do it?

#24 Updated by Jose Lopez over 4 years ago

David Djerf wrote:

After ca 45 minutes, it said installation completed. I rebooted got stuck at "Loading kernel modules" again.

Btw, I tried a "normal" and a verbose boot, and it looks the same when it hangs.
I'm new to freeNAS, maybe I didn't do it right. I selected Verbose boot in grub, is there another way I should do it?

I'm having same issue with a new build.
My hardware:
Gigabyte GA-H270-HD3, Bios version F1
Intel i5-7600K
32 GB DDR4 Kingston memory
3x 2TiB Seagate HD
Booting from a Samsung 32 GB USB 3.0 drive

I've tried normal and verbose... After it hangs for a while on "Loading kernel modules", Ctrl+C gets it to jump, then gets stuck on "Starting mdnsd".... when it jumps, there are what is seemingly hundreds of "ACPI errors" scrolling by. This also happens when I "Ctrl-Alt-Del" to reboot.

Another note, just before hanging on "mdnsd", I see it starts em0, and gets an address, but afterwards, when the console setup screen is reached, there is the following message: "no configured interfaces were found. try to do it manually."

#25 Updated by Philip Hunt over 4 years ago

This issue occurs on 9.10.2, 9.10.2-U1 and is still occurring in 9.10.2-U2. All the same symptoms as described above... hangs at loading kernel modules, then again at starting mdnsd. If I ctrl-C past mdnsd then it gets to the console setup prompt but I'm unable to access the web interface. Networking appeared to be working though as both configured jails appeared to be running and serving their applications normally.

My system boots successfully and runs well on 9.10.1-U3 and 9.10.1-U4

My hardware:
ASRock H270M-ITX/ac
Pentium G4600
8 GB DDR4 Corsair memory
2x2TB Seagate IronWolf drives
1x250GB HGST Travelstar drive
Off-brand 32 GB USB 3.0 boot drive

#26 Updated by Samuel Schüppen over 4 years ago

Same issue here, all versions past 9.10.1 U4 show the above described faulty behavior (including FreeNAS 10 / Corral). It does not matter whether you update or do a clean install.

System:
Gigabyte GA-H110N
8GB DDR4 Crucial RAM (2133)
Intel Celeron G3930
USB 32GB Ultra Fit V2 U3 SDK
3TB WD Red

#27 Updated by tom valenti over 4 years ago

I have noticed every time I see the update all the hardware has 100/200 chipset combination. Starting to think it has to do with that. Maybe a fix coming soon?

#28 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Assignee changed from Chris Torek to Kris Moore

I'll take this one for now. I'm had to build a new NAS dev-box anyway, so I've grabbed that ASRock H270M-ITX/ac + a Kaby-Lake i3 to go with. I'll do some testing and hopefully find a fix here soon.

#29 Updated by Юрий Багинский over 4 years ago

  • Hardware Configuration updated (diff)

Good day!
I have the same problem after the update to 9.10.1-U4 > 9.10.2 - the server hangs on "loading kernel modules" what to do !!! waiting for problem solution

System:
GIGABYTE GA-Z170N-WIFI (rev. 1.0);
DIMM DDR4 (2400) 16Gb ECC Kingston KVR24E17D8/16, CL17, 2Rx8, 1.2V, Retail;
Жесткий диск 4.0Tb Seagate NAS HDD ST4000VN000 SATA 6 Gb/s, 64 MB Cache, 5900 RPM -2 pieces;
CPU Intel Socket 1151 Pentium G4400 (3.30Ghz/3Mb) Box
USB Flash drive SanDisk 16GB

#30 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

Ok, we've narrowed this down to an ACPI issue with FreeBSD 11/stable. It works in 12/current just fine. Looks like some support needs to be backported. Stay Tuned...

#31 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Assignee changed from Kris Moore to Chris Torek

Sending this back to Chris Torek. I've confirmed (With Mav's help) that its an ACPI issue which is fixed upstream in HEAD. We will need to MFC the latest ACPI code back to freenas/11-stable branch to get this fixed up.

#32 Updated by Юрий Багинский over 4 years ago

Kris Moore wrote:

Sending this back to Chris Torek. I've confirmed (With Mav's help) that its an ACPI issue which is fixed upstream in HEAD. We will need to MFC the latest ACPI code back to freenas/11-stable branch to get this fixed up.

Good day! Thanks waiting for the update with the solution of the bug "loading kernel modules"

#33 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Assignee changed from Chris Torek to Kris Moore

Taking this back. Spoke with jkim, they will MFC this to stable/11 here soon and we'll grab the fixes then. I'll keep an eye on it in the meantime.

#34 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Has duplicate Bug #20656: Kaby Lake Support added

#35 Updated by Юрий Багинский over 4 years ago

Kris Moore wrote:

Taking this back. Spoke with jkim, they will MFC this to stable/11 here soon and we'll grab the fixes then. I'll keep an eye on it in the meantime.

I Updated to version FreeNAS Corral/STABLE/10.0.2/ and the problem with "loading kernel modules" - disappeared !!!!!

#36 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Status changed from Investigation to Resolved

This is resolved in the 9.10 nightlies for 4/2 and later, and in upcoming 9.10.3.

https://github.com/freenas/os/commit/16f86484f4de9cf680d56e983d44216d7e152958

#38 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Target version changed from 9.10.3 to 11.0

#39 Updated by Bonnie Follweiler about 4 years ago

  • Related to Bug #23604: FreeNAS 9.10.2-U3 hangs at boot "Loading kernel modules:" added

#41 Updated by Vaibhav Chauhan about 4 years ago

  • Target version changed from 11.0 to 11.0-RC

Also available in: Atom PDF