Project

General

Profile

Bug #14336

linux kernel module not loaded even when installed plugins require it

Added by Eric Blau over 4 years ago. Updated about 4 years ago.

Status:
Closed: Cannot reproduce
Priority:
Nice to have
Assignee:
-
Category:
Middleware
Target version:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

I noticed some issues with the CrashPlan plugin in FreeNAS 9.10. I installed the CrashPlan plugin, but it wasn't working. I got "ELF unknown type" errors when trying to start it. These came out even when invoking "java -version" manually.

I tracked down the problem to a missing Linux kernel module in FreeNAS. kldstat shows:

[root@freenas] ~# kldstat
Id Refs Address Size Name
1 83 0xffffffff80200000 17dd000 kernel
2 1 0xffffffff81cae000 fdd44 ispfw.ko
3 1 0xffffffff82011000 f934 geom_mirror.ko
4 1 0xffffffff82021000 4694 geom_stripe.ko
5 1 0xffffffff82026000 ffb1 geom_raid3.ko
6 1 0xffffffff82036000 ec6a geom_raid5.ko
7 1 0xffffffff82045000 573b geom_gate.ko
8 1 0xffffffff8204b000 4a26 geom_multipath.ko
9 1 0xffffffff82050000 895 dtraceall.ko
10 10 0xffffffff82051000 3ad02 dtrace.ko
11 1 0xffffffff8208c000 463a dtmalloc.ko
12 1 0xffffffff82091000 224a dtnfscl.ko
13 1 0xffffffff82094000 63d7 fbt.ko
14 1 0xffffffff8209b000 580fb fasttrap.ko
15 1 0xffffffff820f4000 49cb lockstat.ko
16 1 0xffffffff820f9000 15ec sdt.ko
17 1 0xffffffff820fb000 d8d7 systrace.ko
18 1 0xffffffff82109000 d486 systrace_freebsd32.ko
19 1 0xffffffff82117000 4d94 profile.ko
20 1 0xffffffff8211c000 7fe3 ipmi.ko
21 1 0xffffffff82124000 1a62f hwpmc.ko
22 2 0xffffffff8213f000 2b32 vboxnetflt.ko
23 2 0xffffffff82142000 45370 vboxdrv.ko
24 1 0xffffffff82188000 41c2 ng_ether.ko
25 1 0xffffffff8218d000 3fd4 vboxnetadp.ko
26 1 0xffffffff82191000 9fc3 linprocfs.ko
27 2 0xffffffff8219b000 66e1 linux_common.ko

Running "kldload linux" loads the linux module and allows CrashPlan and other Linux binaries to work again. When I reboot FreeNAS, the linux module is not loaded again.

I believe a linux_load="YES" line is needed in /etc/rc.conf to make sure the module is loaded.


Related issues

Has duplicate FreeNAS - Bug #14364: Crashplan does not start after updating to 9.10-STABLEClosed: Duplicate2016-03-30

History

#1 Updated by Joshua Ruehlig over 4 years ago

  • Status changed from Unscreened to Screened
  • Assignee changed from Joshua Ruehlig to John Hixson

this actually isn't loaded for the plugin using the rc.conf setting. I believe the plugin system has an api call that can be used to load needed modules. forwarding to John H. as the plugin didn't change, but possibly the call isn't working properly in 9.10?

#2 Updated by Jordan Hubbard over 4 years ago

  • Assignee changed from John Hixson to Jordan Hubbard

Again, John doesn't work on FreeNAS anymore. I will take this one.

#3 Updated by Jordan Hubbard over 4 years ago

  • Has duplicate Bug #14364: Crashplan does not start after updating to 9.10-STABLE added

#4 Updated by john aylward over 4 years ago

In my "System -> Tunables" I do see both linux_enable and linux_load set to YES. I can't remember if I added them myself before the 9.10 update, or if they were added by the plugin.

#5 Updated by Jordan Hubbard over 4 years ago

  • Status changed from Screened to 15

They can't have been added by the plugin since plugins don't have access to the tunable space. I suspect we just always discussed these tunables being necessary for crashplan, since I see nothing in 9.3 that ever added them by default. What I'm not clear on is why the tunables were not migrated during the upgrade - was this an upgrade or a fresh install?

#6 Updated by Eric Blau over 4 years ago

Jordan Hubbard wrote:

They can't have been added by the plugin since plugins don't have access to the tunable space. I suspect we just always discussed these tunables being necessary for crashplan, since I see nothing in 9.3 that ever added them by default. What I'm not clear on is why the tunables were not migrated during the upgrade - was this an upgrade or a fresh install?

I upgraded from 9.3 to 9.10. Before the upgrade CrashPlan was working and afterwards it was not due to this issue. I did not, either before or after the upgrade, have a tunable (at least visible through the WebUI) that showed linux_enable or linux_load set to YES.

#7 Updated by john aylward over 4 years ago

I also performed the update from 9.3 -> 9.10

Any tunables in there would have been added when I was still 9.3. However, I see the same issue mentioned in the description, so I'm guessing the tunables aren't working either.

#8 Updated by john aylward over 4 years ago

hmm, both my tunables were of type "loader" for some reason, let me try changing that to "rc.conf" and see if that works on reboot

#9 Updated by Jordan Hubbard over 4 years ago

  • Status changed from 15 to Screened
  • Priority changed from No priority to Nice to have
  • Target version set to 261

BRB: We're still not entirely sure what the behavioral change between FreeBSD 9.3 and 10.3 was to cause the linux module to no longer be loaded by default, but obviously the "right thing to do" is load it unconditionally in 9.10. For a future SU.

#10 Updated by Jordan Hubbard over 4 years ago

  • Status changed from Screened to Ready For Release

Added COMPAT_LINUX options to the default kernel config; will be in next 9.10-Nightly build.

#11 Updated by Jordan Hubbard over 4 years ago

  • Status changed from Ready For Release to Screened

False alarm, sorry, COMPAT_LINUX didn't work. Trying other things...

#12 Updated by Jordan Hubbard over 4 years ago

  • Assignee changed from Jordan Hubbard to Anonymous

This is a good low-hanging fruit ticket to start with: Kaustubh - please look at the jail code in 9.3, which apparently used to load the linux emulator by default when the crashplan plugin was loaded but doesn't do it now in 9.10.

#13 Updated by Jordan Hubbard over 4 years ago

  • Status changed from Screened to Unscreened

#14 Updated by Anonymous over 4 years ago

  • Status changed from Unscreened to Investigation

Tried reproducing the issue...

Installed crashplan plugin.
It did not start. It failed with error "some error occurred".
Tried to figure out the reason. The HTTP response shows error: "Error: hostname nor servname provided, or not known"
Running 'kldload linux' didn't work in this case either.

Trying again to reproduce.

#15 Updated by Anonymous over 4 years ago

Do I need to follow this to make crashplan work for the first time?

https://github.com/sirkkalap/freenas-crashplan-howto

#16 Updated by john aylward over 4 years ago

@Kaustubh you may need to edit /usr/pbi/crashplan-amd64/share/crashplan/bin/run.conf to look similar to this:

SRV_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx3096m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider" 
GUI_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanDesktop -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider"

#17 Updated by john aylward over 4 years ago

Actually, looking through some links on the github that kaustubh provided, I found this:

https://forums.freenas.org/index.php?threads/crashplan-not-updating.40374/

it looks like the plugin itself may be due for an update to a newer version of crashplan since the auto-updates have stopped working.

#18 Updated by Eric Blau over 4 years ago

Kaustubh Dhokte wrote:

Do I need to follow this to make crashplan work for the first time?

https://github.com/sirkkalap/freenas-crashplan-howto

I followed the instructions here to update CrashPlan from 3.6.3 to 4.5.0:

https://forums.freenas.org/index.php?threads/crashplan-not-updating.40374/#post-254182

Then when starting CrashPlan for the first time, it auto-updates to 4.6.0. 3.6.3 is an ancient version. The plugin would ideally be upgraded to 4.6.0.

I also had to add "export LD_LIBRARY_PATH=/usr/pbi/crashplan-amd64/share/crashplan/linux-sun-jre1.7.0/lib/i386/jli" to /usr/pbi/crashplan-amd64/share/crashplan/bin/CrashPlanEngine to get it to start properly.

#19 Updated by Jordan Hubbard over 4 years ago

We’re not totally sure what is causing the problem with this bug - various people have had their own theories, and my own testing has basically just involved going into the plugin (with jexec <jailid> sh) and then running one of the plugin binaries manually (/usr/pbi/crashplan-amd64/.sbin/CrashPlanDesktop is what I’ve been using)

If it falls over with an emulation error (ELF binary type "0" not known) then I know it’s not loading the linux compatibility module

William claims that the jail code used to load the linux kernel module automatically by looking in the jail, but we don’t know how this code worked before in 9.3 - the bug is basically to go find out how it works in 9.3

#20 Updated by Luc Stroobant over 4 years ago

I can confirm that after updating to freenas 9.10 and applying the config change of #18 my Crashplan starts again. (well, I used export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/pbi/crashplan-amd64/share/crashplan/jre/lib/i386/jli ). It also works with the change of #16, but you don't need that. It already starts with my original run.conf:

SRV_JAVA_OPTS="-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx1024m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false" 
GUI_JAVA_OPTS="-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider -Dfile.encoding=UTF-8 -Dapp=CrashPlanDesktop -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false" 

#21 Updated by Kim Kam over 4 years ago

It seems that linux modules (kldload linux) are loaded from folowing script:
/usr/local/share/warden/scripts/backend/startjail.sh

To check if jail is linux it also uses:
/usr/local/share/warden/scripts/backend/functions.sh

#22 Updated by Kim Kam over 4 years ago

It is this code in functions.sh which checks if jail is linux:

is_linux_jail() {
file "${JAILDIR}/sbin/sysctl" | \
cut -d ',' -f 5 | \
awk '{ print $2 }' | \
cut -f2 -d'/' | \
grep -iq Linux >/dev/null 2>&1
if [ "$?" = "0" ] ; then
return 0
fi

return 1
}

I already don't have 9.3, can somebody put here outputs for some jail from:

  1. file /mnt/pool-name/jails/jail-name/sbin/sysctl

from both freenas 9.3 and 9.10 to compare if output did not changed

#23 Updated by Kim Kam over 4 years ago

as I wrote checking for Linux jail seems to be wrong here in functions.sh
on more places is check for sbin/sysctl using file command with:
cut -d ',' -f 5 | \
which probably should be
cut -d ',' -f 6 | \
it is on more places in warden scripts.
so output of "file" command changed between freebsd 9 and 10?

#24 Updated by Anonymous over 4 years ago

Hey folks, I was trying to reproduce this bug for quite a long time but I could see some different behaviours on different systems.
I finally figured out the the way to install crashplan plugin and get it working and now though I see same behaviour, I am little confused about where exactly this issue is being produced. I followed below steps trying to reproduce this one but cant really locate where exactly it should fail.

1. Booted a fresh freenas-9.10-STABLE-201603252134 system.
2. Installed the crashplan plugin. I did it by uploading the pre-downloaded crashplan pbi http://download.freenas.org/plugins/9/x64/crashplan-3.6.3_1-amd64.pbi as I observed that installing it by clicking on install takes way too much time than uploading the predownloaded pbi and installing, as it anyways downloads the same pbi everytime to best of my understanding. (I hope this done not cause the difference)
3. Accept the TOS.

At this point I did kldstat on freenas shell and I could see linux.ko and linux-common.ko modules loaded.

4. Enabled the plugin by clicking on ON button on the UI. It worked.
5. I went to jail console and executed /usr/pbi/crashplan-amd64/.sbin/CrashPlanDesktop which worked. (It did not fail with the error 'ELF binary type "0" not known' or any other error)
6. I rebooted freenas system
7. Plugin was in ON state by default after rebooting.
7. Again ran kldstat on freenas shell and could see linux.ko and linux-common.ko modules loaded.
8. I repeated step 5 with same results.
9. Updated the crashplan to 4.6.0 by following https://forums.freenas.org/index.php?threads/crashplan-not-updating.40374/#post-254182 (Thanks to Eric #18) as I found that my desktop crashplan version and crashplan version in the jail(on a headless computer) should be same.
https://support.code42.com/CrashPlan/4/Configuring/Using_CrashPlan_On_A_Headless_Computer
10. Somehow I was not being able to connect to freenas in jail from my desktop.
I followed this guide for the same - https://github.com/sirkkalap/freenas-crashplan-howto
I suspect I was doing some misconfiguration of IP and Ports (port forwarding) here to get it going. or else it is this bug because of which I was not being able to connect to the plugin inside the jail.

I did not need to update /usr/pbi/crashplan-amd64/share/crashplan/bin/run.conf as mentioned by John in #16 as the original file had almost no difference.

As many of you are facing this issue I am little confused exactly at which point it is being produced. As I am new to freenas and freenas development a litte help here would be appreciated as it will help me to better understand and reproduce this bug and apply the fix.

#25 Updated by Eric Blau over 4 years ago

Kim Kam wrote:

as I wrote checking for Linux jail seems to be wrong here in functions.sh
on more places is check for sbin/sysctl using file command with:
cut -d ',' -f 5 | \
which probably should be
cut -d ',' -f 6 | \
it is on more places in warden scripts.
so output of "file" command changed between freebsd 9 and 10?

I think you're on the right track here. The Linux modules are loaded if is_linux_jail() returns true / non-zero. The code does the following:

is_linux_jail()
{
   file "${JAILDIR}/sbin/sysctl" | \
      cut -d ',' -f 5 | \
      awk '{ print $2 }' | \
      cut -f2 -d'/' | \
      grep -iq Linux >/dev/null 2>&1
   if [ "$?" = "0" ] ; then
      return 0
   fi

   return 1
}

So it is checking the file type of /sbin/sysctl inside the jail and looking for "Linux" in the interpreter portion file command. When I run "file /sbin/sysctl" inside the FreeNAS 9.10 CrashPlan jail, I get:

root@crashplan_1:/ # file /sbin/sysctl
/sbin/sysctl: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.3, stripped

There's no indication that the binary is a Linux binary at all.

If I run file on the java binary that is part of the CrashPlan plugin, I get output showing the interpreter is the Linux interpreter:

root@crashplan_1:/usr/pbi/crashplan-amd64/share/crashplan/linux-sun-jre1.7.0/bin # file java
java: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.9, BuildID[sha1]=3f324968e275755244b6a10609e81796be828e15, not stripped

Something must have changed between FreeNAS 9.3 and 9.10 where the /sbin/sysctl binary within jails is no longer a Linux binary and is an ELF FreeBSD binary instead, at least according to the file command.

Can the is_linux_jail() function check another binary in the jail to see whether it is a Linux jail or not? Is there a more appropriate binary to check? Does anyone have a FreeNAS 9.3 system with the CrashPlan plugin where they can capture the file output frm the /sbin/sysctl binary in the jail?

#26 Updated by john aylward over 4 years ago

Checking the output of binaries seems very fragile to me. Would it be reasonable adjust the linux plugin jails have a standard file in ${JAILDIR}/usr/local/etc that could be checked? Then the linux check isn't some convoluted awk/cut mess and instead is a simple file-exists check.

I'm not sure how the warden configs work, but would having the jail set a warden variable upon install be a viable option as well?

Last suggestion is that obviously the install process is setting some kind of values into a DB so it knows which plugins are installed and what version. Maybe setting a flag in that same table that could be checked at system start could be used to indicate if the jail needs any special kernel modules.

#27 Updated by Jordan Hubbard over 4 years ago

I tend to agree that a sentinel file inside a plugin (which knows whether it's "linux" or not) would be a much more elegant solution, particularly given that there is one linux plugin at present. Then the crashplan plugin could be dissected, the sentinel added, and the version number bumped so folks could upgrade to the "newest" crashplan with the sentinel file in place. Perhaps crashplan itself could also be updated at the same time, since I guess there are newer versions of it out there.

#28 Updated by Anonymous over 4 years ago

  • Status changed from Investigation to Closed: Cannot reproduce

I tried reproducing the issue number of times. But every time I install the crashplan plugin, linux module is loaded and stays loaded even after reboot.

#29 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Target version changed from 261 to N/A

Also available in: Atom PDF