Project

General

Profile

Bug #40872

Add ability to allocate TUN devices dynamically in iocage

Added by Marc-Olivier Leblanc over 1 year ago. Updated about 1 year ago.

Status:
Done
Priority:
No priority
Assignee:
Brandon Schneider
Category:
Middleware
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Since updating from 11.2-Beta1 to 11.2-Beta2 my jail with OpenVPN can't create its TUN interface automatically.

The "devfs rule -s 4 add path 'tun*' unhide" pre-init command doesn't help in Beta2 but it worked fine in Beta1.

Nuking the jail and re-creating it doesn't fix the issue.


Related issues

Related to FreeNAS - Bug #27548: Can't dynamically create tun* devices in iocage jailsClosed: Behaves correctly
Related to FreeNAS - Feature #51083: Add plugin for OpenVPN serverDone
Related to FreeNAS - Bug #53334: Correctly add bpf to devfs rulesDone
Related to FreeNAS - Bug #56616: Enable allow_tun for OpenVPN pluginDone
Related to FreeNAS - Bug #73101: Add ability to allocate TAP devices dynamically in iocageClosed
Has duplicate FreeNAS - Bug #45919: OpenVPN cannot allocate tun in jailClosed
Copied to FreeNAS - Bug #56340: Add ability to allocate iocage TUN devices in new UIDone

History

#2 Updated by Dru Lavigne over 1 year ago

  • Status changed from Unscreened to Closed
  • Target version changed from Backlog to N/A
  • Reason for Closing set to Duplicate Issue

#3 Updated by Chris Bailes over 1 year ago

+1 for this.

This is not a straight duplicate of Bug #27548 as the proposed solution there does not work. Please consider reopening this issue.

All best,
Chris.

#4 Updated by Dru Lavigne over 1 year ago

  • Related to Bug #27548: Can't dynamically create tun* devices in iocage jails added

#5 Updated by Dru Lavigne over 1 year ago

Chris: we'll need a debug in order to reopen this ticket and investigate (System -> Advanced -> Save debug created after reproducing the issue).

#6 Updated by Matthew T over 1 year ago

  • File debug (1).tgz added

I have the same issue, tried using jail versions 11.0, 11.1, 11.2. All do the same, and when using the command from the above mentioned Bug #27548 it just creates tun0-tun255 and then openvpn stops. I attached my debug zip as asked, hopefully you find something.

#7 Updated by Dru Lavigne over 1 year ago

  • Category changed from OS to Middleware
  • Status changed from Closed to Unscreened
  • Assignee changed from Release Council to William Grzybowski
  • Target version deleted (N/A)
  • Private changed from No to Yes
  • Reason for Closing deleted (Duplicate Issue)

#8 Updated by William Grzybowski over 1 year ago

  • Assignee changed from William Grzybowski to Brandon Schneider
  • Target version set to 11.2-RC1

#9 Updated by Brandon Schneider over 1 year ago

  • Status changed from Unscreened to Not Started

#10 Updated by William Grzybowski over 1 year ago

  • Has duplicate Bug #45919: OpenVPN cannot allocate tun in jail added

#11 Updated by Brandon Schneider over 1 year ago

  • Status changed from Not Started to In Progress

#12 Updated by Brandon Schneider over 1 year ago

#13 Updated by Brandon Schneider over 1 year ago

  • Status changed from In Progress to Ready for Testing
  • Needs Merging changed from Yes to No

#14 Updated by Dru Lavigne over 1 year ago

  • File deleted (debug (1).tgz)

#15 Updated by Dru Lavigne over 1 year ago

  • Subject changed from 11.2B2 - Cannot allocate TUN/TAP dev dynamically in IOCAGE jail to Add ability to allocate TUN/TAP devices dynamically in iocage
  • Private changed from Yes to No
  • Needs Doc changed from Yes to No

#17 Updated by Dru Lavigne over 1 year ago

#18 Updated by Dru Lavigne over 1 year ago

  • Status changed from Ready for Testing to Done
  • Needs QA changed from Yes to No

#20 Updated by Bonnie Follweiler over 1 year ago

  • Status changed from Done to Passed Testing

#21 Updated by Dru Lavigne over 1 year ago

  • Status changed from Passed Testing to Done

#22 Updated by Steve Levey over 1 year ago

Sorry, I'm not sure how to use this fix. Could you explain how this fixes Bug #27548?

Thanks

#23 Updated by Dru Lavigne over 1 year ago

Steve: if updating to RC1 does not resolve your issue, please create a new ticket that contains your debug.

#24 Updated by Brandon Schneider over 1 year ago

Steve: When you create a jail in RC1, add allow_tun=1 to the command.

#25 Updated by Steve Levey over 1 year ago

OK.. I think I figured out this option is not available in the GUI. :)

I'll give it a try at the command line.

Should this option be in the GUI?

#26 Updated by Brandon Schneider over 1 year ago

Come RC1 it will be in the GUI yes. If it’s not, please file a bug. It absolutely should :)

#27 Updated by Steve Levey over 1 year ago

ok.. i don't see it in RC1.. so I'll file a bug.

Also, create iocage with allow_tun="1". Still getting "Cannot allocate TUN/TAP dev dynamically", so I'll file a new bug for that as well, although I'm guessing I'm just missing something obvious.

We still need the devfs rule -s 4 add path 'tun*' unhide pre-init script, right?

#28 Updated by Brandon Schneider over 1 year ago

No this property does that for you. It shouldn’t have any more issues with creating tun devices in those jails. Try to manually create one to make sure.

#29 Updated by Dan Jacques over 1 year ago

  • Seen in changed from 11.2-BETA2 to 11.2-RC1

I just upgraded to 11.2-RC1 and believe that I'm experiencing this issue: FreeBSD 11.2-STABLE (FreeNAS.amd64) #0 r325575+97f4f541349(freenas/11.2-stable)

The PR to iocage seems to be included. I added allow_tun=1 to my jail's config, and the devfs.conf rule that it generated seems to be present:

(snip /etc/devfs.rules)

## IOCAGE -- vpn ruleset
[vpn_ruleset=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'zfs' unhide
add path 'tun*' unhide

(snip)

However, openvpn is still failing with the same error message.

Prior, I was running openvpn in an 11.1-5 iocage jail using the devfs preinit hack that Marc-Oliver noted; however, that hack doesn't seem to be working anymore, as noted in this bug. I've rebooted several times and enabled/disabled the previous preinits.

Happy to help diagnose the issue and get to the bottom of this, but ATM I think this may still be broken.

#30 Updated by Steve Levey over 1 year ago

I concur.. still having issues getting this to work.

#31 Updated by Brandon Schneider over 1 year ago

  • Status changed from Done to In Progress

I’ll look again, but the patch allows me to create tuns in the jail. I didn’t try openvpn specifically.

We’ll hopefully figure this out :)

#32 Updated by William Grzybowski over 1 year ago

  • Target version changed from 11.2-RC1 to 11.2-RC2

#33 Updated by Mark R over 1 year ago

Can confirm that allow_tun does not fix it for me either.

After setting allow_tun on a existing jail via cli and confirming it and then rebooting the entire machine.
The jail came up with tun0 to tun255 passed in. But OpenVPN still cannot open tun0 with the same Cannot allocate TUN/TAP dev dynamically message.

#34 Updated by Dan Jacques over 1 year ago

Brandon Schneider wrote:

I’ll look again, but the patch allows me to create tuns in the jail. I didn’t try openvpn specifically.

For what it's worth, my experience was that creating TUN devices claimed success, but didn't really work. Here's a rundown. This is in an iocage jail with VNET enabled and allow_tun=1:

$ ls /dev/tun*
ls: /dev/tun*: No such file or directory

$ ifconfig tun create
tun256

$ ls /dev/tun*
ls: /dev/tun*: No such file or directory

$ ifconfig tun256
tun256: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        nd6 options=1<PERFORMNUD>
        groups: tun

So in the jail, the interface is available, but the device is not. Exiting the jail and looking at the outer NAS, I see the devices!

$ ls /dev/tun*
... tons of tuns, including ...
/dev/tun256

I have, sadly, replaced my "working" (with hack) 11.1 version, so I can't easily test if this behavior is consistent, but it seems weird to me that the device wouldn't appear within the jail.

In the openvpn source code, it appears that the device is opened after creation using the actual file path: https://github.com/OpenVPN/openvpn/blob/43a5a4f3b4e411419639c195fee8a76495fdc88e/src/openvpn/tun.c#L1604

That code also includes the dreaded "Cannot allocate TUN/TAP dev dynamically" error messages when the file path fails to open.

AFAICT, what's happening in openvpn is:

  • tun.c is looping from 0 to 255 trying to create a TUN device (this probably explains why there are 256 devices!)
  • The /dev/tun... device that was created within the jail isn't showing up in the jail.
  • tun.c is erroring b/c it can't see/open the /dev/tun... file.

This also explains why only 256 exist, since it explicitly tries 0..255, and not the next 255 after the last observed tun. Weird, but shrug I'm sure someone had a reason.

We’ll hopefully figure this out :)

AFAICT the solution to this may be to figure out why the /dev/tun# devices aren't showing up in the jail that created them. If that happened, I think openvpn would be mollified. If this is correct, you could confirm that the patch is working based on the visibility of the /dev/tun# device within the jail that created it.

(Or I'm 100% off base, this isn't something I'm very familiar with.)

#35 Updated by Dru Lavigne over 1 year ago

  • Status changed from In Progress to Not Started

#36 Updated by Todd Martin over 1 year ago

Moving away from dynamically configured tun devices. I tested with static tun device to help find where things are breaking down.

Using "ifconfig tun create" will create tun devices tun0~tun255. Replacing with a specific tun device i.e."ifconfig tun0 create" only creates the one device requested.

Using "ifconfig tun0 create" from inside a jail creates /dev/tun0 on hostOS side but does not map the /dev/tun0 to the jail. so the file 'exists' on the hostOS side, but since it is unmapped to the jail, nothing from inside the jail (openvpn) can connect to it.
This leads me to believe the issue lies in how devfs is being mapped to the jails. I have "allow_tun=1" defined on the jail I am testing with, which properly sets up devfs.rules for the jail to unhide tun devices, yet they are still hidden.

#37 Updated by Alex Lantagne over 1 year ago

I have the same problem and have tried everything said above. Running 11.2-RC1. Used the devfs ruleset 4 unhide tun hack in 11.1-U5 and was working fine.

I am seeing more and more people reporting problems in different forums/blog with tun not created dynamically when using openvpn in a jail since 11.1-UX.

Does anybody know a workaround or have at least a clue what to look at?

I also believe it has something to do with devfs mounted to the jail.

#38 Updated by Brandon Schneider over 1 year ago

  • Status changed from Not Started to In Progress
  • Target version changed from 11.2-RC2 to 11.2-U2

Still in progress, it got errantly set to Not Started.

Currently, it's seeming to be an upstream bug. I can reproduce this in FreeBSD 11.2-R, trying other versions to narrow down things.

#39 Updated by Brandon Schneider over 1 year ago

  • Needs Doc changed from No to Yes
  • Needs Merging changed from No to Yes

Alright, turns out it was just me. That's a relief!

master PR: https://github.com/freenas/iocage/pull/73
11.2-stable PR:

#40 Updated by Brandon Schneider over 1 year ago

  • Related to Bug #53334: Correctly add bpf to devfs rules added

#41 Updated by Dru Lavigne over 1 year ago

  • Target version changed from 11.2-U2 to 11.2-RC2

#42 Updated by Dan Jacques over 1 year ago

Brandon Schneider wrote:

Alright, turns out it was just me. That's a relief!

master PR: https://github.com/freenas/iocage/pull/73
11.2-stable PR:

I started my jails with an `iocage` including this PR and can confirm that the `tun` devices show up within the jail, and that `openvpn` seems to work! I'll test again after a fresh FreeNAS restart later today.

Thanks for the quick and comprehensive fix!

#43 Updated by Peter Smith over 1 year ago

Dan Jacques wrote:

Brandon Schneider wrote:

Alright, turns out it was just me. That's a relief!

master PR: https://github.com/freenas/iocage/pull/73
11.2-stable PR:

I started my jails with an `iocage` including this PR and can confirm that the `tun` devices show up within the jail, and that `openvpn` seems to work! I'll test again after a fresh FreeNAS restart later today.

Thanks for the quick and comprehensive fix!

Sorry can someone explain what I need to do to fix the openvpn issue?

#44 Updated by Dan Jacques over 1 year ago

Sorry can someone explain what I need to do to fix the openvpn issue?

The best answer is to wait for an 11.2-stable release with Brandon's patch in it. Since the patch is still a pull request, and hasn't landed in upstream iocage, you would need to wait for it to land, then wait for a FreeNAS 11.2 release with the latest iocage.
  • This would probably be the next RC release.
  • If you are open to more risk, you can install from the FreeNAS nightly build.
  • NOTE: The iocage pull request hasn't even landed yet, so no official FreeNAS build includes it at the moment.
Regardless, you will need to:
  1. Remove any existing openvpn support hacks, such as the devfs pre-init command.
  2. Stop all of the jails that are using OpenVPN
    iocage stop <jail-name>
    
  3. Update your openvpn jail(s) to include the allow_tun=1 setting. FreeNAS should support this via UI eventually, but as of now I'm unaware of a way to do this with UI. This can be done by running:
    iocage set allow_tun=1 <jail-name>
    
  4. Reboot your NAS, to clear any resident state from previous hack(s).
  5. Start the jail again.
    iocage start <jail-name>
    

Now, if you waited for a release that includes the fix, the jail should "just work" from here on out without any additional effort!

If you are impatient or desperate, you can do what I did and manually install/use the upstream iocage with the fix.
  • These instructions and upstream iocage are works in progress, and may be buggy, causing harm to your NAS or jails.
  • I am not affiliated with FreeeNAS or this patch, so this is just user-to-user advice.
  • You can no longer use the UI to manage affected jails util a release with the fix rolls out, but then you should be OK.
If this is acceptable to you, here is what I did. For each OpenVPN jail:
  1. Disable autostart on the jail (since you will need to manually use the upstream iocage to start them). You can use UI or CLI for this.
    iocage set boot=off <jail-name>
    
  2. Do all of the steps outlines above, except the final start command.

Now, you can check out iocage using git:

# Get a "root" shell, if you're not already root.
$ sudo -s

# Checkout "iocage" into "/usr/local/var/lib/iocage_with_pr73" 
$ git clone https://github.com/freenas/iocage.git /usr/local/var/lib/iocage_with_pr73

# Checkout the latest version of Brandon's PR.
# NOTE: You can run this sequence each time the PR is updated to refresh it.
# NOTE2: Once the PR has landed into "master", you can skip this step and just update "master" using "git pull".
$ git -C /usr/local/var/lib/iocage_with_pr73 fetch origin pull/73/head
$ git -C /usr/local/var/lib/iocage_with_pr73 checkout FETCH_HEAD

Now that you have an iocage version with the fix present, you can mange your jails using that version, and it should properly handle things.

# Start your jail.
$ /usr/local/var/lib/iocage_with_pr73/iocage start <jail-name>

# Stop your jail (when you want to).
$ /usr/local/var/lib/iocage_with_pr73/iocage stop <jail-name>

You can confirm that the new jail is working by looking at the devfs_ruleset for the jail once it's running.

# Get your jail's ID by listing jails and identifying its <id>.
$ jls
...
<id> <addrs> <name> <root>
...

# Find its associated "devfs" ruleset ID.
$ jls -j <id> devfs_ruleset

# Examine its "devfs" ruleset. You are looking for "path tun* unhide".
$ devfs rule -s <id> show
...
<rule-id> path tun* unhide
...

A word of caution: you must use whatever version of iocage that you used to start your jail on that jail. Mixing iocage versions on the same jail instance could lead to an unmanaged or unpredictable system state. In this case, if you started with /usr/local/var/lib/iocage_with_pr73/iocage, you must use that command exclusively (not FreeNAS's built-in /usr/local/bin/iocage) to manage your jail until you stop it or reboot.

If you have any questions about these instructions or want some help, feel free to send me a private message. Hope this helps!

#45 Updated by Peter Smith over 1 year ago

Thanks Very much for the comprehensive response.

#46 Updated by Alex Lantagne over 1 year ago

Marc-Olivier Leblanc wrote:

Since updating from 11.2-Beta1 to 11.2-Beta2 my jail with OpenVPN can't create its TUN interface automatically.

The "devfs rule -s 4 add path 'tun*' unhide" pre-init command doesn't help in Beta2 but it worked fine in Beta1.

Nuking the jail and re-creating it doesn't fix the issue.

Dan Jacques wrote:

Sorry can someone explain what I need to do to fix the openvpn issue?

The best answer is to wait for an 11.2-stable release with Brandon's patch in it. Since the patch is still a pull request, and hasn't landed in upstream iocage, you would need to wait for it to land, then wait for a FreeNAS 11.2 release with the latest iocage.
  • This would probably be the next RC release.
  • If you are open to more risk, you can install from the FreeNAS nightly build.
  • NOTE: The iocage pull request hasn't even landed yet, so no official FreeNAS build includes it at the moment.
Regardless, you will need to:
  1. Remove any existing openvpn support hacks, such as the devfs pre-init command.
  2. Stop all of the jails that are using OpenVPN
    [...]
  3. Update your openvpn jail(s) to include the allow_tun=1 setting. FreeNAS should support this via UI eventually, but as of now I'm unaware of a way to do this with UI. This can be done by running:
    [...]
  4. Reboot your NAS, to clear any resident state from previous hack(s).
  5. Start the jail again.
    [...]

Now, if you waited for a release that includes the fix, the jail should "just work" from here on out without any additional effort!

If you are impatient or desperate, you can do what I did and manually install/use the upstream iocage with the fix.
  • These instructions and upstream iocage are works in progress, and may be buggy, causing harm to your NAS or jails.
  • I am not affiliated with FreeeNAS or this patch, so this is just user-to-user advice.
  • You can no longer use the UI to manage affected jails util a release with the fix rolls out, but then you should be OK.
If this is acceptable to you, here is what I did. For each OpenVPN jail:
  1. Disable autostart on the jail (since you will need to manually use the upstream iocage to start them). You can use UI or CLI for this.
    [...]
  2. Do all of the steps outlines above, except the final start command.

Now, you can check out iocage using git:
[...]

Now that you have an iocage version with the fix present, you can mange your jails using that version, and it should properly handle things.

[...]

You can confirm that the new jail is working by looking at the devfs_ruleset for the jail once it's running.

[...]

A word of caution: you must use whatever version of iocage that you used to start your jail on that jail. Mixing iocage versions on the same jail instance could lead to an unmanaged or unpredictable system state. In this case, if you started with /usr/local/var/lib/iocage_with_pr73/iocage, you must use that command exclusively (not FreeNAS's built-in /usr/local/bin/iocage) to manage your jail until you stop it or reboot.

If you have any questions about these instructions or want some help, feel free to send me a private message. Hope this helps!

Wonderful. Worked right away. I believe we can consider this issue fixed, can't we?

#47 Updated by Spencer Payne over 1 year ago

I can't seem to get this fix to work. I followed the instructions to the letter, however, when I run /usr/local/var/lib/iocage_with_pr73 start <jail-name> I get a "permission denied" message. Running it with sudo returns a "command not found" message. Any guidance would be appreciated.

#48 Updated by Dan Jacques over 1 year ago

Spencer Payne wrote:

I can't seem to get this fix to work. I followed the instructions to the letter, however, when I run /usr/local/var/lib/iocage_with_pr73 start <jail-name> I get a "permission denied" message. Running it with sudo returns a "command not found" message. Any guidance would be appreciated.

I believe that the problem is that /usr/local/var/lib/iocage_with_pr73 is the directory that the iocage tool resides in, not the tool itself. You want to run /usr/local/var/lib/iocage_with_pr73/iocage.

That's my mistake; I'll see if I can update my post. Thanks for the correction!

#50 Updated by Brandon Schneider over 1 year ago

  • Status changed from In Progress to Ready for Testing
  • Needs Merging changed from Yes to No

#51 Updated by Dru Lavigne over 1 year ago

  • Needs QA changed from No to Yes
  • Needs Doc changed from Yes to No

#52 Updated by Dru Lavigne over 1 year ago

  • Copied to Bug #56340: Add ability to allocate iocage TUN devices in new UI added

#53 Updated by Bonnie Follweiler over 1 year ago

  • Status changed from Ready for Testing to Blocked
  • Reason for Blocked set to Dependent on a related task to be completed

#55 Updated by Dru Lavigne over 1 year ago

  • Status changed from Blocked to Ready for Testing
  • Reason for Blocked deleted (Dependent on a related task to be completed)

#56 Updated by Dru Lavigne over 1 year ago

  • Related to Bug #56616: Enable allow_tun for OpenVPN plugin added

#57 Updated by Dru Lavigne over 1 year ago

  • Status changed from Ready for Testing to Done
  • Needs QA changed from Yes to No

#59 Updated by Lorenz Pressler about 1 year ago

this fixes only tun devices not tap, would you consider reopening this and adding tap support also? Or should I create a new ticket?

#60 Updated by William Grzybowski about 1 year ago

Lorenz Pressler wrote:

this fixes only tun devices not tap, would you consider reopening this and adding tap support also? Or should I create a new ticket?

Create a new ticket please

#61 Updated by Peter Brille about 1 year ago

So this ticket clearly asks for a solution for TUN/TAP and you're saying you cannot fix tap because you want a dedicated ticket for that?
Please excuse my bluntness but I find this quite odd.

#62 Updated by Dru Lavigne about 1 year ago

  • Subject changed from Add ability to allocate TUN/TAP devices dynamically in iocage to Add ability to allocate TUN devices dynamically in iocage

#63 Updated by Dru Lavigne about 1 year ago

  • Related to Bug #73101: Add ability to allocate TAP devices dynamically in iocage added

Also available in: Atom PDF