Project

General

Profile

Bug #8209

Problems with warden and plugin jails

Added by Randy Westlund over 5 years ago. Updated about 4 years ago.

Status:
Closed: Behaves correctly
Priority:
Nice to have
Assignee:
John Hixson
Category:
Middleware
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

After updating to FreeNAS-9.3-STABLE-201502232343 today, I've had the following problems:

When trying to install a plugin, the installation fails with no error message visible from the GUI. /var/log/messages has this:

Feb 26 01:49:41 freenas warden: Building new Jail... Please wait...
Feb 26 01:49:41 freenas warden: zfs clone freenas-boot/usr/jails/.warden-template-pluginjail@clean freenas-boot/usr/jails/owncloud_2
Feb 26 01:49:42 freenas warden: Success!
Feb 26 01:49:42 freenas warden: Jail created at /usr/jails/owncloud_2
Feb 26 01:49:42 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: [Errno 2] No such file or directory: u'/mnt/tank/jails/owncloud_2/.plugins/PLUGIN']

My jail root is set at /mnt/tank/jails. It looks like the plugin jail is being created in the wrong location:

[randy@freenas] /mnt/tank# zfs list
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
freenas-boot                                          3.04G  11.4G    31K  none
freenas-boot/ROOT                                     3.03G  11.4G    31K  none
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201502232343     3.02G  11.4G  2.11G  /
freenas-boot/ROOT/default                             2.42M  11.4G   934M  legacy
freenas-boot/grub                                     7.82M  11.4G  7.78M  legacy
freenas-boot/usr                                        96K  11.4G    31K  none
freenas-boot/usr/jails                                  65K  11.4G    31K  none
freenas-boot/usr/jails/.warden-template-pluginjail      31K  11.4G    31K  /freenas-boot/usr/jails/.warden-template-pluginjail
freenas-boot/usr/jails/owncloud_1                        1K  11.4G    31K  none
freenas-boot/usr/jails/owncloud_2                        1K  11.4G    31K  none
freenas-boot/usr/jails/owncloud_3                        1K  11.4G    31K  none
tank                                                  1.17T  9.03T   151K  /mnt/tank
tank/.system                                          7.91M  9.03T  2.65M  legacy
tank/.system/cores                                    2.74M  9.03T  2.74M  legacy
tank/.system/rrd-480590e21d954ace9dd512462524ec0b      140K  9.03T   140K  legacy
tank/.system/samba4                                    546K  9.03T   546K  legacy
tank/.system/syslog-480590e21d954ace9dd512462524ec0b  1.84M  9.03T  1.84M  legacy
[snip]
tank/jails                                            85.1M  9.03T  85.1M  /mnt/tank/jails

I've also been seeing error messages like this appear in the GUI when I try to change settings on jail-related pages:

Request Method:     POST
Request URL:     http://freenas/jails/configuration/?inline=true
Software Version:     FreeNAS-9.3-STABLE-201502232343
Exception Type:     KeyError
Exception Value:     'ix-warden'

Environment:

Software Version: FreeNAS-9.3-STABLE-201502232343
Request Method: POST
Request URL: http://freenas/jails/configuration/?inline=true

Traceback:
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  105.                     response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/usr/local/www/freenasUI/../freenasUI/freeadmin/middleware.py" in process_view
  157.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  22.                 return view_func(request, *args, **kwargs)
File "/usr/local/www/freenasUI/../freenasUI/jails/views.py" in jailsconfiguration
  83.             form.save()
File "/usr/local/lib/python2.7/site-packages/django/forms/models.py" in save
  446.                              construct=False)
File "/usr/local/lib/python2.7/site-packages/django/forms/models.py" in save_instance
  99.         instance.save()
File "/usr/local/www/freenasUI/../freenasUI/jails/models.py" in save
  439.         notifier().start("ix-warden")
File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py" in start
  325.         self._simplecmd("start", what)
File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py" in _simplecmd
  216.             procname, pidfile = self.__service2daemon[what]

Exception Type: KeyError at /jails/configuration/
Exception Value: 'ix-warden'
Request information
GET
Variable     Value
inline     u'true'
POST
Variable     Value
jc_collectionurl     u'http://download.freenas.org/latest/RELEASE/x64/jails'
jc_ipv6_network_start     u''
jc_ipv4_network     u'192.168.1.0/24'
__form_id     u'dialogForm_'
jc_path     u'/mnt/tank/jails'
jc_ipv6_network     u''
jc_ipv4_network_start     u'192.168.1.220'
jc_ipv6_network_end     u''
jc_ipv4_network_end     u'192.168.1.240'
__all__     u''
FILES

No FILES data
COOKIES
Variable     Value
csrftoken     '4CzIjPwtdHC21WZGKdCS2jomN4rGuk7z'
sessionid     'ce7pmm4etoq0kn7za1s76etfu6x2kfvy'
fntreeSaveStateCookie     'root%2Croot%2F258'
META
Variable     Value
wsgi.input     <flup.server.fcgi_base.InputStream object at 0x81516f650>
QUERY_STRING     'inline=true'
HTTP_ACCEPT     'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
CONTENT_LENGTH     '308'
REQUEST_URI     '/jails/configuration/?inline=true'
wsgi.version     (1, 0)
HTTP_ACCEPT_ENCODING     'gzip, deflate'
HTTP_PRAGMA     'no-cache'
HTTP_REFERER     'http://freenas/'
DOCUMENT_ROOT     '/usr/local/etc/nginx/html'
HTTP_CACHE_CONTROL     'no-cache'
REMOTE_ADDR     '192.168.1.30'
SERVER_PORT     '80'
GATEWAY_INTERFACE     'CGI/1.1'
wsgi.url_scheme     'http'
wsgi.multiprocess     False
CSRF_COOKIE     u'4CzIjPwtdHC21WZGKdCS2jomN4rGuk7z'
HTTP_ACCEPT_LANGUAGE     'en-US,en;q=0.5'
SERVER_SOFTWARE     'nginx/1.6.2'
wsgi.run_once     False
HTTP_COOKIE     'csrftoken=4CzIjPwtdHC21WZGKdCS2jomN4rGuk7z; sessionid=ce7pmm4etoq0kn7za1s76etfu6x2kfvy; fntreeSaveStateCookie=root%2Croot%2F258'
REMOTE_PORT     '31819'
SCRIPT_NAME     u''
CONTENT_TYPE     'application/x-www-form-urlencoded; charset=UTF-8'
HTTP_CONTENT_LENGTH     '308'
HTTP_DNT     '1'
SERVER_PROTOCOL     'HTTP/1.1'
HTTP_X_REQUESTED_WITH     'XMLHttpRequest'
REDIRECT_STATUS     '200'
wsgi.errors     <flup.server.fcgi_base.TeeOutputStream object at 0x815198dd0>
HTTP_CONTENT_TYPE     'application/x-www-form-urlencoded; charset=UTF-8'
HTTP_CONNECTION     'keep-alive'
PATH_INFO     u'/jails/configuration/'
DOCUMENT_URI     '/jails/configuration/'
HTTP_USER_AGENT     'Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0'
SERVER_ADDR     '192.168.1.250'
REQUEST_METHOD     'POST'
SERVER_NAME     'localhost'
wsgi.multithread     True
HTTP_HOST     'freenas'
HTTP_X_CSRFTOKEN     '4CzIjPwtdHC21WZGKdCS2jomN4rGuk7z'
HTTPS     ''

Debug log is attached.

2015-02-26-235244_1366x768_scrot.png (176 KB) 2015-02-26-235244_1366x768_scrot.png Randy Westlund, 02/26/2015 08:56 PM
2261

History

#1 Updated by John Hixson over 5 years ago

  • Status changed from Unscreened to 15
  • Priority changed from Important to Nice to have
  • Target version set to Unspecified

I'm not clear on what's going on but it's certainly wrong. Do you have any other jails or plugins installed? Can you attach your /usr/local/etc/warden.conf here? Can you also provide the output of warden list -v? If you don't have any other jails or plugins, I think you should just blow away the existing jails and templates.

#2 Updated by Randy Westlund over 5 years ago

John Hixson wrote:

I'm not clear on what's going on but it's certainly wrong. Do you have any other jails or plugins installed? Can you attach your /usr/local/etc/warden.conf here? Can you also provide the output of warden list -v? If you don't have any other jails or plugins, I think you should just blow away the existing jails and templates.

No, I have no other jails or plugins. This is the first time I've tried to install a plugin.

[randy@freenas] /usr/local/etc# cat warden.conf 
#!/bin/sh
# Configuration options for the Warden
######################################################################
# Network Interface for the jails to use
NIC: re0
# Directory to use for compressing / decompressing files 
WTMP: /usr/jails
# Location of the jails
JDIR: /usr/jails
# FreeBSD release to use
FREEBSD_RELEASE: 9.1-RELEASE
[randy@freenas] /mnt/tank/home/randy# warden list -v
id: 1
host: owncloud_1
iface: 
ipv4: 192.168.1.3/24
alias-ipv4: 
bridge-ipv4: 
alias-bridge-ipv4: 
defaultrouter-ipv4: 
ipv6: 
alias-ipv6: 
bridge-ipv6: 
alias-bridge-ipv6: 
defaultrouter-ipv6: 
autostart: Disabled
vnet: Disabled
nat: Disabled
mac: 
status: Stopped
type: pluginjail
flags: allow.raw_sockets=true

id: 2
host: owncloud_2
iface: 
ipv4: 192.168.1.4/24
alias-ipv4: 
bridge-ipv4: 
alias-bridge-ipv4: 
defaultrouter-ipv4: 
ipv6: 
alias-ipv6: 
bridge-ipv6: 
alias-bridge-ipv6: 
defaultrouter-ipv6: 
autostart: Disabled
vnet: Disabled
nat: Disabled
mac: 
status: Stopped
type: pluginjail
flags: allow.raw_sockets=true

id: 3
host: owncloud_3
iface: 
ipv4: DHCP
alias-ipv4: 
bridge-ipv4: 
alias-bridge-ipv4: 
defaultrouter-ipv4: 
ipv6: 
alias-ipv6: 
bridge-ipv6: 
alias-bridge-ipv6: 
defaultrouter-ipv6: 
autostart: Disabled
vnet: Disabled
nat: Disabled
mac: 
status: Stopped
type: pluginjail
flags: allow.raw_sockets=true

How should I reset it all?

#3 Updated by John Hixson over 5 years ago

In your warden.conf, you have /usr/jails as the jails path. That can't be right. Can you go to jails->configuration and pick your jail root and save it?

#4 Updated by Randy Westlund over 5 years ago

2261

John Hixson wrote:

In your warden.conf, you have /usr/jails as the jails path. That can't be right. Can you go to jails->configuration and pick your jail root and save it?

The current value for my jail root in jails->configuration is /mnt/tank/jails. When I click save on that page, I get the error that I posted above with the python stack trace. I've attached a screenshot as well.

I'm not sure why these have different values, but I didn't change anything manually and have never attempted to create a jail or install a plugin before. I assume this was caused by the last update, but I don't know for sure.

#5 Updated by John Hixson over 5 years ago

There was an update yesterday that may have addressed this issue. I see in the traceback a bug that was indeed fixed. Can you grab the latest update(s) and see if this resolves your issue?

#6 Updated by Randy Westlund over 5 years ago

John Hixson wrote:

There was an update yesterday that may have addressed this issue. I see in the traceback a bug that was indeed fixed. Can you grab the latest update(s) and see if this resolves your issue?

The update helped, but not completely. The stack trace no longer appears when I save jail settings and /usr/local/etc/warden.conf looks good now. But when I try to install a plugin, it fails after downloading and trying to create a jail with no message in the UI. A jail for the plugin is not created.

This is in /var/log/messages:

Feb 27 22:19:26 freenas nmbd[2592]:   STATUS=daemon 'nmbd' finished starting up and ready to serve connections*****
Feb 27 22:19:26 freenas nmbd[2592]:   
Feb 27 22:19:26 freenas nmbd[2592]:   Samba name server FREENAS is now a local master browser for workgroup WORKGROUP on subnet 192.168.1.250
Feb 27 22:19:26 freenas nmbd[2592]:   
Feb 27 22:19:26 freenas nmbd[2592]:   *****
Feb 27 22:19:29 freenas syslog-ng[3860]: syslog-ng starting up; version='3.5.6'
Feb 27 22:19:29 freenas kernel: pid 1862 (syslog-ng), uid 0: exited on signal 6 (core dumped)
Feb 27 22:19:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:20:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:20:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:21:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:21:32 freenas su: randy to root on /dev/pts/0
Feb 27 22:21:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:22:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:22:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:23:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:23:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:24:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:24:40 freenas manage.py: [common.pipesubr:58] Popen()ing: /sbin/ping -q -t 2 -o 192.168.1.220
Feb 27 22:24:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:25:02 freenas manage.py: [common.pipesubr:58] Popen()ing: /sbin/ping -q -t 2 -o 192.168.1.220
Feb 27 22:25:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:25:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:26:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:26:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:27:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:27:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:28:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:28:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:29:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:29:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:30:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:30:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:31:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:31:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:32:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:32:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:33:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:33:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:34:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:34:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:35:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:35:58 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory
Feb 27 22:36:28 freenas init: can't exec getty '/usr/local/sbin/zfsd' for port /dev/zfsd: No such file or directory

#7 Updated by Randy Westlund over 5 years ago

I came back an hour later and the jail, dataset, and plugin all exist now. Maybe it was still processing but the UI popup closed? The owncloud plugin is working now, but I don't know what's going on with syslog.

#8 Updated by John Hixson over 5 years ago

  • Status changed from 15 to Closed: Behaves correctly

#9 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Target version changed from Unspecified to N/A

#10 Updated by Dru Lavigne almost 3 years ago

  • File deleted (debug-freenas-20150226022755.tgz)

Also available in: Atom PDF