Project

General

Profile

Bug #6632

Upgrade from 9.2.1.8 to 9.3-BETA Broke AD Connector

Added by DENNY VANDEMAELE about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Nice to have
Assignee:
John Hixson
Category:
Services
Target version:
Start date:
11/12/2014
Due date:
% Done:

0%

Seen in:
Backlog Priority:
Hardware Configuration:
ChangeLog Required:
No
Needs QA:
Yes
QA Status:
Not Tested

Description

I had a 9.2.1.8 FreeNAS (non production) box properly connected to Active Directory (since 9.2.1.8 came out) that I wanted to test 9.3-BETA out on.

After upgrading to 9.3-BETA, I have not been able to get it to connect to AD.

Troubleshooting steps taken:
  • time is in sync with AD (FN and AD have same timezone and ntp servers)
  • dns records for client and server (including srv records) can be looked up successfully bi-directionally.
  • my other 9.2.1.8 servers AD Service can be stopped and restarted without issue.

Commandline Output of troubleshooting so far... fails on ix-kinit:

FreeBSD 9.3-RELEASE-p5 (FREENAS.amd64) #0 3ea9a5b: Fri Nov 7 16:44:15 PST 2014
[root@FREENAS] ~# sqlite3 /data/freenas-v1.db "update services_services set srv_enable = 1 where srv_service = 'directoryservice'"
[root@FREENAS] ~# service ix-kerberos start
[root@FREENAS] ~# service ix-kinit start
[root@FREENAS] ~# service ix-kinit status
[root@FREENAS] ~# echo $?
1
[root@FREENAS] ~#

History

#1 Updated by John Hixson about 3 years ago

  • Status changed from Unscreened to Screened

#2 Updated by Mathieu Gauthier-Lafaye about 3 years ago

I think I have the same issue on a fresh install. Is there a way to get an explicit error ? In the interface it just say : "The service failed to restart". And in the logs :

Nov 14 11:06:26 testnas02 ActiveDirectory: /usr/local/bin/python /usr/local/www/freenasUI/middleware/notifier.py stop cifs
Nov 14 11:06:28 testnas02 notifier: winbindd not running? (check /var/run/samba/winbindd.pid).
Nov 14 11:06:28 testnas02 notifier: smbd not running? (check /var/run/samba/smbd.pid).
Nov 14 11:06:28 testnas02 notifier: nmbd not running? (check /var/run/samba/nmbd.pid).
Nov 14 11:06:29 testnas02 ActiveDirectory: /usr/sbin/service ix-kerberos quietstart
Nov 14 11:06:31 testnas02 ActiveDirectory: /usr/sbin/service ix-nsswitch quietstart
Nov 14 11:06:32 testnas02 ActiveDirectory: /usr/sbin/service ix-kinit quietstart
Nov 14 11:06:34 testnas02 ActiveDirectory: /usr/sbin/service ix-kinit status

#3 Updated by Kevin Woodbrey about 3 years ago

I have the same situation. "The service failed to restart" This happened on an upgrade and a fresh install.

Nov 25 23:01:50 solar2 ActiveDirectory: /usr/local/bin/python /usr/local/www/freenasUI/middleware/notifier.py stop cifs
Nov 25 23:01:53 solar2 notifier: winbindd not running? (check /var/run/samba/winbindd.pid).
Nov 25 23:01:53 solar2 notifier: smbd not running? (check /var/run/samba/smbd.pid).
Nov 25 23:01:53 solar2 notifier: nmbd not running? (check /var/run/samba/nmbd.pid).
Nov 25 23:01:54 solar2 ActiveDirectory: /usr/sbin/service ix-kerberos quietstart
Nov 25 23:01:56 solar2 ActiveDirectory: /usr/sbin/service ix-nsswitch quietstart
Nov 25 23:01:58 solar2 ActiveDirectory: /usr/sbin/service ix-kinit quietstart
Nov 25 23:02:08 solar2 ActiveDirectory: /usr/sbin/service ix-kinit status

#4 Updated by Mathieu Gauthier-Lafaye about 3 years ago

I just upgrade the system to check if the bug still exists. I'm now on version 9.3-BETA 2014-11-26 02:26:21 GMT.

I'm trying to join an domain in a forest (eg: mydomain.inforest.ad).

Nov 26 11:57:32 testnas02 manage.py: [common.freenasldap:1067] FreeNAS_ActiveDirectory_Base.get_SRV_records: no SRV records for _gc._tcp.mydomain.inforest.ad found, fail!
Nov 26 11:57:32 testnas02 manage.py: [directoryservice.models:935] ActiveDirectory: Unable to create kerberos realm: Unable to find global catalog servers for mydomain.inforest.ad

Effectively, there is no "_gc._tcp" in subdomain zone. The "_gc._tcp" record can be found in the parent (inforest.ad). That seems normal behavior as you can read in this document : http://technet.microsoft.com/en-us/library/cc961719.aspx

_ gc._tcp. DnsForestName .

Allows a client to locate a Global Catalog (gc) server for this domain. The server is not necessarily a domain controller. Only a server that is running the LDAP service and functioning as the Global Catalog server for the forest named in DnsForestName registers this SRV record (for example, _gc._tcp.reskit.com.).

Tell me if you need more information !

#5 Updated by Mathieu Gauthier-Lafaye about 3 years ago

To complete my previous message. When I set the Global Catalog in the Freenas interface. That's make it work. Even if set a specific controller in the configuration is not for me the best practice.

The wbinfo command give the list of the users but not getent. It's seem that SSSD is not functional and cannot be started.

# cat /etc/nsswitch.conf | grep sss
group: files sss
passwd: files sss

# /usr/local/etc/sssd/sssd.conf
[sssd]
config_file_version = 2
full_name_format = %2$s\%1$s
re_expression = (((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(^(?P<name>[^@\\]+)$))
services = nss,pam
[nss]
[pam]

#6 Updated by John Hixson about 3 years ago

  • Target version set to 9.3-RELEASE

#7 Updated by John Hixson about 3 years ago

Mathieu Gauthier-Lafaye wrote:

To complete my previous message. When I set the Global Catalog in the Freenas interface. That's make it work. Even if set a specific controller in the configuration is not for me the best practice.

The wbinfo command give the list of the users but not getent. It's seem that SSSD is not functional and cannot be started.

[...]

Does your Active Directory have UNIX attributes? (SFU schema)? If not, then SSSD won't work (and you should not have the UNIX extensions option checked as well).

#8 Updated by DENNY VANDEMAELE about 3 years ago

Hello John,

I wrote the original Bug Report, and two other (non-ix) users replied to it. Not sure if the bug reports are supposed to work this way? It's a bit confusing.

Are there any additional troubleshooting steps I can take, or logs to submit?

I'm a little worried about my AD breaking since I have a production TrueNAS server (with maintenance agreement) that will some day be on 9.3 Release.

Thank You,

Denny

#9 Updated by John Hixson about 3 years ago

DENNY VANDEMAELE wrote:

Hello John,

I wrote the original Bug Report, and two other (non-ix) users replied to it. Not sure if the bug reports are supposed to work this way? It's a bit confusing.

Are there any additional troubleshooting steps I can take, or logs to submit?

I'm a little worried about my AD breaking since I have a production TrueNAS server (with maintenance agreement) that will some day be on 9.3 Release.

Thank You,

Denny

The troubleshooting steps you've taken aren't valid for 9.3. I'll need to update that thread for 9.3. Can you try to start AD, then attach your /var/log/messages to this ticket?

#10 Updated by Mathieu Gauthier-Lafaye about 3 years ago

Hello Denny and John,

All my apologies Denny, I was just trying to not create a new ticket if the problem was the same and at the first look it seemed the same. I guess can create a new ticket if you think it's not related to your problem. That's something I forgot to ask... Let me know.

John, yes we use unix attributes. Actually, the two controllers 2008R2 have the extension installed. And, we plan to install it also on the two controllers 2012R2. But I guess the schema is global and that should not be a problem. Maybe, I'm wrong.

Mathieu

#11 Updated by DENNY VANDEMAELE about 3 years ago

John,

Here are the /var/log/message entries as my box was attempting to join to domain:

Nov 27 07:57:20 FREENAS winbindd2353: [2014/11/27 07:57:20.078649, 0] ../source3/winbindd/winbindd_samr.c:769(sam_rids_to_names)
Nov 27 07:57:20 FREENAS winbindd2353: sam_rids_to_names: possible deadlock - trying to lookup SID S-1-5-21-2165810240-2858082434-4084120154
Nov 27 07:57:28 FREENAS mountd1861: umountall request succeeded from 192.168.200.244
Nov 27 08:00:04 FREENAS ActiveDirectory: /usr/local/bin/python /usr/local/www/freenasUI/middleware/notifier.py stop cifs
Nov 27 08:00:06 FREENAS notifier: Stopping winbindd.
Nov 27 08:00:06 FREENAS winbindd2348: STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig15 terminate (is_parent=1)
Nov 27 08:00:06 FREENAS winbindd2391: [2014/11/27 08:00:06.806902, 0] ../source3/winbindd/winbindd.c:234(winbindd_sig_term_handler)
Nov 27 08:00:06 FREENAS winbindd2391: Got sig15 terminate (is_parent=0)
Nov 27 08:00:06 FREENAS winbindd2361: [2014/11/27 08:00:06.807216, 0] ../source3/winbindd/winbindd.c:234(winbindd_sig_term_handler)
Nov 27 08:00:06 FREENAS winbindd2361: Got sig15 terminate (is_parent=0)
Nov 27 08:00:06 FREENAS winbindd2353: [2014/11/27 08:00:06.807966, 0] ../source3/winbindd/winbindd.c:234(winbindd_sig_term_handler)
Nov 27 08:00:06 FREENAS winbindd2353: Got sig15 terminate (is_parent=0)
Nov 27 08:00:06 FREENAS notifier: Waiting for PIDS: 2348.
Nov 27 08:00:06 FREENAS notifier: Stopping smbd.
Nov 27 08:00:06 FREENAS notifier: Waiting for PIDS: 2345.
Nov 27 08:00:06 FREENAS notifier: Stopping nmbd.
Nov 27 08:00:06 FREENAS nmbd2342: STATUS=daemon 'nmbd' finished starting up and ready to serve connectionsGot SIGTERM: going down...
Nov 27 08:00:06 FREENAS notifier: Waiting for PIDS: 2342.
Nov 27 08:00:07 FREENAS ActiveDirectory: /usr/sbin/service ix-kerberos quietstart
Nov 27 08:00:09 FREENAS ActiveDirectory: /usr/sbin/service ix-nsswitch quietstart
Nov 27 08:00:11 FREENAS ActiveDirectory: /usr/sbin/service ix-kinit quietstart
Nov 27 08:00:16 FREENAS ActiveDirectory: /usr/sbin/service ix-kinit status

Matthew, no problem in that we do have the same problem (needing to know how to diagnose).
Thanks all!

#12 Updated by William Grzybowski about 3 years ago

I wonder if thats not yet another problem with the password encryption.

Can you do this:

[root@freenas] ~# sh -c 'python /usr/local/www/freenasUI/middleware/notifier.py pwenc_decrypt $(sqlite3 /data/freenas-v1.db "select ad_bindpw from directoryservice_activedirectory")'

And make sure it outputs the correct password?

What is the exact BETA version you're trying?

#13 Updated by DENNY VANDEMAELE about 3 years ago

Hello William,

I've been checking for updates daily since upgrading to 9.3 beta. My current version is: FreeNAS 9.3-BETA 2014-11-27 02:55:32 GMT

The command line you provided does return the correct password. (also note that when I enter the wrong password in the gui, it does notify me).

Thank You, Denny

#14 Updated by John Hixson about 3 years ago

Denny,

Can you try the following? (from the CLI of course)

sqlite3 /data/freenas-v1.db

update directoryservice_activedirectory set ad_enable=1;
.quit

After you do that (you're tricking the system into thinking AD is enabled), do this:

service ix-kerberos start
service ix-kinit start
klist

You should see a kerberos ticket granting ticket. If you don't, kerberos is failing and we will need to figure out why. Attack your /etc/krb5.conf file if this does in fact fail.

#15 Updated by DENNY VANDEMAELE about 3 years ago

Thanks John,

Output of commands:

[root@FREENAS] ~# sqlite3 /data/freenas-v1.db
SQLite version 3.8.6 2014-08-15 11:46:33
Enter ".help" for usage hints.
sqlite> update directoryservice_activedirectory set ad_enable=1;
sqlite> .quit

[root@FREENAS] ~# service ix-kerberos start
[root@FREENAS] ~# service ix-kinit start
Traceback (most recent call last):
File "/usr/local/bin/adtool", line 606, in <module>
main()
File "/usr/local/bin/adtool", line 592, in main
adts = ADToolShell()
File "/usr/local/bin/adtool", line 36, in init
self.adc = ActiveDirectoryConfig(flags=FLAGS_DBINIT)
File "/usr/local/www/freenasUI/common/freenasldap.py", line 2059, in init
super(FreeNAS_ActiveDirectory, self).__init__(**kwargs)
File "/usr/local/www/freenasUI/common/freenasldap.py", line 1437, in init
self.set_servers()
File "/usr/local/www/freenasUI/common/freenasldap.py", line 1542, in set_servers
self.set_domain_controller()
File "/usr/local/www/freenasUI/common/freenasldap.py", line 1502, in set_domain_controller
"Unable to find domain controllers for %s" % self.domainname)
freenasUI.common.freenasldap.FreeNAS_ActiveDirectory_Exception: Unable to find domain controllers for NASSAL.LOCAL

[root@FREENAS] ~# klist
klist: No ticket file: /tmp/krb5cc_0

[root@FREENAS] ~# host -t srv _ldap._tcp.nassal.local
_ldap._tcp.nassal.local has SRV record 0 100 389 server.nassal.local.
_ldap._tcp.nassal.local has SRV record 0 100 389 server2.nassal.local.

[root@FREENAS] ~# cat /etc/krb5.conf
[appdefaults]
pam = {
forwardable = true
ticket_lifetime = 86400
renew_lifetime = 86400
}

[libdefaults]
dns_lookup_realm = true
dns_lookup_kdc = true
ticket_lifetime = 24h
clockskew = 300
forwardable = yes

[logging]
default = SYSLOG:INFO:LOCAL7

#16 Updated by John Hixson about 3 years ago

The problem here is very clear. Why it is happening, however, is not clear ;-) I'd like to get a webex setup to troubleshoot this more. Are you available tomorrow ? Shoot me your info if so: .

#17 Updated by John Hixson about 3 years ago

  • Status changed from Screened to Resolved

So the issue here ended up being there was no subnet associated with the site. This is not common, but it's an edge case that needs to be handled, so I'll write code to handle it ;-) Denny created a subnet and associated it with the site and everything worked fine afterwards. So, I consider this resolved.

#18 Updated by Arthur Brownlee IV about 3 years ago

John Hixson wrote:

So the issue here ended up being there was no subnet associated with the site. This is not common, but it's an edge case that needs to be handled, so I'll write code to handle it ;-) Denny created a subnet and associated it with the site and everything worked fine afterwards. So, I consider this resolved.

Would this also apply for IPv6? Or would it fall over to IPv4 for the site?

Also available in: Atom PDF