Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Thanks for using the TrueNAS Community Edition issue tracker! TrueNAS Enterprise users receive direct support for their reports from our support portal.

LDAP Directory Service Connection Error after update to SCALE 22.02.1

Description

After updating TrueNAS Scale from 22.02.0.1 to 22.02.1 I can't connect the LDAP Directory Service anymore:

[2022/05/05 10:01:43] (WARNING) application.call_method():210 - Exception while calling ldap.update(*[{'hostname': ['ipa01.ipa.mydomain.com'], 'basedn': 'dc=ipa,dc=mydomain,dc=com', 'binddn': 'uid=truenas,cn=users,cn=accounts,dc=ipa,dc=mydomain,dc=com', 'enable': True, 'anonbind': False, 'ssl': 'ON', 'certificate': None, 'validate_certificates': True, 'disable_freenas_cache': False, 'kerberos_realm': 1, 'kerberos_principal': '', 'timeout': 30, 'dns_timeout': 30, 'has_samba_schema': False, 'auxiliary_parameters': 'base passwd cn=users,cn=accounts,dc=ipa,dc=mydomain,dc=com\nbase group cn=groups,cn=compat,dc=ipa,dc=mydomain,dc=com', 'schema': 'RFC2307'}])
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 175, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/service.py", line 574, in update
rv = await self.middleware._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1129, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 1013, in do_update
await self.middleware.call('ldap.ldap_validate', new, verrors)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 835, in ldap_validate
await self.middleware.call('ldap.validate_credentials', data)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 1072, in validate_credentials
await self.middleware.call('ldapclient.validate_credentials', client_conf)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 185, in validate_credentials
await self.middleware.run_in_executor(self.thread_pool, self._open, data, True)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1169, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 328, in _open
raise saved_error
File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 307, in _open
self._handle.sasl_gssapi_bind_s()
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 489, in sasl_gssapi_bind_s
self.sasl_non_interactive_bind_s('GSSAPI',serverctrls,clientctrls,sasl_flags,authz_id)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 477, in sasl_non_interactive_bind_s
self.sasl_interactive_bind_s('',auth,serverctrls,clientctrls,sasl_flags)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 467, in sasl_interactive_bind_s
return self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 331, in _ldap_call
reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib/python3/dist-packages/ldap/compat.py", line 44, in reraise
raise exc_value
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 315, in _ldap_call
result = func(args,*kwargs)
ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server", 'errno': 115, 'info': '(unknown error code)'}

Manually doing a "ldapsearch" from CLI works fine. DNS also works fine. Disabling certificate validation in LDAP settings did not help (I tried as there was something about legacy certificates in the update notes).
Booting back into 22.02.0.1 and LDAP DS is working as expected again.

Forgot to grab debug logs while I was booted into 22.02.1 but I can try to reboot and get them later/during off-hours if needed.

Created a forum thread:
https://www.truenas.com/community/threads/truenas-scale-22-02-1-ldap-directory-service-error.100961/

Problem/Justification

None

Impact

None

Activity

Show:

Andrew Walker 
May 10, 2022 at 3:52 PM

We resolved this issue in a teamviewer session. There is some history to this ticket.

Prior to FreeNAS 11.3 users could select a single CA cert to use as the cacertfile in libldap. This often caused problems because users may have needed to add more than one cacert and / or needed access the the default root cacerts otherwise certificate validation would fail. To remedy this in FreeNAS 11.3 release, a single concatenated cacert file was created that contained the root cacerts as well as user-provided one `/etc/ssl/truenas_cacerts.pem`.

Over time the certificate / certificateauthority plugins evolved to eventually create a similarly concatenated file `/etc/ssl/certs/ca-certificates.crt`, with the crucial distinction that the certificateauthority plugin had new key `add_to_trusted_store` that was used to determine whether a particular CA should be added to consolidated ca-certificates file. The migration that added said key, set it to `false` for all existing user-provided ca-certs.

The 22.02.1 update included a change to remove the legacy truenas_cacerts.pem file and use the newly properly-managed ca-certificates store instead. Unfortunately, due to the defaults in the migration some users with ca certificates added prior to the migration (or who had uploaded ca certificates but not set the "add to trusted store" option) broke on upgrade to 22.02.1.

Since there is not a clear path forward to remedy the situation programmatically (all legacy cacerts were migrated to not be in the cacert store), I will create a corresponding documentation team ticket to add this as a known-impact for 22.02.1.

Andrew Walker 
May 5, 2022 at 1:01 PM

Yeah, might be a break caused by changes in our cert-handling code. Can we schedule a teamviewer session for some time tomorrow between (6:00 AM CDT and 4:00 PM CDT)? We can coordinate through email awalker@ixsystems.com.

Pascal Pascher 
May 5, 2022 at 11:40 AM
(edited)

It appears I got it working again. Here is what I did:

Credentials->Directory Services->LDAP->Settings
-> uncheck "enabled" – > save

Credentials->Directory Services->Advanced Settings->Idmap

-> noticed with url set to "ldaps://ipa01.ipa.mydomain.com" I got an error with duplicate "ldaps://ldaps://...".
-> removed explicit "ldaps://" from URL entry.
-> set "Encryption Mode" to "Off"
-> save -> working
-> reenabled "Encryption Mode" "SSL" -> save -> working

Credentials->Directory Services->LDAP->Settings->Advanced Options
-> unchecked "Validate Certificates" -> checked "Enable" -> works
-> checked "Validate Certificates" to test -> does not work (same error as original post).

Not sure if all those steps were needed, but only after getting the existing Idmap Entry under "Advanced Settings" to connect again did disabling "Validate Certificates" help solve the issue. Before it would not change anything.

So maybe it is indeed linked to the certificate change for LDAP mentioned in the changelog?
https://ixsystems.atlassian.net/browse/NAS-115211#icft=NAS-115211 - Remove legacy LDAP truenas_cacerts.pem file

Pascal Pascher 
May 5, 2022 at 11:25 AM

Hi Andrew,
done.
I tried both
ldapsearch -h ipa01.ipa.mydomain.com -Y GSSAPI -b cn=groups,cn=compat,dc=ipa,dc=mydomain,dc=com
and
ldapsearch -h ipa01.ipa.mydomain.com -D uid=truenas,cn=users,cn=accounts,dc=ipa,dc=mydomain,dc=com -W -b cn=groups,cn=compat,dc=ipa,dc=mydomain,dc=com
from CLI and both work (latter prompting for PW, former GSSAPI without password).

Andrew Walker 
May 5, 2022 at 10:15 AM

Can you attach a debug please.

Behaves as Intended

Details

Assignee

Reporter

Labels

Time remaining

0m

Components

Fix versions

Affects versions

Priority

Katalon Platform

Created May 5, 2022 at 10:01 AM
Updated July 6, 2022 at 9:01 PM
Resolved May 10, 2022 at 4:13 PM