Appendix A. Troubleshooting
Appendix A. Troubleshooting A.1. Troubleshooting SSSD Section A.1.1, "Setting Debug Logs for SSSD Domains" Section A.1.2, "Checking SSSD Log Files" Section A.1.3, "Problems with SSSD Configuration" A.1.1. Setting Debug Logs for SSSD Domains Each domain sets its own debug log level. Increasing the log level can provide more information about problems with SSSD or with the domain configuration. To change the log level, set the debug_level parameter for each section in the sssd.conf file for which to produce extra logs. For example: Table A.1. Debug Log Levels Level Description 0 Fatal failures. Anything that would prevent SSSD from starting up or causes it to cease running. 1 Critical failures. An error that does not kill the SSSD, but one that indicates that at least one major feature is not going to work properly. 2 Serious failures. An error announcing that a particular request or operation has failed. 3 Minor failures. These are the errors that would percolate down to cause the operation failure of 2. 4 Configuration settings. 5 Function data. 6 Trace messages for operation functions. 7 Trace messages for internal control functions. 8 Contents of function-internal variables that may be interesting. 9 Extremely low-level tracing information. To change the debug level while SSSD is running, use the sss_debuglevel utility, which is part of the sssd-tools package. For more information about how it works, see the sss_debuglevel man page. A.1.2. Checking SSSD Log Files SSSD uses a number of log files to report information about its operation, located in the /var/log/sssd/ directory. SSSD produces a log file for each domain, as well as an sssd_pam.log and an sssd_nss.log file. krb5_child.log : log file for the short-lived helper process involved in Kerberos authentication ldap_child.log : log file for the short-lived helper process involved in communicating with the LDAP server selinux_child.log : log file for the short-lived helper process that retrieves SELinux information sssd.log : log file for SSSD communicating with its responder processes sssd_[domain].log : each SSSD domain section logs information about communication with the LDAP server to a separate log file sssd_ifp.log : log file for the InfoPipe responder, provides a public D-Bus interface accessible over the system bus sssd_nss.log : log file for the Name Services Switch (NSS) responder that retrieves user and group information sssd_pac.log : log file for the Microsoft Privilege Attribute Certificate (PAC) responder service that defines how SSSD works with Kerberos to manage Active Directory users and groups sssd_pam.log : log file for the Pluggable Authentication Module (PAM) responder sssd_ssh.log : log file for the SSH responder process Additionally, the /var/log/secure file logs authentication failures and the reason for the failure. A.1.3. Problems with SSSD Configuration Q: SSSD fails to start Q: I do not see any groups with id or group members with getent group. Q: Authentication fails against LDAP. Q: Connecting to LDAP servers on non-standard ports fail. Q: NSS fails to return user information Q: NSS returns incorrect user information Q: Setting the password for the local SSSD user prompts twice for the password Q: An Active Directory identity provider is properly configured in my sssd.conf file, but SSSD fails to connect to it, with GSS-API errors. Q: I configured SSSD for central authentication, but now several of my applications (such as Firefox or Adobe) will not start. Q: SSSD is showing an automount location that I removed. Q: SSSD fails to start A: SSSD requires that the configuration file be properly set up, with all the required entries, before the daemon will start. SSSD requires at least one properly configured domain before the service will start. Without a domain, attempting to start SSSD returns an error that no domains are configured: Edit the /etc/sssd/sssd.conf file and create at least one domain. SSSD also requires at least one available service provider before it will start. If the problem is with the service provider configuration, the error message indicates that there are no services configured: Edit the /etc/sssd/sssd.conf file and configure at least one service provider. Important SSSD requires that service providers be configured as a comma-separated list in a single services entry in the /etc/sssd/sssd.conf file. If services are listed in multiple entries, only the last entry is recognized by SSSD. SSSD also requires the ownership and permissions of the /etc/sssd/sssd.conf to be set correctly. If the ownership or permissions are set incorrectly, attempt to start SSSD returns these error messages: Set the correct ownership and permissions of the /etc/sssd/sssd.conf file: Q: I do not see any groups with id or group members with getent group . A: This may be due to an incorrect ldap_schema setting in the [domain/DOMAINNAME] section of sssd.conf . SSSD supports RFC 2307 and RFC 2307bis schema types. By default, SSSD uses the more common RFC 2307 schema. The difference between RFC 2307 and RFC 2307bis is the way which group membership is stored in the LDAP server. In an RFC 2307 server, group members are stored as the multi-valued memberuid attribute, which contains the name of the users that are members. In an RFC2307bis server, group members are stored as the multi-valued member or uniqueMember attribute which contains the DN of the user or group that is a member of this group. RFC2307bis allows nested groups to be maintained as well. If group lookups are not returning any information: Set ldap_schema to rfc2307bis . Delete /var/lib/sss/db/cache_DOMAINNAME.ldb . Restarting SSSD. If that does not work, add this line to sssd.conf : Then delete the cache and restart SSSD again. Q: Authentication fails against LDAP. A: To perform authentication, SSSD requires that the communication channel be encrypted. This means that if sssd.conf is configured to connect over a standard protocol ( ldap:// ), it attempts to encrypt the communication channel with Start TLS. If sssd.conf is configured to connect over a secure protocol ( ldaps:// ), then SSSD uses SSL. This means that the LDAP server must be configured to run in SSL or TLS. TLS must be enabled for the standard LDAP port (389) or SSL enabled on the secure LDAPS port (636). With either SSL or TLS, the LDAP server must also be configured with a valid certificate trust. An invalid certificate trust is one of the most common issues with authenticating against LDAP. If the client does not have proper trust of the LDAP server certificate, it is unable to validate the connection, and SSSD refuses to send the password. The LDAP protocol requires that the password be sent in plain text to the LDAP server. Sending the password in plain text over an unencrypted connection is a security problem. If the certificate is not trusted, a syslog message is written, indicating that TLS encryption could not be started. The certificate configuration can be tested by checking if the LDAP server is accessible apart from SSSD. For example, this tests an anonymous bind over a TLS connection to test.example.com : If the certificate trust is not properly configured, the test fails with this error: To trust the certificate: Obtain a copy of the public CA certificate for the certificate authority used to sign the LDAP server certificate and save it to the local system. Add a line to the sssd.conf file that points to the CA certificate on the filesystem. If the LDAP server uses a self-signed certificate, remove the ldap_tls_reqcert line from the sssd.conf file. This parameter directs SSSD to trust any certificate issued by the CA certificate, which is a security risk with a self-signed CA certificate. Q: Connecting to LDAP servers on non-standard ports fail. A: When running SELinux in enforcing mode, the client's SELinux policy has to be modified to connect to the LDAP server over the non-standard port. For example: Q: NSS fails to return user information A: This usually means that SSSD cannot connect to the NSS service. Ensure that the NSS service is running: NSS service is running when SSSD is in the Active: active (running) state and when the output includes sssd_nss . If NSS is running, make sure that the provider is properly configured in the [nss] section of the /etc/sssd/sssd.conf file. Especially check the filter_users and filter_groups attributes. Make sure that NSS is included in the list of services that SSSD uses. Check the configuration in the /etc/nsswitch.conf file. For more information, see the section called "Configure NSS Services to Use SSSD" . Q: NSS returns incorrect user information A: If searches are returning the incorrect user information, check that there are not conflicting user names in separate domains. When there are multiple domains, set the use_fully_qualified_domains attribute to true in the /etc/sssd/sssd.conf file. This differentiates between different users in different domains with the same name. Q: Setting the password for the local SSSD user prompts twice for the password A: When attempting to change a local SSSD user's password, it may prompt for the password twice: This is the result of an incorrect PAM configuration. Ensure that the use_authtok option is correctly configured in your /etc/pam.d/system-auth file. For examples of the correct configuration, see Section 7.5.2, "Configuring Services: PAM" . Q: An Active Directory identity provider is properly configured in my sssd.conf file, but SSSD fails to connect to it, with GSS-API errors. A: SSSD can only connect with an Active Directory provider using its host name. If the host name is not given, the SSSD client cannot resolve the IP address to the host, and authentication fails. For example, with this configuration: The SSSD client returns this GSS-API failure, and the authentication request fails: To avoid this error, set the ad_server to the name of the Active Directory host, or use the _srv_ keyword to use the DNS service discovery, as described in Section 7.4.3, "Configuring DNS Service Discovery" . Q: I configured SSSD for central authentication, but now several of my applications (such as Firefox or Adobe) will not start. A: Even on 64-bit systems, 32-bit applications require a 32-bit version of SSSD client libraries to use to access the password and identity cache. If a 32-bit version of SSSD is not available, but the system is configured to use the SSSD cache, then 32-bit applications can fail to start. For example, Firefox can fail with permission denied errors: For Adobe Reader, the error shows that the current system user is not recognized: Other applications may show similar user or permissions errors. Q: SSSD is showing an automount location that I removed. A: The SSSD cache for the automount location persists even if the location is subsequently changed or removed. To update the autofs information in SSSD: Remove the autofs cache, as described in Section A.1.4, "A User Cannot Log In After UID or GID Changed" . Restart SSSD: A.1.4. A User Cannot Log In After UID or GID Changed After a user or group ID changed, SSSD prevents a user from logging in. What this means: SSSD recognizes users and groups based on user IDs (UID) and group IDs (GID). When the UID or GID of an existing user or group changes, SSSD does not recognize the user or group. To fix the problem: Clear the SSSD cache using the sss_cache utility: Make sure the sssd-tools is installed. To clear the SSSD cache for a specific user and leave the rest of the cache records intact: To clear the cache for an entire domain: The utility invalidates records in the SSSD cache for a user, group, or domain. After this, SSSD retrieves the records from the identity provider to refresh the cache. For details on sss_cache , see the sss_cache (8) man page. A.1.5. SSSD Control and Status Utility SSSD provides the sssctl utility to obtain status information, manage data files during troubleshooting, and other SSSD-related tasks. To use sssctl , install the sssd-tools package: Some options of sssctl use the SSSD InfoPipe responder. To enable it, add ifp to the services option of your /etc/sssd/sssd.conf : Restart SSSD: A.1.5.1. SSSD Configuration Validation The sssctl config-check command performs a static analysis of the SSSD configuration files. This enables you to validate the /etc/sssd/sssd.conf file and /etc/sssd/conf.d/* files to receive a report before restarting SSSD. The command performs the following checks on SSSD configuration files: Permissions The owner and group owner must be set to root:root and the permissions to 600 . File names File names in /etc/sssd/conf.d/ must use the suffix .conf and not start with a period (hidden files). Typographical errors Typographical errors are checked in section and option names. Note that values are not checked. Options Options must be placed in the correct sections. To verify the configuration, run: A.1.5.2. Displaying User Data The sssctl user-checks command helps to debug problems in applications that use SSSD as a back end for user lookup, authentication, and authorization. The command displays user data available through NSS and the InfoPipe responder for the D-Bus interface. The displayed data shows whether the user is authorized to log in using the system-auth PAM services. For example, to display the user data of the admin user: For further options, see the output of the sssctl user-checks --help command. A.1.5.3. Domain Information The domain status displays a list of domains SSSD accesses, and enables you to retrieve their status. List all domains available within SSSD, including trusted domains: Retrieve the status of the domain idm.example.com : A.1.5.4. Cached Entries Information The sssctl command enables you to retrieve information about a cached entry, to investigate and debug cache-related authentication problems. To query cache information for the user account admin , run: To query the cache information for a group or netgroup, use: A.1.5.5. Truncating the Log Files When you debug a problem, you can use sssctl logs-remove to truncate all SSSD log files in the /var/log/sssd/ directory to capture only newly created entries. A.1.5.6. Removing the SSSD Cache To remove the SSSD cache database files, the sssctl command provides the remove-cache option. Before the databases are removed, the command creates automatically a backup. Use the following command to back up all local data and remove the SSSD cache: Note The backup stores only local data, such as local overrides, in the /var/lib/sss/backup/ directory. To automatically import the backed-up content, run sssctl restore-local-data . A.1.5.7. Obtaining Information about an LDAP Group Takes Long Operations that involve looking up information about an LDAP group take very long, especially in case of large groups or nested groups. What this means: By default, LDAP group information lookups return all members for the group. For operations that involve large groups or nested groups, returning all members makes the process longer. To fix the problem: The membership lists returned in group lookups are not used when evaluating whether a user belongs to a group. To improve performance, especially for identity lookups, disable the group membership lookup: Open the /etc/sssd/sssd.conf file. In the [domain] section, set the ignore_group_members option to true . Note This option should not be set to true when the deployment involves an IdM server with the compat tree.
[domain/LDAP] cache_credentials = true debug_level = 9 sssd -d4 -i [sssd] [ldb] (3): server_sort:Unable to register control with rootdse! [sssd] [confdb_get_domains] (0): No domains configured, fatal error! [sssd] [get_monitor_config] (0): No domains configured. [sssd] [get_monitor_config] (0): No services configured! [sssd] [confdb_ldif_from_ini_file] (0x0020): Permission check on config file failed. [sssd] [confdb_init_db] (0x0020): Cannot convert INI to LDIF [1]: [Operation not permitted] [sssd] [confdb_setup] (0x0010): ConfDB initialization has failed [1]: Operation not permitted [sssd] [load_configuration] (0x0010): Unable to setup ConfDB [1]: Operation not permitted [sssd] [main] (0x0020): Cannot read config file /etc/sssd/sssd.conf. Please check that the file is accessible only by the owner and owned by root.root. chmod 600 /etc/sssd/sssd.conf chown root:root /etc/sssd/sssd.conf ldap_group_name = uniqueMember ldapsearch -x -ZZ -h test.example.com -b dc=example,dc=com ldap_start_tls: Connect error (-11) additional info: TLS error -8179:Unknown code ___f 13 ldap_tls_cacert = /path/to/cacert semanage port -a -t ldap_port_t -p tcp 1389 service sssd status Redirecting to /bin/systemctl status sssd.service sssd.service - System Security Services Daemon Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled) Active: active (running) since Wed 2015-01-14 10:17:26 CET; 1min 30s ago Process: 683 ExecStart=/usr/sbin/sssd -D -f (code=exited, status=0/SUCCESS) Main PID: 745 (sssd) CGroup: /system.slice/sssd.service ├─745 /usr/sbin/sssd -D -f ├─746 /usr/libexec/sssd/sssd_be --domain default --debug-to-files ├─804 /usr/libexec/sssd/sssd_nss --debug-to-files └─805 /usr/libexec/sssd/sssd_pam --debug-to-files passwd user1000 Changing password for user user1000. New password: Retype new password: New Password: Reenter new Password: passwd: all authentication tokens updated successfully. [domain/ADEXAMPLE] debug_level = 0xFFF0 id_provider = ad ad_server = 172.16.0.1 ad_domain = example.com krb5_canonicalize = False (Fri Jul 27 18:27:44 2012) [sssd[be[ADTEST]]] [sasl_bind_send] (0x0020): ldap_sasl_bind failed (-2)[Local error] (Fri Jul 27 18:27:44 2012) [sssd[be[ADTEST]]] [sasl_bind_send] (0x0080): Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot determine realm for numeric host address)] Failed to contact configuration server. See http://www.gnome.org/projects/gconf/ for information. (Details - 1: IOR file '/tmp/gconfd-somebody/lock/ior' not opened successfully, no gconfd located: Permission denied 2: IOR file '/tmp/gconfd-somebody/lock/ior' not opened successfully, no gconfd located: Permission denied) [jsmith@server ~]USD acroread (acroread:12739): GLib-WARNING **: getpwuid_r(): failed due to unknown user id (366) systemctl restart sssd sss_cache --user user_name sss_cache --domain domain_name yum install sssd-tools [sssd] services = nss, sudo, pam, ssh , ifp systemctl restart sssd.service sssctl config-check Issues identified by validators: 3 [rule/allowed_sections]: Section [paM] is not allowed. Check for typos. [rule/allowed_domain_options]: Attribute 'offline_timeoutX' is not allowed in section 'domain/idm.example.com'. Check for typos. [rule/allowed_sudo_options]: Attribute 'homedir_substring' is not allowed in section 'sudo'. Check for typos. Messages generated during configuration merging: 2 File /etc/sssd/conf.d/wrong-file-permissions.conf did not pass access check. Skipping. File configuration.conf.disabled did not match provided patterns. Skipping. Used configuration snippet files: 1 /etc/sssd/conf.d/sample.conf sssctl user-checks admin user: admin action: acct service: system-auth SSSD nss user lookup result: - user name: admin - user id: 1194200000 - group id: 1194200000 - gecos: Administrator - home directory: /home/admin - shell: /bin/bash SSSD InfoPipe user lookup result: - name: admin - uidNumber: 1194200000 - gidNumber: 1194200000 - gecos: Administrator - homeDirectory: /home/admin - loginShell: /bin/bash testing pam_acct_mgmt pam_acct_mgmt: Success PAM Environment: - no env - sssctl domain-list idm.example.com ad.example.com sssctl domain-status idm.example.com Online status: Online sssctl user-show admin Name: admin Cache entry creation date: 07/10/16 16:09:18 Cache entry last update time: 07/14/16 10:13:32 Cache entry expiration time: 07/14/16 11:43:32 Initgroups expiration time: Expired Cached in InfoPipe: No sssctl group-show groupname sssctl netgroup-show netgroupname sssctl logs-remove Truncating log files sssctl cache-remove SSSD must not be running. Stop SSSD now? (yes/no) [yes] Creating backup of local data Removing cache files SSSD needs to be running. Start SSSD now? (yes/no) [yes] [domain/ domain_name ] [... file truncated ...] ignore_group_members = true