Chapter 13. Troubleshooting authentication with SSSD in IdM
Chapter 13. Troubleshooting authentication with SSSD in IdM Authentication in an Identity Management (IdM) environment involves many components: On the IdM client: The SSSD service. The Name Services Switch (NSS). Pluggable Authentication Modules (PAM). On the IdM server: The SSSD service. The IdM Directory Server. The IdM Kerberos Key Distribution Center (KDC). If you are authenticating as an Active Directory (AD) user: The Directory Server on an AD Domain Controller. The Kerberos server on an AD Domain Controller. To authenticate users, you must be able to perform the following functions with the SSSD service: Retrieve user information from the authentication server. Prompt the user for their credentials, pass those credentials to the authentication server, and process the outcome. To learn more about how information flows between the SSSD service and servers that store user information, so you can troubleshoot failing authentication attempts in your environment, see the following: Data flow when retrieving IdM user information with SSSD Data flow when retrieving AD user information with SSSD Data flow when authenticating as a user with SSSD in IdM Narrowing the scope of authentication issues SSSD log files and logging levels Enabling detailed logging for SSSD in the sssd.conf file Enabling detailed logging for SSSD with the sssctl command Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM server Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM client Tracking client requests in the SSSD backend Tracking client requests using the log analyzer tool 13.1. Data flow when retrieving IdM user information with SSSD The following diagram is a simplification of the information flow between an IdM client and an IdM server during a request for IdM user information with the command getent passwd <idm_user_name> . The getent command triggers the getpwnam call from the libc library. The libc library references the /etc/nsswitch.conf configuration file to check which service is responsible for providing user information, and discovers the entry sss for the SSSD service. The libc library opens the nss_sss module. The nss_sss module checks the memory-mapped cache for the user information. If the data is present in the cache, the nss_sss module returns it. If the user information is not in the memory-mapped cache, the request is passed to the SSSD sssd_nss responder process. The SSSD service checks its cache. If the data is present in the cache and valid, the sssd_nss responder reads the data from the cache and returns it to the application. If the data is not present in the cache or it is expired, the sssd_nss responder queries the appropriate back-end process and waits for a reply. The SSSD service uses the IPA backend in an IdM environment, enabled by the setting id_provider=ipa in the sssd.conf configuration file. The sssd_be back-end process connects to the IdM server and requests the information from the IdM LDAP Directory Server. The SSSD back-end on the IdM server responds to the SSSD back-end process on the IdM client. The SSSD back-end on the client stores the resulting data in the SSSD cache and alerts the responder process that the cache has been updated. The sssd_nss front-end responder process retrieves the information from the SSSD cache. The sssd_nss responder sends the user information to the nss_sss responder, completing the request. The libc library returns the user information to the application that requested it. 13.2. Data flow when retrieving AD user information with SSSD If you have established a cross-forest trust between your IdM environment and an Active Directory (AD) domain, the information flow when retrieving AD user information about an IdM client is very similar to the information flow when retrieving IdM user information, with the additional step of contacting the AD user database. The following diagram is a simplification of the information flow when a user requests information about an AD user with the command getent passwd <ad_user_name@ad.example.com> . This diagram does not include the internal details discussed in the Data flow when retrieving IdM user information with SSSD section. It focuses on the communication between the SSSD service on an IdM client, the SSSD service on an IdM server, and the LDAP database on an AD Domain Controller. The IdM client looks to its local SSSD cache for AD user information. If the IdM client does not have the user information, or the information is stale, the SSSD service on the client contacts the extdom_extop plugin on the IdM server to perform an LDAP extended operation and requests the information. The SSSD service on the IdM server looks for the AD user information in its local cache. If the IdM server does not have the user information in its SSSD cache, or its information is stale, it performs an LDAP search to request the user information from an AD Domain Controller. The SSSD service on the IdM server receives the AD user information from the AD domain controller and stores it in its cache. The extdom_extop plugin receives the information from the SSSD service on the IdM server, which completes the LDAP extended operation. The SSSD service on the IdM client receives the AD user information from the LDAP extended operation. The IdM client stores the AD user information in its SSSD cache and returns the information to the application that requested it. 13.3. Data flow when authenticating as a user with SSSD in IdM Authenticating as a user on an IdM server or client involves the following components: The service that initiates the authentication request, such as the sshd service. The Pluggable Authentication Module (PAM) library and its modules. The SSSD service, its responders, and back-ends. A smart card reader, if smart card authentication is configured. The authentication server: IdM users are authenticated against an IdM Kerberos Key Distribution Center (KDC). Active Directory (AD) users are authenticated against an AD Domain Controller (DC). The following diagram is a simplification of the information flow when a user needs to authenticate during an attempt to log in locally to a host via the SSH service on the command line. The authentication attempt with the ssh command triggers the libpam library. The libpam library references the PAM file in the /etc/pam.d/ directory that corresponds to the service requesting the authentication attempt. In this example involving authenticating via the SSH service on the local host, the libpam library checks the /etc/pam.d/system-auth configuration file and discovers the pam_sss.so entry for the SSSD PAM: To determine which authentication methods are available, the libpam library opens the pam_sss module and sends an SSS_PAM_PREAUTH request to the sssd_pam PAM responder of the SSSD service. If smart card authentication is configured, the SSSD service spawns a temporary p11_child process to check for a smart card and retrieve certificates from it. If smart card authentication is configured for the user, the sssd_pam responder attempts to match the certificate from the smart card with the user. The sssd_pam responder also performs a search for the groups that the user belongs to, since group membership might affect access control. The sssd_pam responder sends an SSS_PAM_PREAUTH request to the sssd_be back-end responder to see which authentication methods the server supports, such as passwords or 2-factor authentication. In an IdM environment, where the SSSD service uses the IPA responder, the default authentication method is Kerberos. For this example, the user authenticates with a simple Kerberos password. The sssd_be responder spawns a temporary krb5_child process. The krb5_child process contacts the KDC on the IdM server and checks for available authentication methods. The KDC responds to the request: The krb5_child process evaluates the reply and sends the results back to the sssd_be backend process. The sssd_be backend process receives the result. The sssd_pam responder receives the result. The pam_sss module receives the result. If password authentication is configured for the user, the pam_sss module prompts the user for their password. If smart card authentication is configured, the pam_sss module prompts the user for their smart card PIN. The module sends an SSS_PAM_AUTHENTICATE request with the user name and password, which travels to: The sssd_pam responder. The sssd_be back-end process. The sssd_be process spawns a temporary krb5_child process to contact the KDC. The krb5_child process attempts to retrieve a Kerberos Ticket Granting Ticket (TGT) from the KDC with the user name and password the user provided. The krb5_child process receives the result of the authentication attempt. The krb5_child process: Stores the TGT in a credential cache. Returns the authentication result to the sssd_be back-end process. The authentication result travels from the sssd_be process to: The sssd_pam responder. The pam_sss module. The pam_sss module sets an environment variable with the location of the user's TGT so other applications can reference it. 13.4. Narrowing the scope of authentication issues To successfully authenticate a user, you must be able to retrieve user information with the SSSD service from the database that stores user information. The following procedure describes steps to test different components of the authentication process so you can narrow the scope of authentication issues when a user is unable to log in. Procedure Verify that the SSSD service and its processes are running. Verify that the client can contact the user database server via the IP address. If this step fails, check that your network and firewall settings allow direct communication between IdM clients and servers. See Using and configuring firewalld . Verify that the client can discover and contact the IdM LDAP server (for IdM users) or AD domain controller (for AD users) via the fully qualified host name. If this step fails, check your Dynamic Name Service (DNS) settings, including the /etc/resolv.conf file. See Configuring the order of DNS servers . Note By default, the SSSD service attempts to automatically discover LDAP servers and AD DCs through DNS service (SRV) records. Alternatively, you can restrict the SSSD service to use specific servers by setting the following options in the sssd.conf configuration file: ipa_server = <fully_qualified_host_name_of_the_server> ad_server = <fully_qualified_host_name_of_the_server> ldap_uri = <fully_qualified_host_name_of_the_server> If you use these options, verify you can contact the servers listed in them. Verify that the client can authenticate to the LDAP server and retrieve user information with ldapsearch commands. If your LDAP server is an IdM server, like server.example.com , retrieve a Kerberos ticket for the host and perform the database search authenticating with the host Kerberos principal: If your LDAP server is an Active Directory (AD) Domain Controller (DC), like server.ad.example.com , retrieve a Kerberos ticket for the host and perform the database search authenticating with the host Kerberos principal: If your LDAP server is a plain LDAP server, and you have set the ldap_default_bind_dn and ldap_default_authtok options in the sssd.conf file, authenticate as the same ldap_default_bind_dn account: If this step fails, verify that your database settings allow your host to search the LDAP server. Since the SSSD service uses Kerberos encryption, verify you can obtain a Kerberos ticket as the user that is unable to log in. If your LDAP server is an IdM server: If LDAP server database is an AD server: If this step fails, verify that your Kerberos server is operating properly, all servers have their times synchronized, and that the user account is not locked. Verify you can retrieve user information about the command line. If this step fails, verify that the SSSD service on the client can receive information from the user database: Review errors in the /var/log/messages log file. Enable detailed logging in the SSSD service, collect debugging logs, and review the logs for indications to the source of the issue. Optional: Open a Red Hat Technical Support case and provide the troubleshooting information you have gathered. If you are allowed to run sudo on the host, use the sssctl utility to verify the user is allowed to log in. If this step fails, verify your authorization settings, such as your PAM configuration, IdM HBAC rules, and IdM RBAC rules: Ensure that the user's UID is equal to or higher than UID_MIN , which is defined in the /etc/login.defs file. Review authorization errors in the /var/log/secure and /var/log/messages log files. Enable detailed logging in the SSSD service, collect debugging logs, and review the logs for indications to the source of the issue. Optional: Open a Red Hat Technical Support case and provide the troubleshooting information you have gathered. Additional resources Enabling detailed logging for SSSD in the sssd.conf file Enabling detailed logging for SSSD with the sssctl command Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM server Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM client 13.5. SSSD log files and logging levels Each SSSD service logs into its own log file in the /var/log/sssd/ directory. For an IdM server in the example.com IdM domain, its log files might look like this: 13.5.1. SSSD log file purposes krb5_child.log Log file for the short-lived helper process involved in Kerberos authentication. ldap_child.log Log file for the short-lived helper process involved in getting a Kerberos ticket for the communication with the LDAP server. sssd_<example.com>.log For each domain section in the sssd.conf file, the SSSD service logs information about communication with the LDAP server to a separate log file. For example, in an environment with an IdM domain named example.com , the SSSD service logs its information in a file named sssd_example.com.log . If a host is directly integrated with an AD domain named ad.example.com , information is logged to a file named sssd_ad.example.com.log . Note If you have an IdM environment and a cross-forest trust with an AD domain, information about the AD domain is still logged to the log file for the IdM domain. Similarly, if a host is directly integrated to an AD domain, information about any child domains is written in the log file for the primary domain. selinux_child.log Log file for the short-lived helper process that retrieves and sets SELinux information. sssd.log Log file for SSSD monitoring and communicating with its responder and backend processes. sssd_ifp.log Log file for the InfoPipe responder, which provides a public D-Bus interface accessible over the system bus. sssd_nss.log Log file for the Name Services Switch (NSS) responder that retrieves user and group information. sssd_pac.log Log file for the Microsoft Privilege Attribute Certificate (PAC) responder, which collects the PAC from AD Kerberos tickets and derives information about AD users from the PAC, which avoids requesting it directly from AD. sssd_pam.log Log file for the Pluggable Authentication Module (PAM) responder. sssd_ssh.log Log file for the SSH responder process. 13.5.2. SSSD logging levels Setting a debug level also enables all debug levels below it. For example, setting the debug level at 6 also enables debug levels 0 through 5. Table 13.1. SSSD logging levels Level Description 0 Fatal failures . Errors that prevent the SSSD service from starting up or cause it to terminate. 1 Critical failures . Errors that do not terminate the SSSD service, but at least one major feature is not working properly. 2 Serious failures . Errors announcing that a particular request or operation has failed. This is the default debug log level. 3 Minor failures . Errors that cause the operation failures captured at level 2. 4 Configuration settings. 5 Function data. 6 Trace messages for operation functions. 7 Trace messages for internal control functions. 8 Contents of function-internal variables. 9 Extremely low-level tracing information. 13.6. Enabling detailed logging for SSSD in the sssd.conf file By default, the SSSD service only logs serious failures (debug level 2), but it does not log at the level of detail necessary to troubleshoot authentication issues. To enable detailed logging persistently across SSSD service restarts, add the option debug_level= <integer> in each section of the /etc/sssd/sssd.conf configuration file, where the <integer> value is a number between 0 and 9. Debug levels up to 3 log larger failures, and levels 8 and higher provide a large number of detailed log messages. Level 6 is a good starting point for debugging authentication issues. Prerequisites You need the root password to edit the sssd.conf configuration file and restart the SSSD service. Procedure Open the /etc/sssd/sssd.conf file in a text editor. Add the debug_level option to every section of the file, and set the debug level to the verbosity of your choice. Save and close the sssd.conf file. Restart the SSSD service to load the new configuration settings. Additional resources SSSD log files and logging levels 13.7. Enabling detailed logging for SSSD with the sssctl command By default, the SSSD service only logs serious failures (debug level 2), but it does not log at the level of detail necessary to troubleshoot authentication issues. You can change the debug level of the SSSD service on the command line with the sssctl debug-level <integer> command, where the <integer> value is a number between 0 and 9. Debug levels up to 3 log larger failures, and levels 8 and higher provide a large number of detailed log messages. Level 6 is a good starting point for debugging authentication issues. Prerequisites You need the root password to run the sssctl command. Procedure Use the sssctl debug-level command to set the debug level of your choiceto your desired verbosity. Additional resources SSSD log files and logging levels 13.8. Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM server If you experience issues when attempting to authenticate as an IdM user to an IdM server, enable detailed debug logging in the SSSD service on the server and gather logs of an attempt to retrieve information about the user. Prerequisites You need the root password to run the sssctl command and restart the SSSD service. Procedure Enable detailed SSSD debug logging on the IdM server. Invalidate objects in the SSSD cache for the user that is experiencing authentication issues, so you do not bypass the LDAP server and retrieve information SSSD has already cached. Minimize the troubleshooting dataset by removing older SSSD logs. Attempt to switch to the user experiencing authentication problems, while gathering timestamps before and after the attempt. These timestamps further narrow the scope of the dataset. Optional: Lower the debug level if you do not wish to continue gathering detailed SSSD logs. Review SSSD logs for information about the failed request. For example, reviewing the /var/log/sssd/sssd_example.com.log file shows that the SSSD service did not find the user in the cn=accounts,dc=example,dc=com LDAP subtree. This might indicate that the user does not exist, or exists in another location. If you are unable to determine the cause of the authentication issue: Collect the SSSD logs you recently generated. Open a Red Hat Technical Support case and provide: The SSSD logs: sssd-logs-Mar29.tar The console output, including the time stamps and user name, of the request that corresponds to the logs: 13.9. Gathering debugging logs from the SSSD service to troubleshoot authentication issues with an IdM client If you experience issues when attempting to authenticate as an IdM user to an IdM client, verify that you can retrieve user information about the IdM server. If you cannot retrieve the user information about an IdM server, you will not be able to retrieve it on an IdM client (which retrieves information from the IdM server). After you have confirmed that authentication issues do not originate from the IdM server, gather SSSD debugging logs from both the IdM server and IdM client. Prerequisites The user only has authentication issues on IdM clients, not IdM servers. You need the root password to run the sssctl command and restart the SSSD service. Procedure On the client: Open the /etc/sssd/sssd.conf file in a text editor. On the client: Add the ipa_server option to the [domain] section of the file and set it to an IdM server. This avoids the IdM client autodiscovering other IdM servers, thus limiting this test to just one client and one server. On the client: Save and close the sssd.conf file. On the client: Restart the SSSD service to load the configuration changes. On the server and client: Enable detailed SSSD debug logging. On the server and client: Invalidate objects in the SSSD cache for the user experiencing authentication issues, so you do not bypass the LDAP database and retrieve information SSSD has already cached. On the server and client: Minimize the troubleshooting dataset by removing older SSSD logs. On the client: Attempt to switch to the user experiencing authentication problems while gathering timestamps before and after the attempt. These timestamps further narrow the scope of the dataset. Optional: On the server and client: Lower the debug level if you do not wish to continue gathering detailed SSSD logs. On the server and client: Review SSSD logs for information about the failed request. Review the request from the client in the client logs. Review the request from the client in the server logs. Review the result of the request in the server logs. Review the outcome of the client receiving the results of the request from the server. If you are unable to determine the cause of the authentication issue: Collect the SSSD logs you recently generated on the IdM server and IdM client. Label them according to their hostname or role. Open a Red Hat Technical Support case and provide: The SSSD debug logs: sssd-logs-server-Mar29.tar from the server sssd-logs-client-Mar29.tar from the client The console output, including the time stamps and user name, of the request that corresponds to the logs: 13.10. Tracking client requests in the SSSD backend SSSD processes requests asynchronously and as messages from different requests are added to the same log file, you can use the unique request identifier and client ID to track client requests in the back-end logs. The unique request identifier is added to the debug logs in the form of RID#<integer> and the client ID in the form [CID #<integer] . This allows you to isolate logs pertaining to an individual request, and you can track requests from start to finish across log files from multiple SSSD components. Prerequisites You have enabled debug logging and a request has been submitted from an IdM client. You must have root privileges to display the contents of the SSSD log files. Procedure To review your SSSD log file, open the log file using the less utility. For example, to view the /var/log/sssd/sssd_example.com.log : Review the SSSD logs for information about the client request. This sample output from an SSSD log file shows the unique identifiers RID#3 and RID#4 for two different requests. However, a single client request to the SSSD client interface often triggers multiple requests in the backend and as a result it is not a 1-to-1 correlation between client request and requests in the backend. Though the multiple requests in the backend have different RID numbers, each initial backend request includes the unique client ID so an administrator can track the multiple RID numbers to the single client request. The following example shows one client request [sssd.nss CID #1] and the multiple requests generated in the backend, [RID#5] to [RID#13] : 13.11. Tracking client requests using the log analyzer tool The System Security Services Daemon (SSSD) includes a log parsing tool that can be used to track requests from start to finish across log files from multiple SSSD components. 13.11.1. How the log analyzer tool works Using the log parsing tool, you can track SSSD requests from start to finish across log files from multiple SSSD components. You run the analyzer tool using the sssctl analyze command. The log analyzer tool helps you to troubleshoot NSS and PAM issues in SSSD and more easily review SSSD debug logs. You can extract and print SSSD logs related only to certain client requests across SSSD processes. SSSD tracks user and group identity information ( id , getent ) separately from user authentication ( su , ssh ) information. The client ID (CID) in the NSS responder is independent of the CID in the PAM responder and you see overlapping numbers when analyzing NSS and PAM requests. Use the --pam option with the sssctl analyze command to review PAM requests. Note Requests returned from the SSSD memory cache are not logged and cannot be tracked by the log analyzer tool. Additional resources sudo sssctl analyze request --help sudo sssctl analyze --help sssd.conf and sssctl man pages on your system 13.11.2. Running the log analyzer tool Follow this procedure to use the log analyzer tool to track client requests in SSSD. Prerequisites You must set debug_level to at least 7 in the [USDresponder] section, and [domain/USDdomain] section of the /etc/sssd/sssd.conf file to enable log parsing functionality. Logs to analyze must be from a compatible version of SSSD built with libtevent chain ID support, that is SSSD in RHEL 8.5 and later. Procedure Run the log analyzer tool in list mode to determine the client ID of the request you are tracking, adding the -v option to display verbose output: A verbose list of recent client requests made to SSSD is displayed. Note If analyzing PAM requests, run the sssctl analyze request list command with the --pam option. Run the log analyzer tool with the show [unique client ID] option to display logs pertaining to the specified client ID number: If required, you can run the log analyzer tool against log files, for example: Additional resources sssctl analyze request list --help sssctl analyze request show --help sssctl man page on your system 13.12. Additional resources General SSSD Debugging Procedures (Red Hat Knowledgebase)
auth sufficient pam_sss.so pstree -a | grep sssd |-sssd -i --logger=files | |-sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files | |- sssd_be --domain example.com --uid 0 --gid 0 --logger=files | |-sssd_ifp --uid 0 --gid 0 --logger=files | |- sssd_nss --uid 0 --gid 0 --logger=files | |-sssd_pac --uid 0 --gid 0 --logger=files | |- sssd_pam --uid 0 --gid 0 --logger=files | |- sssd_ssh --uid 0 --gid 0 --logger=files | `-sssd_sudo --uid 0 --gid 0 --logger=files |-sssd_kcm --uid 0 --gid 0 --logger=files [user@client ~]USD ping <IP_address_of_the_database_server> [user@client ~]USD dig -t SRV _ldap._tcp.example.com @<name_server> [user@client ~]USD ping <fully_qualified_host_name_of_the_server> [user@client ~]USD kinit -k 'host/client.example.com@EXAMPLE.COM' [user@client ~]USD ldapsearch -LLL -Y GSSAPI -h server.example.com -b "dc=example,dc=com" uid= <user_name> [user@client ~]USD kinit -k 'CLIENTUSD@AD.EXAMPLE.COM' [user@client ~]USD ldapsearch -LLL -Y GSSAPI -h server.ad.example.com -b "dc=example,dc=com" sAMAccountname= <user_name> [user@client ~]USD ldapsearch -xLLL -D "cn=ldap_default_bind_dn_value" -W -h ldapserver.example.com -b "dc=example,dc=com" uid= <user_name> [user@client ~]USD kinit <user_name> [user@client ~]USD kinit <user_name@AD.EXAMPLE.COM> [user@client ~]USD getent passwd <user_name> [user@client ~]USD id <user_name> [user@client ~]USD sudo sssctl user-checks -a auth -s ssh <user_name> ls -l /var/log/sssd/ total 620 -rw-------. 1 root root 0 Mar 29 09:21 krb5_child.log -rw-------. 1 root root 14324 Mar 29 09:50 ldap_child.log -rw-------. 1 root root 212870 Mar 29 09:50 sssd_example.com.log -rw-------. 1 root root 0 Mar 29 09:21 sssd_ifp.log -rw-------. 1 root root 0 Mar 29 09:21 sssd_implicit_files.log -rw-------. 1 root root 0 Mar 29 09:21 sssd.log -rw-------. 1 root root 219873 Mar 29 10:03 sssd_nss.log -rw-------. 1 root root 0 Mar 29 09:21 sssd_pac.log -rw-------. 1 root root 13105 Mar 29 09:21 sssd_pam.log -rw-------. 1 root root 9390 Mar 29 09:21 sssd_ssh.log -rw-------. 1 root root 0 Mar 29 09:21 sssd_sudo.log [domain/example.com] debug_level = 6 id_provider = ipa [sssd] debug_level = 6 services = nss, pam, ifp, ssh, sudo domains = example.com [nss] debug_level = 6 [pam] debug_level = 6 [sudo] debug_level = 6 [ssh] debug_level = 6 [pac] debug_level = 6 [ifp] debug_level = 6 systemctl restart sssd sssctl debug-level 6 sssctl debug-level 6 sssctl cache-expire -u idmuser sssctl logs-remove date; su idmuser ; date Mon Mar 29 15:33:48 EDT 2021 su: user idmuser does not exist Mon Mar 29 15:33:49 EDT 2021 sssctl debug-level 2 (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [dp_get_account_info_send] (0x0200): Got request for [0x1][BE_REQ_USER][ name=idmuser@example.com ] (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(uid=idmuser)(objectclass=posixAccount)(uid= )(&(uidNumber= )(!(uidNumber=0))))][cn=accounts,dc=example,dc=com]. (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results. (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_search_by_name] (0x0400): No such entry (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory) (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_search_by_name] (0x0400): No such entry (Mon Mar 29 15:33:49 2021) [sssd[be[example.com]]] [ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending request sssctl logs-fetch sssd-logs-Mar29.tar date; id idmuser; date Mon Mar 29 15:33:48 EDT 2021 id: 'idmuser': no such user Mon Mar 29 15:33:49 EDT 2021 [domain/example.com] ipa_server = server.example.com systemctl restart sssd sssctl debug-level 6 sssctl debug-level 6 sssctl cache-expire -u idmuser sssctl cache-expire -u idmuser sssctl logs-remove sssctl logs-remove date; su idmuser; date Mon Mar 29 16:20:13 EDT 2021 su: user idmuser does not exist Mon Mar 29 16:20:14 EDT 2021 sssctl debug-level 0 sssctl debug-level 0 sssctl logs-fetch sssd-logs-server-Mar29.tar sssctl logs-fetch sssd-logs-client-Mar29.tar date; su idmuser ; date Mon Mar 29 16:20:13 EDT 2021 su: user idmuser does not exist Mon Mar 29 16:20:14 EDT 2021 less /var/log/sssd/sssd_example.com.log (2021-07-26 18:26:37): [be[testidm.com]] [dp_req_destructor] (0x0400): [RID#3] Number of active DP request: 0 (2021-07-26 18:26:37): [be[testidm.com]] [dp_req_reply_std] (0x1000): [RID#3] DP Request AccountDomain #3: Returning [Internal Error]: 3,1432158301,GetAccountDomain() not supported (2021-07-26 18:26:37): [be[testidm.com]] [dp_attach_req] (0x0400): [RID#4] DP Request Account #4: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-07-26 18:26:37): [be[testidm.com]] [dp_attach_req] (0x0400): [RID#4] Number of active DP request: 1 (2021-10-29 13:24:16): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#5] DP Request [Account #5]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:16): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#6] DP Request [AccountDomain #6]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:16): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#7] DP Request [Account #7]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#8] DP Request [Initgroups #8]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#9] DP Request [Account #9]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#10] DP Request [Account #10]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#11] DP Request [Account #11]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#12] DP Request [Account #12]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. (2021-10-29 13:24:17): [be[ad.vm]] [dp_attach_req] (0x0400): [RID#13] DP Request [Account #13]: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001]. sssctl analyze request list -v sssctl analyze request show 20 sssctl analyze request --logdir=/tmp/var/log/sssd