Chapter 9. Tuning SSSD performance for large IdM-AD trust deployments
Chapter 9. Tuning SSSD performance for large IdM-AD trust deployments Retrieving user and group information is a very data-intensive operation for the System Security Services Daemon (SSSD), especially in an IdM deployment with a trust to a large Active Directory (AD) domain. You can improve this performance by adjusting which information SSSD retrieves from identity providers and for how long. 9.1. Tuning SSSD in IdM servers for large IdM-AD trust deployments This procedure applies tuning options to the configuration of the SSSD service in an IdM server to improve its response time when retrieving information from a large AD environment. Prerequisites You need root permissions to edit the /etc/sssd/sssd.conf configuration file. Procedure Open the /etc/sssd/sssd.conf configuration file in a text editor. Add the following options to the [domain] section for your Identity Management (IdM) domain: Note Settings listed in the subdomain_inherit options apply to both the main (IdM) domain as well as the trusted AD domain(s). Save and close the /etc/sssd/sssd.conf file on the server. Restart the SSSD service to load the configuration changes. Additional resources Options for tuning SSSD in IdM servers and clients for large IdM-AD trust deployments 9.2. Tuning the config timeout for the ipa-extdom plugin on IdM servers IdM clients cannot receive information about users and groups from Active Directory (AD) directly, so IdM servers use the ipa-extdom plugin to receive information about AD users and groups, and that information is forwarded to the requesting client. The ipa-extdom plug-in sends a request to SSSD for the data about AD users. If the information is not in the SSSD cache, SSSD requests the data from an AD domain controller (DC). You can adjust the config timeout value, which defines how long the ipa-extdom plug-in waits for a reply from SSSD before the plug-in cancels the connection and returns a timeout error to the caller. The default value is 10000 milliseconds (10 seconds). The following example adjusts the config timeout to 20 seconds (20000 milliseconds). Warning Exercise caution when adjusting the config timeout: If you set a value that is too small, such as 500 milliseconds, SSSD might not have enough time to reply and requests will always return a timeout. If you set a value that is too large, such as 30000 milliseconds (30 seconds), a single request might block the connection to SSSD for this amount of time. Because only one thread can connect to SSSD at a time, all other requests from the plug-in have to wait. If there are many requests sent by IdM clients, they can block all available workers configured for the Directory Server on the IdM server. As a consequence, the server might not be able to reply to any kind of request for some time. Only change the config timeout in the following situations: If IdM clients frequently receive timeout errors before their own search timeout is reached when requesting information about AD users and groups, the config timeout value is too small . If the Directory Server on the IdM server is often locked and the pstack utility reports that many or all worker threads are handling ipa-extdom requests at this time, the value is too large . Prerequisites The LDAP Directory Manager password Procedure Use the following command to adjust the config timeout to 20000 milliseconds: 9.3. Tuning the maximum buffer size for the ipa-extdom plugin on IdM servers IdM clients cannot receive information about users and groups from Active Directory (AD) directly, so IdM servers use the ipa-extdom plugin to receive information about AD users and groups, and that information is forwarded to the requesting client. You can tune the maximum buffer size for the ipa-extdom plugin, which adjusts the size of the buffer where SSSD can store the data it receives. If the buffer is too small, SSSD returns an ERANGE error and the plug-in retries the request with a larger buffer. The default buffer size is 134217728 bytes (128 MB). The following example adjusts the maximum buffer size to 256 MB (268435456 bytes). Prerequisites The LDAP Directory Manager password Procedure Use the following command to set the maximum buffer size to 268435456 bytes: 9.4. Tuning the maximum number of instances for the ipa-extdom plugin on IdM servers As IdM clients cannot receive information about users and groups from Active Directory (AD) directly, IdM servers use the ipa-extdom plugin to receive information about AD users and groups and then forward this information to the requesting client. By default, the ipa-extdom plugin is configured to use up to 80% of the LDAP worker threads to handle requests from IdM clients. If the SSSD service on an IdM client has requested a large amount of information about AD trust users and groups, this operation can halt the LDAP service if it uses most of the LDAP threads. If you experience these issues, you might see similar errors in the SSSD log file for your AD domain, /var/log/sssd/sssd__your-ad-domain-name.com_.log : You can adjust the maximum number of ipa-extdom instances by setting the value for the ipaExtdomMaxInstances option, which must be an integer larger than 0 and less than the total number of worker threads. Prerequisites The LDAP Directory Manager password Procedure Retrieve the total number of worker threads: This means that the current value for ipaExtdomMaxInstances is 13. Adjust the maximum number of instances. This example changes the value to 14: Retrieve the current value of ipaExtdomMaxInstances : Monitor the IdM directory server's performance and if it does not improve, repeat this procedure and adjust the value of the ipaExtdomMaxInstances variable. 9.5. Tuning SSSD in IdM clients for large IdM-AD trust deployments This procedure applies tuning options to SSSD service configuration in an IdM client to improve its response time when retrieving information from a large AD environment. Prerequisites You need root permissions to edit the /etc/sssd/sssd.conf configuration file. Procedure Determine the number of seconds a single un-cached login takes. Clear the SSSD cache on the IdM client client.example.com . Measure how long it takes to log in as an AD user with the time command. In this example, from the IdM client client.example.com , log into the same host as the user ad-user from the ad.example.com AD domain. Type in the password as soon as possible. Log out as soon as possible to display elapsed time. In this example, a single un-cached login takes about 9 seconds. Open the /etc/sssd/sssd.conf configuration file in a text editor. Add the following options to the [domain] section for your Active Directory domain. Set the pam_id_timeout and krb5_auth_timeout options to the number of seconds an un-cached login takes. If you do not already have a domain section for your AD domain, create one. Add the following option to the [pam] section: Save and close the /etc/sssd/sssd.conf file on the server. Restart the SSSD service to load the configuration changes. Additional resources Options for tuning SSSD in IdM servers and clients for large IdM-AD trust deployments 9.6. Mounting the SSSD cache in tmpfs The System Security Services Daemon (SSSD) constantly writes LDAP objects to its cache. These internal SSSD transactions write data to disk, which is much slower than reading and writing from Random-Access Memory (RAM). To improve this performance, mount the SSSD cache in RAM. Considerations Cached information does not persist after a reboot if the SSSD cache is in RAM. It is safe to perform this change on IdM servers, as the SSSD instance on an IdM server cannot lose connectivity with the Directory Server on the same host. If you perform this adjustment on an IdM client and it loses connectivity to IdM servers, users will not be able to authenticate after a reboot until you reestablish connectivity. Prerequisites You need root permissions to edit the /etc/fstab configuration file. Procedure Create a tmpfs temporary filesystem: On RHEL 8.6 and later, confirm that the SSSD user owns the config.ldb file: Add the following entry to the /etc/fstab file as a single line: On RHEL 8 versions lower than 8.6, the config.ldb file is owned by the root user: Add the following entry to the /etc/fstab file as a single line: This example creates a 300MB cache. Tune the size parameter according to your IdM and AD directory size, estimating 100 MBs per 10,000 LDAP entries. Mount the new SSSD cache directory. Restart SSSD to reflect this configuration change. 9.7. Options in sssd.conf for tuning IdM servers and clients for large IdM-AD trust deployments You can use the following options in the /etc/sssd/sssd.conf configuration file to tune the performance of SSSD in IdM servers and clients when you have a large IdM-AD trust deployment. 9.7.1. Tuning options for IdM servers ignore_group_members Knowing which groups a user belongs to, as opposed to all the users that belong to a group, is important when authenticating and authorizing a user. When ignore_group_members is set to true , SSSD only retrieves information about the group objects themselves and not their members, providing a significant performance boost. Note The id user@ad-domain.com command still returns the correct list of groups, but getent group ad-group@ad-domain.com returns an empty list. Default value false Recommended value true Note You should not set this option to true when the deployment involves an IdM server with the compat tree. subdomain_inherit With the subdomain_inherit option, you can apply the ignore_group_members setting to the trusted AD domains' configuration. Settings listed in the subdomain_inherit options apply to both the main (IdM) domain as well as the AD subdomain. Default value none Recommended value subdomain_inherit = ignore_group_members 9.7.2. Tuning options for IdM clients pam_id_timeout This parameter controls how long results from a PAM session are cached, to avoid excessive round-trips to the identity provider during an identity lookup. The default value of 5 seconds might not be enough in environments where complex group memberships are populated on the IdM Server and IdM client side. Red Hat recommends setting pam_id_timeout to the number of seconds a single un-cached login takes. Default value 5 Recommended value the number of seconds a single un-cached login takes krb5_auth_timeout Increasing krb5_auth_timeout allows more time to process complex group information in environments where users are members of a large number of groups. Red Hat recommends setting this value to the number of seconds a single un-cached login takes. Default value 6 Recommended value the number of seconds a single un-cached login takes ldap_deref_threshold A dereference lookup is a means of fetching all group members in a single LDAP call. The ldap_deref_threshold value specifies the number of group members that must be missing from the internal cache to trigger a dereference lookup. If less members are missing, they are looked up individually. Dereference lookups may take a long time in large environments and decrease performance. To disable dereference lookups, set this option to 0 . Default value 10 Recommended value 0 9.8. Additional resources Performance tuning SSSD for large IdM-AD trust deployments
[domain/ idm.example.com ] ignore_group_members = true subdomain_inherit = ignore_group_members systemctl restart sssd ldapmodify -D "cn=directory manager" -W dn: cn=ipa_extdom_extop,cn=plugins,cn=config changetype: modify replace: ipaExtdomMaxNssTimeout ipaExtdomMaxNssTimeout: 20000 ldapmodify -D "cn=directory manager" -W dn: cn=ipa_extdom_extop,cn=plugins,cn=config changetype: modify replace: ipaExtdomMaxNssBufSize ipaExtdomMaxNssBufSize: 268435456 (2022-05-22 5:00:13): [be[ad.example.com]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed. (2022-05-22 5:00:13): [be[ad.example.com]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed. (2022-05-22 5:00:13): [be[ad.example.com]] [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Server is busy(51), Too many extdom instances running. ldapsearch -xLLLD cn=directory\ manager -W -b cn=config -s base nsslapd-threadnumber Enter LDAP Password: dn: cn=config nsslapd-threadnumber: 16 ldapmodify -D "cn=directory manager" -W dn: cn=ipa_extdom_extop,cn=plugins,cn=config changetype: modify replace: ipaExtdomMaxInstances ipaExtdomMaxInstances: 14 ldapsearch -xLLLD "cn=directory manager" -W -b "cn=ipa_extdom_extop,cn=plugins,cn=config" |grep ipaextdommaxinstances Enter LDAP Password: ipaextdommaxinstances: 14 sss_cache -E time ssh ad-user @ ad.example.com @client.example.com Password: Last login: Sat Jan 23 06:29:54 2021 from 10.0.2.15 [ad-user@ad.example.com@client ~]USD [ad-user@ad.example.com@client /]USD exit logout Connection to client.example.com closed. real 0m8.755s user 0m0.017s sys 0m0.013s [domain/ example.com / ad.example.com ] krb5_auth_timeout = 9 ldap_deref_threshold = 0 [pam] pam_id_timeout = 9 systemctl restart sssd ls -al /var/lib/sss/db/config.ldb -rw-------. 1 sssd sssd 1286144 Jun 8 16:41 /var/lib/sss/db/config.ldb tmpfs /var/lib/sss/db/ tmpfs size= 300M ,mode=0700, uid=sssd,gid=sssd ,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0 ls -al /var/lib/sss/db/config.ldb -rw-------. 1 root root 1286144 Jun 8 14:15 /var/lib/sss/db/config.ldb tmpfs /var/lib/sss/db/ tmpfs size= 300M ,mode=0700,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0 mount /var/lib/sss/db/ systemctl restart sssd