Chapter 3. Setting an index scan limit to improve the performance when loading long lists of IDs
Chapter 3. Setting an index scan limit to improve the performance when loading long lists of IDs In large directories, the search results list can be huge. For example, a directory with one million entries with inetorgperson attributes would return all these entries in a search with a filter, such as (objectclass=inetorgperson) . Loading a long ID list from the database significantly reduces search performance. An ID list scan limit sets a limit on the number of IDs Directory Server reads before a key is considered to match the entire primary index. This means that Directory Server treats the search as an unindexed search with a different set of resource limits. For large indexes, it is actually more efficient to treat any search which matches the index as an unindexed search. The search operation only has to look in one place, the entire directory, to process results rather than searching through an index that is nearly the size of a directory, plus the directory itself. You can set an index scan limit globally or for specific databases. 3.1. Setting a global index scan limit using the command line By default, the ID list scan limit in Directory Server is 4000 . In most scenarios, this value provides good performance for a common range of database sizes and access patterns, and you do not need to change the default value. If the database index is slightly larger than 4000 entries, but still significantly smaller than the overall directory, raising the ID list scan limit improves searches. On the other hand, lowering the limit can significantly speed up searches that would otherwise hit the 4000 entry limit, but where it is not necessary to scan every entry. Procedure Update the ID list scan limit: # dsconf -D " cn=Directory Manager " ldap://server.example.com backend config set --idlistscanlimit= 8000 This command sets the limit to 8000 entries. Restart the instance: # dsctl instance_name restart 3.2. Setting a global index scan limit using the web console By default, the ID list scan limit in Directory Server is 4000 . In most scenarios, this value provides good performance for a common range of database sizes and access patterns, and you do not need to change the default value. If the database index is slightly larger than 4000 entries, but still significantly smaller than the overall directory, raising the ID list scan limit improves searches. On the other hand, lowering the limit can significantly speed up searches that would otherwise hit the 4000 entry limit, but where it is not necessary to scan every entry. Procedure Navigate to Database Global Database Configuration . Update the ID List Scan Limit field. Click Save Config . Click Actions in the top right corner, and select Restart Instance . 3.3. Setting an index scan limit to a database using the command line In some cases, it is useful to define a limit for certain indexes, or to not use an ID list at all. You can configure individual settings for ID list scan limits for different types of search filters. For example, in a large database with 10 million entries that contain the object class inetOrgPerson , the (&(objectClass=inetOrgPerson)(uid=user)) filter creates first an ID list containing all 10 million IDs matching objectClass=inetOrgPerson . When the database applies the second part of the filter, it searches the result list for objects matching uid=user . In this case, it is useful to define a limit for certain indexes, or to not use an ID list at all. This procedure demonstrates how to configure Directory Server to note create an ID list for objectClass=inetOrgPerson conditions in AND clauses. Procedure Set the nsIndexIDListScanLimit parameter: # ldapmodify -D " cn=Directory Manager " -W -H ldap://server.example.com -x dn: cn=objectclass,cn=index,cn= userRoot ,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsIndexIDListScanLimit nsIndexIDListScanLimit: limit=0 type=eq flags=AND values=inetOrgPerson With these settings, Directory Server does not create any ID list for objectClass=inetOrgPerson conditions in AND clauses. In all other situations, Directory Server applies the global ID list scan limit value. The nsIndexIDListScanLimit parameter uses the following syntax: nsIndexIDListScanLimit: limit=NNN [type=eq[,sub,...]] [flags=AND[,XXX,...]] [values=val[,val,...]] limit : Sets the maximum size of the ID list. Valid values are: -1 : Unlimited 0 : Do not use the index 1 to the maximum of the 32-bit integer ( 2147483647 ): Maximum number of IDs type : Optional: Sets flags that alter the scan limit's behavior. Valid values are: AND : Apply the scan limit only to searches in which the attribute appears in an AND clause. OR : Apply the scan limit only to searches in which the attribute appears in an OR clause. values : Optional: A comma-separated list of values which must match the search filter in order for the limit to be applied. Since the matches are done one at a time, the values will match if any of the values matches. Use the values only with one type at a time. The values must correspond to the index type and to the syntax of the attribute to which the index is applied. For example, if you specified the integer-based attribute uidNumber and it is indexed for the eq type, you cannot use type=eq values= abc . If the value contains spaces, commas, NULL, or other values which require escaping, use the LDAP filter escape syntax: A backslash (\) followed by the 2 hex digit code of the character. In the following example, the commas in the DN value are escaped with \2C : nsIndexIDListScanLimit: limit=0 type=eq values=uid=user\2Cou=People\2Cdc=example\2Cdc=com
dsconf -D " cn=Directory Manager " ldap://server.example.com backend config set --idlistscanlimit= 8000 dsctl instance_name restart ldapmodify -D " cn=Directory Manager " -W -H ldap://server.example.com -x dn: cn=objectclass,cn=index,cn= userRoot ,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsIndexIDListScanLimit nsIndexIDListScanLimit: limit=0 type=eq flags=AND values=inetOrgPerson nsIndexIDListScanLimit: limit=NNN [type=eq[,sub,...]] [flags=AND[,XXX,...]] [values=val[,val,...]] nsIndexIDListScanLimit: limit=0 type=eq values=uid=user\2Cou=People\2Cdc=example\2Cdc=com