How to store and retrieve S/MIME certificates in Open Directory (LDAP) of macOS Server?
Found it, and it's fairly easy (as one would expect from macOS). I'll explain step by step. 

How to store S/MIME certificates in Open Directory


Launch the Directory Utility app, either directly from Spotlight, or (that's how I found it) by going through System Preferences > Users &amp; Groups > (authenticate if necessary) Network Account Server - Edit (or Connect if you haven't yet) > Open Directory Utility
Select the Directory Editor on the top
Select Viewing: Users, if not selected by default, then click on a user, and the user's set attributes will be shown
To add an attribute, by clicking on the lock above, authenticate to get into edit mode - note that, this time, you must authenticate with the directory admin credentials, because now we want to edit the directory entries
Click on the + sign below the attribute list to add a new attribute
As you will see, the userCertificate and userSMIMECertificate attributes are already available from the dropdown list 
Select one of them, then press OK
When you click on the attribute, its data are shown below the attribute list
You can toggle between Text and Data view – to add an S/MIME certificate, either copy paste into the text view or just drop the file in the Data view
That's all!


I believe, the S/MIME certificate data should in PEM format, but I am not sure about it. 

How to retrieve S/MIME certificates from Open Directory 

In order to enable clients to retrieve S/MIME certificates from Open Directory (or any LDAP actually), here's how to do this on a Mac client:

Use the Directory Utility app, as explained above, to add the LDAP server to the search list on the client Mac (also explained here). Make sure that the LDAP server is added in the Search Policy list under Contacts. The Contacts server in the Server app need not be enabled. 

After you have successfully added the LDAP server, open your Contacts app on the client. There should be a new entry Directory Services on the left sidebar. Click on it, then, in the Search field, enter the name of a user in your LDAP directory. The directory entry should then be displayed, and left of the email address, you should see a small certificate symbol  similar to the symbol in Mail app that is displayed for signed emails. When you click on it, the S/MIME certificate retrieved for this user will be displayed.