How to send encrypted email using Mail app?
This is how it works here, with the help of a self-signed root certificate:

Step 1: Alice creates a self-signed root certificate


In the Keychain app, select Keychain Access > Certificate Assistent > Create a Certificate Authority
Select   


Identity Type: Self Signed Root CA
User Certificate: S/MIME (Email)
Let me override defaults: yes
Email from: Enter an email address for the root CA (could be Alice's)
On the next dialog, select Sign your invitation: no - optionally create a CA web site
On the next dialog, enter whatever CA data you want 
On the next two dialogs, leave Key Pair Information as is
On the next dialog Key Usage Extension

Include Key Usage Extension: yes
This extension is critical: no
Select: Signature, Certificate Signing (optionally more, but not required for a root certificate)
On the next dialog, leave Key Usage Extension for the user's certificate as is
On the next dialog, optionally set Include Extended Key Usage Extension = yes, if you want to use the root CA for more than just email certificates - but always leave This extension is critical = no 
On the next dialog, leave Use Basic Constraints Extension = Use this certificate as a certificate authority
On the remaining dialogs, leave selections as they are
On the last dialog Specify a Location For The Certificate, you can select System, and also select On this machine, trust certificates signed by this CA = yes


Step 2: Alice and Bob trust the the root CA


If Alice hasn't already trusted the CA certificate during the previous step, she now selects the new root CA certificate in her Keychain app and explicitly trusts the CA certificate
Alice exports the public CA certificate in .cer format and publishes it to Bob
Bob imports the public certificate and explicitly trusts this CA


Since Alice and Bob have explicitly trusted the CA, all certificates signed by this CA will be automatically trusted on their computers.

Step 3: Alice creates an email certificate for herself.


In the Keychain app, select Keychain Access > Certificate Assistent > Create a Certificate
Enter Alice's name, then select   


Identity Type: Leaf
User Certificate: S/MIME (Email)
Let me override defaults: yes

Leave next dialog (serial no. etc.) as is
On the next dialog, for Email Address, enter Alice's email address this certificate is intended for, optionally fill out remaining fields as desired
On the next dialog Choose An Issuer, select the CA created in step 1
On the next dialog, keep Key Pair Information as is
On the next dialog Key Usage Extension
Include Key Usage Extension: yes
This extension is critical: yes
Select: Signature, Key Encipherment (optionally, adding Data Encipherment seems to be possible, but do not add more!)

On the next dialog, leave Key Usage Extension for the user's certificate as is

On the next dialog, optionally set Include Extended Key Usage Extension = yes, but set This extension is critical = no 


Select "Email Protection*, add more if you want to use the same certificate for other purposes

On the next dialog, leave Include Basic Constraints Extension = no
On the next dialog, select Include Subject Alternate Name Extension = yes 


Leave This extension is critical = no
Enter the email address (again) in the rfc822Name field. Optionally add more email addresses, separated by space, which you may want to use the certificate for

On the last dialog Specify a Location For The Certificate, you can select Login


Step 4: Alice creates a certificate for Bob

Now Alice repeats step 3 to create a certificate for Bob, then exports his certificate in .p12 format, gives it to him, and Bob imports it into his keychain. 

Note: This is the easiest way to create Bob's certificate, yet not the recommended way. This is because Alice creates Bob's private key, so Bob has to trust Alice. Actually Bob would use the Certificate Assistent on his computer to Request a Certificate From a Certificate Authority. 

Step 5: Alice and Bob send each other signed messages

Alice sends a signed email message to Bob, and Bob sends a signed email message to Alice. If all went well, the signatures will display as trusted because both trust the CA certificate. 

Step 6: Alice and Bob can exchange encrypted email messages

Both can reply to the signed message using encryption with the reply message.