I can't use Mail.app in El Capitan to encrypt email with a self-signed certificate
First, to get the certificate to show up in the TLS section of the account, you need to import the cert into your keychain in P12 format. This is important because if you only import the cert (.crt), then that is only the public-facing portion of the identity (i.e. the public key). When you import the cert as a .p12, you are importing both the .crt and the private key associated with it. Now, keychain recognizes this as YOUR identity and will let you select it in mail as the source of encryption and signing. Also be sure that the certificate you are using has the correct authority to encrypt and sign email. You can verify this by checking for the presence of "email protection" under extended key usage and "Digital signature" and "key encipherment" under Key usage. Without these, you will likely have similar problems.

Similarly, be sure the .crt of the person you are encrypting is in your keychain, or you won't be able to encrypt TO them.