Apple Mail is trying to use certificate and won't send without
I just noticed that there is a new toggle button available in Apple Mail when writing a new mail that lets me send the mail without digitally signing it: