I get an error message when unlocking an APFS FileVault volume with an Institutional Recovery Key
The thing I forgot to mention was I didn't generate the Keychain myself. I started off with a keychain with a certificate and private key that I could unlock, and an MDM profile with the exported certificate.

Because I couldn't find another solution, I created a new Keychain. I used this Apple Support Document. I didn't remove the private key, because I deploy it by exporting the certificate, and distributing it via MDM.

However, I did use the "Use the private key to unlock a user's startup disk" procedure from this document to unlock the volume and it worked! 

So this is not an actual answer to the asked question, but certainly a solution. I double checked the computer encrypted with the old keychain certificate with this procedure, but got the same error message. So probably there was something wrong with the keychain and/or certificate...