x509: improve error reporting

Distinguish between not being able to extract a public key versus not knowing
the key's type.

Alternative to #15921

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15944)
