setting the MGF hash to a different value other than the regular hash with openssl for PSS signatures In PKCS1's Probabilistic Signature Mode you have the Hash and the MGF Hash. Both appear to be, by default, sha1, with the OpenSSL CLI client, however, I can change it to sha256 with the following:

openssl dgst -sha256 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -out signature.txt -sign privatekey.txt plaintext.txt


What if I wanted to have the Hash be sha256 and the MGF Hash by sha1? Is that possible? I just wonder because with Java the default mode for OAEP encryption changes the hash but not the MGF hash (thus making them disjoint)
Meta: since this is about using a tool not the underlying algorithm/math AIUI should be security.SE instead. If anyone can and wants to migrate feel free.

Per 1.0.2 source, -sigopt rsa_mgf1_md:name where name is the name of a hash available to EVP_getdigestbyname -- that is, implemented and not #if'ed out by default (MD2) nor manually.

For FIPS mode if relevant, I think you can set a non-FIPS hash but then the operation using it will fail, but I don't have a FIPS build set up to test.