Why is data signed with SHA256-RSA-PKCS and digest signed with RSA-PKCS different?
TLDR: it's the DigestInfo
The hash-and-RSA-PKCS schemes defined in Current Mechanisms 2.1.14 do the full scheme of RSASSA-PKCS1-v1_5 which is now defined in RFC8017 primarily in 9.2 as a combination of EMSA-PCKS1-v1_5 'encoding' (usually called padding) plus the RSASP1 and RSAVP1 primitives. (In prior versions the document was organized differently, but the substance was the same.) Concretely this consists of:
#1 hash the data
#2 encode the hash value, plus the hash algorithm, in an ASN.1 structure DigestInfo in DER; since each hash algorithm has constant output size, this amounts to prepending a fixed prefix depending only on the algorithm, as given in note 1
#3-5 prepend padding of the form 00 01 FF...(at least 8) 00
(8.2.1#2) treating the result as a number m, apply RSASP1 which does m ^ d mod n (or for verify 8.2.2#2 apply RSAVP1 which does s ^ e mod n; this is stated as before the three padding steps above but actually can just as well be after)
OTOTH RSA-PKCS defined in 2.1.6 for Sign/Verify does ONLY the latter two of those steps, not the first two. As the text says: &quot;This mechanism corresponds only to the part of PKCS #1 v1.5 that involves RSA; it does not compute a message digest or a DigestInfo encoding as specified ... in PKCS #1 v1.5 .&quot; You externally hashed the data which is the first step, but did not do the DigestInfo which is the second (required) step, and thus got the wrong result.
Except for being PKCS11 (hardware) rather than software, this is crossdupe at least:
https://stackoverflow.com/questions/521101/using-sha1-and-rsa-with-java-security-signature-vs-messagedigest-and-cipher
https://stackoverflow.com/questions/9951559/difference-between-openssl-rsautl-and-dgst
https://stackoverflow.com/questions/38767660/multiple-openssl-rsa-signing-methods-produce-different-results
https://stackoverflow.com/questions/63582544/how-does-jwt-implement-rsa256-signature-verification-in-nodejs
https://stackoverflow.com/questions/62955968/how-can-realize-openssl-pkeyutl-sign-by-java
https://stackoverflow.com/questions/48013643/get-the-sha1-hashed-value-from-xml-signature-value
https://stackoverflow.com/questions/43490188/separate-digest-signing-using-java-security-provider
https://stackoverflow.com/questions/57929493/openssl-digitally-sign-digest-only
https://stackoverflow.com/questions/9380856/openssl-signature-difference-when-using-c-routines-and-openssl
https://stackoverflow.com/questions/13419201/why-are-the-rsa-sha256-signatures-i-generate-with-openssl-and-java-different
https://stackoverflow.com/questions/15092298/signing-20-byte-message-with-256-bit-rsa-key-working-with-openssl-exe-but-not-in
https://superuser.com/questions/943972/what-is-the-difference-between-openssl-pkeyutl-sign-and-openssl-rsautl-sign