How to determine the signature type of RSA with a hash function? I have the following function:


  rsaSha512Sign(arg1, arg2) - arg1 is the private key PEM,
  arg2 is the data to sign using SHA-512 with RSA.
  Returns the signature as a Base64 String.


With no additional documentation or source code available.

I assumed the description above meant to first hash with SHA-512, then apply RSA. However the signature is much longer than the hashed arg2.

How can I determine which algorithm was used?
It is extremely likely to be either RSA with PSS padding or RSA with PKCS#1 v1.5 padding, the latter being the most likely.

Although there are many signature schemes, PKCS#1 v1.5 seems to be the most used. When many crypto libraries were devised it was the only scheme that was described by the RSA labs (that created the PKCS standards), so many older libraries forgot to document it. There are however other schemes, such as the aforementioned RSA-PSS and RSA with (partial) message recovery, defined in ISO/IEC 9796-2.

If you sign two times and the output is twice the same then it is not PSS and PKCS#1 v1.5 would be the main suspect:


  RSASSA-PSS is different from other RSA-based signature schemes in
     that it is probabilistic rather than deterministic, incorporating a
     randomly generated salt value.


Both PSS and PKCS#1 v1.5 are defined in RFC 3447. And yes, the hashing itself is almost always included in the method, so usually you just feed it the message itself.



Both PSS and PKCS#1 v1.5 first perform modular exponentiation which produces a signature size that is bounded by the size of the modulus. However, after that the function I2OSP is performed, which makes the signature the same size as the modulus in bytes. As the modulus defines the key size, the signature is the same size as the key size in bytes.

So you cannot use the resulting signature size to determine the signature algorithm.



If you have the RSA public key you could perform a raw modular exponentiation (called textbook RSA, "none with RSA", "raw RSA" or similar) with the public exponent on the (decoded) signature. Then you could compare the result with the padding format. Usually it is easier to just try PKCS#1 v1.5 and PSS though.



Base 64 converts a signature to the base 64 alphabet, expanding the size with about 33/37% compared to the key size, assuming that the string is encoded as ASCII or compatible.