title: OpenSSL RSA signature verification: hash and padding?
body: <p>I am trying to write code to verify some RSA signatures.  The signatures were made using the OpenSSL command-line tool, using the equivalent of this command line:</p>
  <pre><code>openssl dgst -sha1 -sign private_key_file.pem &lt; binary_data_file &gt; sig
  </code></pre>
  <p>I am trying to use <code>libtomcrypt</code> to do the verify:</p>
  <p><a href="https://www.libtom.net/" rel="nofollow noreferrer">https://www.libtom.net/</a></p>
  <p>Here is the calling signature of the RSA verification function in <code>libtomcrypt</code>:</p>
  <pre><code>int rsa_verify_hash_ex(
      const unsigned char *sig, unsigned long siglen,  // signature to verify
      const unsigned char *hash, unsigned long hashlen,  // hash value to check against sig
      int padding,  // defined constant value, see below
      int hash_idx,  // identifies which hash algorithm, see below
      unsigned long saltlen,  // specify salt length, see below
      int *stat,  // output parameter, returns whether verify succeeded or not
      rsa_key *key);  // RSA public key to use for verify
  </code></pre>
  <p>This function returns a 0 if it operates without error, otherwise returns an error code.  If it operates without error, the <code>stat</code> output parameter indicates whether the signature verified.</p>
  <p>Most of the arguments seem straightforward: pass in the signature to check, the hash value to use to compare it, and the RSA key to use for the check.  <code>hash_idx</code> is clear from the example code included with <code>libtomcrypt</code>; it is an index into a table of supported hash algorithms, and I can find the correct value to use with this code snippet: <code>hash_idx = find_hash(&quot;sha1&quot;)</code></p>
  <p>But I'm wondering about the <code>padding</code> and <code>saltlen</code> values.  <code>padding</code> doesn't worry me too much, as there are only two possible values, and I can just try them both.  But what should I pass for <code>saltlen</code>?</p>
  <p>The OpenSSL documentation for the OpenSSL functions for RSA verify don't show a <code>saltlen</code> parameter.  The man page for <code>openssl dgst</code> (i.e. the result of <code>man dgst</code>) does not discuss salt.</p>
  <p>So my questions:</p>
  <ul>
  <li>How can I determine the correct salt length to use?</li>
  <li>Does OpenSSL's <code>dgst</code> command insert any extra stuff in the input, such as: <code>(stdin)=</code></li>
  </ul>
  <p>(I found that <code>(stdin)=</code> thing by searching StackOverflow: <a href="https://stackoverflow.com/questions/13419201/why-are-the-rsa-sha256-signatures-i-generate-with-openssl-and-java-different">Why are the RSA-SHA256 signatures I generate with OpenSSL and Java different?</a>)</p>
  <ul>
  <li><code>libtomcrypt</code> also has a function called <code>pkcs_1_pss_decode()</code> which is documented to &quot;decode a PSS encoded signature block&quot;.  Is there any chance that this is the function I need to call?</li>
  </ul>
  <p>Thanks for any help you can give me.</p>
  <p>EDIT: thanks to the help below, from @Jonathan Ben-Avraham, I was able to get this working today.  The answers to my questions are, respectively:</p>
  <ul>
  <li>Use length 0 for the salt, no salt at all.</li>
  <li>No, OpenSSL did not insert anything extra such as <code>(stdin)=</code></li>
  <li>I needed to call <code>rsa_verify_hash_ex()</code>, and I needed to specify the <code>padding</code> argument as <code>LTC_LTC_PKCS_1_V1_5</code>.</li>
  </ul>