title: TLS: How to verify a certificate self-signed with an RSA key
body: <p>I am trying to use OpenSSL and TLS to provide SSH-like trust model. That means that the two peers have an RSA key pair stored each one, no certificates. They have also exchanged their public keys prior to the TLS session establishment. To achieve this I am using some OpenSSL's undocumented functions so I still have doubts, here is how:</p>
  
  <ul>
  <li><p>During initialisation peers generate an in-memory, temporary x509 certificate. It contains the minimal dummy data I could get away with, i.e. a dummy CN, issuer, and many years before expiring. <em>These are never checked anyway</em>. SSL context is setup to exchange both certificates both ways.</p></li>
  <li><p>The important step is that the certificate contains the host's public key and it is signed using the private key:</p>
  
  <pre><code>EVP_PKEY *pkey = EVP_PKEY_new();
  EVP_PKEY_assign_RSA(pkey, PRIVKEY);
  X509_set_pubkey(x509, pkey);
  X509_sign(x509, pkey, EVP_sha384());
  </code></pre></li>
  <li><p>This certificate is assigned to <code>SSL_CTX</code> and used for the whole application lifetime.</p></li>
  </ul>
  
  <p>What I'm mostly worried about is the verification process when the TLS session is established. I'm disabling all verification performed by OpenSSL using <code>SSL_CTX_set_cert_verify_callback(always_true_func)</code> and roll my own like this:</p>
  
  <pre><code>X509 *received_cert = SSL_get_peer_certificate(conn_info-&gt;ssl);
  EVP_PKEY *received_pubkey = X509_get_pubkey(received_cert);
  if (EVP_PKEY_type(received_pubkey-&gt;type) != EVP_PKEY_RSA)
      error();
  ret = X509_verify(received_cert, received_pubkey);
  if (ret &lt;= 0)
      error("trust_failed");
  
  // Compare received public key with expected one
  RSA *expected_rsa_key = read_RSA_key_from_disk();
  EVP_PKEY expected_pubkey = { 0 };
  EVP_PKEY_assign_RSA(&amp;expected_pubkey, expected_rsa_key);
  EVP_PKEY_cmp(received_pubkey, &amp;expected_pubkey);
  
  if (ret == 1)
      return true; // identity verified!
  else
      return false;
  </code></pre>
  
  <p><strong>The question:</strong> Is this proper usage of the OpenSSL API? Do you see any security holes, especially on the last part, verification of the received key? Any better way to achieve the same result?</p>
  
  <p><strong>EDIT: The answer:</strong> To verify that a self-signed certificate received during TLS handshake matches the stored RSA key, <em>there is no need to check the certificate's signature</em>, i.e. there is no need for <code>X509_verify</code>. The comparison of the received public key to the expected one is enough.</p>
  
  <p>The reason is that (quoting Dr Stephen Henson, OpenSSL project core developer) "<em>Depending on the ciphersuite either an RSA decryption operation or an RSA
  signature operation is performed by the server. So if the handshake completes
  successfully you can be sure that the same key is used as the one present in
  the certificate</em>".</p>