title: RSA_public_decrypt and MS Crypto API equivalent
body: <p>I'm trying to develop a license verification solution. Licenses are encoded on server using OpenSSL's <code>RSA_private_encrypt</code> function.</p>
  
  <p>For Mac OX X I use <code>RSA_public_decrypt</code> and it works like a charm. On Windows I must use very tiny bit of code, so I can not link with OpenSSL or other lib AND I have to use MS Crypto API.</p>
  
  <p>I have spent several days trying to figure out what is wrong, but with no luck. I can successfully import public key, but here my success ends.
  I'm aware that I need to reverse byte order with CAPI so this might not be the issue.</p>
  
  <p>I have tried everything, including <code>CryptVerifyMessageSignatureWithKey</code> and <code>CryptDecodeObject</code> to load the blob with different params, but still no luck.</p>
  
  <p>It always ends up with <code>GetLastError() == CRYPT_E_ASN1_BADTAG</code>, which I assume means that the BLOB is not ASN1 formatted... Google does not tell anything on the output format of RSA_private_encrypt... so I am completely lost here.</p>
  
  <p>Here is the OS X code based on OpenSSL:</p>
  
  <pre class="lang-c prettyprint-override"><code>void cr_license_init(const char* lic) {
      __cr_license_ = lic;
      unsigned char lic_encoded[CR_LIC_LEN];
  
      BIO* b64 = BIO_new(BIO_f_base64());
      BIO* licIn = BIO_new_mem_buf((void*)lic, -1);
      licIn = BIO_push(b64, licIn);
  
      if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) {
  
          const unsigned char* key_data = license_pub_der;
          RSA* r = d2i_RSA_PUBKEY(NULL, &amp;key_data, sizeof(license_pub_der));
  
          if(r != NULL) {
              if(__cr_license_data_ != NULL) {
                  free((void*)__cr_license_data_);
              }
              __cr_license_data_ = malloc(CR_LIC_LEN);
              if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded,
      (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) &amp;lt= 0) {
                  free((void*)__cr_license_data_);
                  __cr_license_data_ = NULL;
              }
              RSA_free(r);
          }
      }
      BIO_free_all(licIn);
  }
  </code></pre>
  
  <p>This part of code on windows works well, so I assume public key is not an issue.</p>
  
  <pre class="lang-c prettyprint-override"><code>__cr_license_ = lic;
  unsigned char lic_encoded[CR_LIC_LEN];
  
  DWORD dwSize;
  if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &amp;dwSize, NULL, NULL) &amp;&amp; dwSize == CR_LIC_LEN) {
  HCRYPTPROV hProv;
  if(CryptAcquireContext(&amp;hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
      PCERT_PUBLIC_KEY_INFO pki = NULL;
      DWORD dwKeySize;
      if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &amp;pki, &amp;dwKeySize)) {
          HCRYPTKEY hKey = 0;
          if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &amp;hKey)) {
  </code></pre>
  
  <p>But after that anything I try to do with message leads to <code>CRYPT_E_ASN1_BADTAG</code>. I tried <code>CryptMsgOpenToDecode</code> with <code>CryptMsgUpdate</code>, <code>CryptDecodeObject</code>, <code>CryptVerifyMessageSignatureWithKey</code> - nothing works.</p>
  
  <p>Basically I think that the problem is in pkcs1 and pkcs7 incompatibility as owlstead mentioned. Does anyone has experience working with pkcs1 format importing/converting/etc with MS CAPI?</p>
  
  <p>Any help or even a clue is appreciated a lot! Thanks in advance!</p>