title: SMIME encryption using openssl , decrypting with java bouncy castle fails
body: <p>Hi i have an application where i generate key pair and x509 certificate, i then use the public key (key.pem) and use openssl to encrypt my data to smime format as shown.</p>
  
  <pre><code>openssl smime -encrypt -out encrypted_smime.p7m -in token.json out form SMIME key.pem
  </code></pre>
  
  <p>the encrypted file looks like </p>
  
  <pre><code>MIME-Version: 1.0
  Content-Disposition: attachment; filename="smime.p7m"
  Content-Type: application/x-pkcs7-mime; smime-type=enveloped-data; name="smime.p7m"
  Content-Transfer-Encoding: base64
  
  ......
  </code></pre>
  
  <p>after this i take this and give it as an input to my application as serverTokenBytes which tries decrypting it using this logic. </p>
  
  <pre><code>   public static String decryptServerToken(final byte[] serverTokenBytes, final X509Certificate certificate, final PrivateKey privateKey) {
          try {
              final RecipientId recId = new JceKeyTransRecipientId(certificate);
              final Properties props = System.getProperties();
              final Session session = Session.getDefaultInstance(props, null);
              final MimeMessage msg = new MimeMessage(session, new ByteArrayInputStream(serverTokenBytes));
              if (msg.getSize() &lt;= 0 || !(msg.isMimeType("application/pkcs7-mime") || msg.isMimeType("application/x-pkcs7-mime"))) {
                  // log error...and throw
              }
              final SMIMEEnveloped m = new SMIMEEnveloped(msg);
              final RecipientInformationStore recipients = m.getRecipientInfos();
  
              final RecipientInformation recipient = recipients.get(recId);
              if (recipient == null) {
                  LOG.error("Error in decrypting the uploaded server token file, the certificate serial number used for encryption differ from the one used for decryption");
                  return null;
              }
              final MimeBodyPart res = SMIMEUtil.toMimeBodyPart(recipient.getContent(new JceKeyTransEnvelopedRecipient(privateKey)));
              return res.getContent().toString();
          } catch (final MessagingException | CMSException | SMIMEException | IOException exn) {
              throw new IllegalStateException("Error in decrypting the server token", exn);
          }
      }
  </code></pre>
  
  <p>For some reason this does not work and i an getting back a empty String back without an of the errors mentioned in this code. </p>
  
  <p>I wanted to know if 
  1. Encrypting with openssl and decrypting with java bouncy castle . Is this ok ?</p>
  
  <ol>
  <li>The decryption doesnt fail and just gives me an empty string , any one sees any problem with this code ?</li>
  </ol>
  
  <p><em><strong></em>+++++++++++++Update 1 +++++++++++++++**</strong></p>
  
  <p>Well while debugging i see that the data is getting decrypted properly and getting attached to header of MimeBodyPart but then things go bad as shown below.</p>
  
  <p>I have narrowed down to this code which is causing problem .</p>
  
  <p>public MimeBodyPart(InputStream is) throws MessagingException {</p>
  
  <pre><code>if (!(is instanceof ByteArrayInputStream) &amp;&amp;
      !(is instanceof BufferedInputStream) &amp;&amp;
      !(is instanceof SharedInputStream))
      is = new BufferedInputStream(is);
  
  headers = new InternetHeaders(is);
  
  if (is instanceof SharedInputStream) {
      SharedInputStream sis = (SharedInputStream)is;
      contentStream = sis.newStream(sis.getPosition(), -1);
  } else {
      try {
      content = ASCIIUtility.getBytes(is);
      } catch (IOException ioex) {
      throw new MessagingException("Error reading input stream", ioex);
      }
  }
  </code></pre>
  
  <p>and the ASCIIUtility getBytes returns []</p>
  
  <pre><code>public static byte[] getBytes(InputStream is) throws IOException {
  
  int len;
  int size = 1024;
  byte [] buf;
  
  
  if (is instanceof ByteArrayInputStream) {
      size = is.available();
      buf = new byte[size];
      len = is.read(buf, 0, size);
  }
  else {
      ByteArrayOutputStream bos = new ByteArrayOutputStream();
      buf = new byte[size];
      while ((len = is.read(buf, 0, size)) != -1)
      bos.write(buf, 0, len);
      buf = bos.toByteArray();
  }
  return buf;
  }
  </code></pre>
  
  <p>the size = is.available(); always returns 0;</p>
  
  <p><em><strong></em>+++++++++++++Update 2 +++++++++++++++**</strong>
  I generated another encrypted message now with using java's BC as a provider.
  the difference between the encrypted messages created by these two are like this in terms of header. </p>
  
  <p>Bouncy castle generated encrypted messages: </p>
  
  <pre><code>Content-Type: application/pkcs7-mime; name="smime.p7m"; smime-type=enveloped-data
  Content-Transfer-Encoding: base64
  MIME-Version: 1.0
  Message-ID: &lt;621547276.1.1394777375030.JavaMail.abcd@localhost&gt;
  </code></pre>
  
  <p>.....</p>
  
  <p>openssl genarted encrypted message header.</p>
  
  <pre><code>MIME-Version: 1.0
  Content-Disposition: attachment; filename="smime.p7m"
  Content-Type: application/x-pkcs7-mime; smime-type=enveloped-data; name="smime.p7m"
  Content-Transfer-Encoding: base64
  </code></pre>
  
  <p>......</p>
  
  <p>The openssl generated encrypted message when it gets passed to 
  final MimeBodyPart res = SMIMEUtil.toMimeBodyPart(...content...) cause the MimeBodyPart headers to be set differently than as compared to when bouncy castle generated message when passed through the same api. </p>