title: Implement PBEWITHHMACSHA512ANDAES_256 of java jasypt in python
body: <p>I am trying to encrypt a password in python and decrypt it in java springboot application using the <a href="https://github.com/jasypt/jasypt" rel="nofollow noreferrer">jasypt library</a> through <a href="https://github.com/ulisesbocchio/jasypt-spring-boot" rel="nofollow noreferrer">jasypt plugin</a>.</p>
  <p><strong>What i have done so far</strong></p>
  <ul>
  <li>For simplicity i have used a zero salt and a fixed iv</li>
  <li>I have written the python script to perform the encryption using hselvarajan's <a href="https://github.com/hselvarajan/pkcs12kdf/blob/master/pkcs12kdf.py" rel="nofollow noreferrer">pkcs12kdf</a>
  <pre><code>import sys
  import math
  import base64
  import hashlib
  from Crypto.Cipher import AES
  from Crypto.Hash import SHA512
  
  from binascii import hexlify
  from binascii import unhexlify
  
  PY2 = sys.version_info[0] == 2
  PY3 = sys.version_info[0] == 3
  if PY2:
          str_encode = lambda s: str(s)
  elif PY3:
          str_encode = lambda s: str(s, 'utf-8')
  
  iterations          = 10000
  salt_block_size     = AES.block_size
  key_size            = 256
  
  password             = &quot;test1&quot;
  plaintext_to_encrypt = &quot;password1&quot;
  salt                 = &quot;0000000000000000&quot;
  iv                   = &quot;0000000000000000&quot;
  
  # -----------------------------------------------------------------------------
  # This is a pure copy paste of
  #  https://github.com/hselvarajan/pkcs12kdf/blob/master/pkcs12kdf.py
  # -----------------------------------------------------------------------------
  class PKCS12KDF:
          &quot;&quot;&quot;This class generates keys and initialization vectors from passwords as specified in RFC 7292&quot;&quot;&quot;
  
          #
          # IDs for Key and IV material as in RFC
          #
  
          KEY_MATERIAL = 1
          IV_MATERIAL = 2
  
          def __init__(self, password, salt, iteration_count, hash_algorithm, key_length_bits):
                  self._password = password
                  self._salt = salt
                  self._iteration_count = iteration_count
                  self._block_size_bits = None
                  self._hash_length_bits = None
                  self._key_length_bytes = key_length_bits/8
                  self._key = None
                  self._iv = None
                  self._hash_algorithm = hash_algorithm
          #
          # Turns a byte array into a long
          #
  
          @staticmethod
          def byte_array_to_long(byte_array, nbytes=None):
                  #
                  # If nbytes is not present
                  #
                  if nbytes is None:
                          #
                          # Convert byte -&gt; hex -&gt; int/long
                          #
                          return int(hexlify(byte_array), 16)
                  else:
                          #
                          # Convert byte -&gt; hex -&gt; int/long
                          #
                          return int(hexlify(byte_array[-nbytes:]), 16)
  
          #
          # Turn a long into a byte array
          #
  
          @staticmethod
          def long_to_byte_array(val, nbytes=None):
                  hexval = hex(val)[2:-1] if type(val) is long else hex(val)[2:]
                  if nbytes is None:
                          return unhexlify('0' * (len(hexval) &amp; 1) + hexval)
                  else:
                          return unhexlify('0' * (nbytes * 2 - len(hexval)) + hexval[-nbytes * 2:])
  
          #
          # Run the PKCS12 algorithm for either the key or the IV, specified by id
          #
  
          def generate_derived_parameters(self, id):
  
                  #
                  # Let r be the iteration count
                  #
  
                  r = self._iteration_count
  
                  if self._hash_algorithm not in hashlib.algorithms_available:
                          raise NotImplementedError(&quot;Hash function: &quot;+self._hash_algorithm+&quot; not available&quot;)
  
                  hash_function = hashlib.new(self._hash_algorithm)
  
                  #
                  # Block size, bytes
                  #
                  #v = self._block_size_bits / 8
                  v = hash_function.block_size
  
                  #
                  # Hash function output length, bits
                  #
                  #u = self._hash_length_bits / 8
                  u = hash_function.digest_size
  
                  # In this specification however, all passwords are created from BMPStrings with a NULL
                  # terminator. This means that each character in the original BMPString is encoded in 2
                  # bytes in big-endian format (most-significant byte first). There are no Unicode byte order
                  # marks. The 2 bytes produced from the last character in the BMPString are followed by
                  # two additional bytes with the value 0x00.
  
                  password = (unicode(self._password) + u'\0').encode('utf-16-be') if self._password is not None else b''
  
                  #
                  # Length of password string, p
                  #
                  p = len(password)
  
                  #
                  # Length of salt, s
                  #
                  s = len(self._salt)
  
                  #
                  # Step 1: Construct a string, D (the &quot;diversifier&quot;), by concatenating v copies of ID.
                  #
  
                  D = chr(id) * v
  
                  #
                  # Step 2: Concatenate copies of the salt, s, together to create a string S of length v * [s/v] bits (the
                  # final copy of the salt may be truncated to create S). Note that if the salt is the empty
                  # string, then so is S
                  #
  
                  S = b''
  
                  if self._salt is not None:
                          limit = int(float(v) * math.ceil((float(s)/float(v))))
                          for i in range(0, limit):
                                  S += (self._salt[i % s])
                  else:
                          S += '0'
  
                  #
                  # Step 3: Concatenate copies of the password, p, together to create a string P of length v * [p/v] bits
                  # (the final copy of the password may be truncated to create P). Note that if the
                  # password is the empty string, then so is P.
                  #
  
                  P = b''
  
                  if password is not None:
                          limit = int(float(v) * math.ceil((float(p)/float(v))))
                          for i in range(0, limit):
                                  P += password[i % p]
                  else:
                          P += '0'
  
                  #
                  # Step 4: Set I=S||P to be the concatenation of S and P.\00\00
                  #
  
                  I = bytearray(S) + bytearray(P)
  
                  #
                  # 5. Set c=[n/u]. (n = length of key/IV required)
                  #
  
                  n = self._key_length_bytes
                  c = int(math.ceil(float(n)/float(u)))
  
                  #
                  # Step 6 For i=1, 2,..., c, do the following:
                  #
  
                  Ai = bytearray()
  
                  for i in range(0, c):
                          #
                          # Step 6a.Set Ai=Hr(D||I). (i.e. the rth hash of D||I, H(H(H(...H(D||I))))
                          #
  
                          hash_function = hashlib.new(self._hash_algorithm)
                          hash_function.update(bytearray(D))
                          hash_function.update(bytearray(I))
  
                          Ai = hash_function.digest()
  
                          for j in range(1, r):
                                  hash_function = hashlib.sha256()
                                  hash_function.update(Ai)
                                  Ai = hash_function.digest()
  
                          #
                          # Step 6b: Concatenate copies of Ai to create a string B of length v bits (the final copy of Ai
                          # may be truncated to create B).
                          #
  
                          B = b''
  
                          for j in range(0, v):
                                  B += Ai[j % len(Ai)]
  
                          #
                          # Step 6c: Treating I as a concatenation I0, I1,..., Ik-1 of v-bit blocks, where k=[s/v]+[p/v],
                          # modify I by setting Ij=(Ij+B+1) mod 2v for each j.
                          #
  
                          k = int(math.ceil(float(s)/float(v)) + math.ceil((float(p)/float(v))))
  
                          for j in range(0, k-1):
                                  I = ''.join([
                                          self.long_to_byte_array(
                                                  self.byte_array_to_long(I[j:j + v]) + self.byte_array_to_long(bytearray(B)), v
                                          )
                                  ])
  
                  return Ai[:self._key_length_bytes]
  
          #
          # Generate the key and IV
          #
  
          def generate_key_and_iv(self):
                  self._key = self.generate_derived_parameters(self.KEY_MATERIAL)
                  self._iv = self.generate_derived_parameters(self.IV_MATERIAL)
                  return self._key, self._iv
  
  # -----------------------------------------------------------------------------
  # Main execution
  # -----------------------------------------------------------------------------
  kdf = PKCS12KDF(
      password        = password,
      salt            = salt,
      iteration_count = iterations,
      hash_algorithm  = &quot;sha512&quot;,
      key_length_bits = key_size
  )
  (key, iv_tmp) = kdf.generate_key_and_iv()
  aes_key = key[:32]
  
  pad = salt_block_size - len(plaintext_to_encrypt) % salt_block_size
  plaintext_to_encrypt = plaintext_to_encrypt + pad * chr(pad)
  
  cipher = AES.new(aes_key, AES.MODE_CBC, iv)
  encrypted = cipher.encrypt(plaintext_to_encrypt)
  
  # Since we selt the salt to be zero's,
  # jasypt needs only the iv + encrypted value,
  # not the salt + iv + encrypted
  result = str_encode(base64.b64encode(iv + encrypted))
  
  # Python output : MDAwMDAwMDAwMDAwMDAwMKWsWH+Ku37n7ddfj0ayxp8=
  # Java output   : MDAwMDAwMDAwMDAwMDAwMAtqAfBtuxf+F5qqzC8QiFc=
  print(result)
  
  </code></pre>
  Run it as
  <pre><code>python2.7 test-PBEWITHHMACSHA512ANDAES_256.py
  paxYf4q7fuft11+PRrLGnw==
  </code></pre>
  </li>
  <li>I have written a unit test in jasypt repository to decrypt
  See <a href="https://github.com/amolofos/jasypt/blob/feature/PBEWITHHMACSHA512ANDAES_256EncryptorTest/jasypt/src/test/java/org/jasypt/encryption/pbe/PBEWITHHMACSHA512ANDAES_256EncryptorTest.java" rel="nofollow noreferrer">PBEWITHHMACSHA512ANDAES_256EncryptorTest</a>.
  Run it as
  <pre><code>$ cd jasypt
  $ mvn clean test -Dtest=org.jasypt.encryption.pbe.PBEWITHHMACSHA512ANDAES_256EncryptorTest
  </code></pre>
  </li>
  </ul>
  <p><strong>The problem: The above setup produces different results in python and in java</strong></p>
  <ul>
  <li>Python output : MDAwMDAwMDAwMDAwMDAwMKWsWH+Ku37n7ddfj0ayxp8=</li>
  <li>Java output   : MDAwMDAwMDAwMDAwMDAwMAtqAfBtuxf+F5qqzC8QiFc=</li>
  </ul>
  <p><strong>What i know</strong></p>
  <ul>
  <li>The failure is due to not using the using the correct key in python. Adding additional logs, the error is
  <pre><code>EncryptionOperationNotPossibleException: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
  </code></pre>
  </li>
  <li>The PBEWITHHMACSHA512ANDAES_256 uses the pkcs12 key derivation function.
  I do not understand where the HMAC is being used.</li>
  <li>I have also tried using the folling implementation to no avail. I am getting &quot;&quot; error in all of them.
  <ul>
  <li><a href="https://github.com/wbond/oscrypto/blob/master/oscrypto/_pkcs12.py" rel="nofollow noreferrer">oscrypto</a></li>
  <li><a href="https://github.com/casebeer/python-hkdf/blob/master/hkdf.py" rel="nofollow noreferrer">python-hkdf</a></li>
  <li>Cryptodome.Protocol.KDF HKDF
  I do not understand where the iterations are being used here.</li>
  </ul>
  <pre><code>self.aes_key = HKDF(master = self.password, key_len = 32, salt = self.salt, hashmod = SHA512, num_keys = 1)
  </code></pre>
  </li>
  </ul>
  <p>I would like some guidance on what i am doing wrong. Any help, any pointers would be much appreciated.</p>
  <hr />
  <p><strong>Update following Cryptodome's PBKDF2 and AES</strong>
  Here is the python script</p>
  <pre><code>import sys
  import base64
  from Cryptodome.Cipher import AES
  from Cryptodome.Hash import SHA512
  from Cryptodome.Protocol.KDF import PBKDF2
  from Cryptodome.Util.Padding import pad
  
  iterations          = 10000
  password             = b'test1'
  plaintext_to_encrypt = b'password1'
  salt                 = b'0000000000000000'
  iv                   = b'0000000000000000'
  
  # -----------------------------------------------------------------------------
  # Main execution
  # -----------------------------------------------------------------------------
  keys = PBKDF2(password, salt, 64, count=iterations, hmac_hash_module=SHA512)
  aes_key = keys[:32]
  
  cipher = AES.new(aes_key, AES.MODE_CBC, iv)
  ct_bytes = cipher.encrypt(pad(plaintext_to_encrypt, AES.block_size))
  encrypted = base64.b64encode(ct_bytes).decode('utf-8')
  
  # Since we selt the salt to be zero's,
  # jasypt needs only the iv + encrypted value,
  # not the salt + iv + encrypted
  result = encrypted
  
  # Python output : 6tCAZbswCh9DZ1EK8utRuA==
  # Java output   : C2oB8G27F/4XmqrMLxCIVw==
  print(result)
  
  </code></pre>
  <p>and its output</p>
  <pre><code>python2.7 test-PBEWITHHMACSHA512ANDAES_256-2.py
  6tCAZbswCh9DZ1EK8utRuA==
  </code></pre>
  <p>I try to decrypt it in java with the following error using the <a href="https://github.com/amolofos/jasypt/blob/feature/PBEWITHHMACSHA512ANDAES_256EncryptorTest/jasypt/src/test/java/org/jasypt/encryption/pbe/PBEWITHHMACSHA512ANDAES_256EncryptorTest.java" rel="nofollow noreferrer">test</a></p>
  <pre><code>mvn clean test -Dtest=org.jasypt.encryption.pbe.PBEWITHHMACSHA512ANDAES_256EncryptorTest
  
  [...]
  
  Running org.jasypt.encryption.pbe.PBEWITHHMACSHA512ANDAES_256EncryptorTest
  Test encr: C2oB8G27F/4XmqrMLxCIVw==
  Error: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.524 sec &lt;&lt;&lt; FAILURE!
  test1(org.jasypt.encryption.pbe.PBEWITHHMACSHA512ANDAES_256EncryptorTest)  Time elapsed: 0.522 sec  &lt;&lt;&lt; ERROR!
  org.jasypt.exceptions.EncryptionOperationNotPossibleException
      at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1173)
      at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:738)
      at org.jasypt.encryption.pbe.PBEWITHHMACSHA512ANDAES_256EncryptorTest.test1(PBEWITHHMACSHA512ANDAES_256EncryptorTest.java:27)
  
  
  Results :
  
  Tests in error:
    test1(org.jasypt.encryption.pbe.PBEWITHHMACSHA512ANDAES_256EncryptorTest)
  
  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
  
  [INFO] ------------------------------------------------------------------------
  [INFO] BUILD FAILURE
  [INFO] ------------------------------------------------------------------------
  [INFO] Total time:  8.648 s
  [INFO] Finished at: 2020-06-24T17:40:04+08:00
  [INFO] ------------------------------------------------------------------------
  [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project jasypt: There are test failures.
  [ERROR]
  [ERROR] Please refer to /space/openbet/git/github-jasypt-jasypt/jasypt/target/surefire-reports for the individual test results.
  [ERROR] -&gt; [Help 1]
  [ERROR]
  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  [ERROR]
  [ERROR] For more information about the errors and possible solutions, please read the following articles:
  [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
  </code></pre>