How does OpenSSL know if a decryption failed? I have a file full of garbage, random data, and I encrypt it with AES256:

openssl -in myfile -out encfile -aes256 -pass pass:abc123


If I try to decrypt it with the wrong password, it says:

bad decrypt
140546891773584:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:516:


But, if I try to decrypt it with the correct password, it doesn't return any errors, meaning it was successful.

How does it know that it was successful?
This code is very near the lines the error mentions for evp_enc.c (there may be some differences between different versions of OpenSSL):

 527                 n=ctx-&gt;final[b-1];
 528                 if (n == 0 || n &gt; (int)b)
 529                         {
 530                         EVPerr(EVP_F_EVP_DECRYPTFINAL_EX,EVP_R_BAD_DECRYPT);
 531                         return(0);
 532                         }
 533                 for (i=0; i&lt;n; i++)
 534                         {
 535                         if (ctx-&gt;final[--b] != n)
 536                                 {
 537                                 EVPerr(EVP_F_EVP_DECRYPTFINAL_EX,EVP_R_BAD_DECRYPT);
 538                                 return(0);
 539                                 }
 540                         }


This is an implementation of PKCS#7 unpadding (b is the block size, retrieved from the "context").

There is no mention of MAC verification or authentication tag inside this code, so we can safely assume this is only called for ECB/CBC decryption (if you don't provide a mode of operation, OpenSSL defaults to CBC with PKCS#7 padding).

This also means that there is no certainty at all that the decryption was correct if you do not receive an error. For instance, you may get "lucky" and create a valid padding even though the wrong ciphertext was decrypted (or, with the wrong IV or key). And if the ciphertext two blocks before was changed then the error won't be caught either because CBC has limited error propagation.

In other words, if you need integrity / authenticity then you need an authentication tag, for instance using an authenticated mode such as AES-GCM.