Is there a bulletproof way to know a file has been successfully decrypted? I am wondering whether there is a bulletproof way to know if one is successful in decrypting a file. 

As an example, say I encrypt a text file and protect it with a 3 letter password. I use brute force to decrypt it, but two different passwords lead to a file that contains English intelligible text. Is there a way for me to know which one is the original one? In other words, is there an infallible way to know I've entered the password correctly?
From the application designer's perspective: Yes, by using an authenticated cipher like AES-GCM or crypto_secretbox_xsalsa20poly1305, which you should do anyway.  See the scrypt tool for an example of an application that does just that: encrypts a single file using a key derived from a password using an authenticated cipher—specifically, the tool uses AES-CTR and HMAC-SHA256 in encrypt-then-MAC composition, but these days it is almost always simpler to reach for AES-GCM or crypto_secretbox_xsalsa20poly1305.



From the adversary's perspective: Can't say without more information about the plaintext in your particular case.

In most applications in practice, the plaintext has some identifiable pattern even if the application foolishly didn't use authenticated encryption.  For example, an HTML document will usually begin with &lt;html&gt; or &lt;!DOCTYPE; an email will usually have a header and be limited to 7-bit US-ASCII.

In general, we consider a cipher to be secure for confidentiality (in the formal sense of IND-CPA) only if an adversary can't distinguish patterns in the plaintext given ciphertext even when the adversary can choose the pattern.  Of course, a 3-letter password admits a very short brute force attack, so we don't consider that to provide any meaningful security.




  I use brute force to decrypt it, but two different passwords lead to a file that contains English intelligible text. Is there a way for me to know which one is the original one?


Again from the adversary's perspective, let's suppose there's no authenticator to verify a decryption, and the pattern you're looking for is that the text is all 7-bit US-ASCII.  Actually you're probably looking for a pattern that is much sparser in the set of all bit strings, but this one is easy to quantify.

We can reasonably model each byte of a wrong decryption as independent uniform random, so there's a $1/2$ chance each byte in a wrong decryption is 7-bit US-ASCII independently, and if your file is $n$ bytes long, there's a $1/2^n$ chance that all bytes in a wrong decryption are 7-bit US-ASCII.

In cryptography we generally consider an event ‘not gonna happen’ when its probability is well below $1/2^{100}$.  If there are $2^t$ keys chosen uniformly at random (say, $2^{256}$ for AES-256 or Salsa20), all it takes is a bit more than $100 + t$ bytes in this model before you stop worrying about any false positives from wrong decryptions—that is, when the probability of a false positive from this pattern is at most the probability of a false positive from a cryptographic authenticator like Poly1305.

But if you do somehow find yourself confronted with two candidate plaintexts that both meet all your criteria, well, there's nothing in cryptography to help discriminate between them, and you'll have to use outside information to help you decide.



This model assumes that the key is substantially smaller than the message and the cryptosystem is not hopelessly broken by related-key attacks; for example, it breaks down when the key is a one-time pad, but it is a reasonable model for, say, AES-CTR or Salsa20.  Specifically, let $K$ be the event of guessing the key correctly, and let $A$ be the event that the ciphertext decrypts under the candidate key to US-ASCII.  The hypothesis is that that $P(A \mid K) = 1$ meaning that the true plaintext is known to be US-ASCII.  By Bayes' rule,

\begin{align*}
  P(K \mid A)
  &amp;= P(K) \frac{P(A \mid K)}{P(A)} \\
  &amp;= P(K)
     \frac{P(A \mid K)}
          {P(A \mid K) P(K) + P(A \mid \lnot K) P(\lnot K)} \\
  &amp;= \frac{P(K)}{P(K) + P(A \mid \lnot K) (1 - P(K))} \\
  &amp;= \frac{1}{1 + P(A \mid \lnot K) (1/P(K) - 1)}.
\end{align*}

The additional modeling assumption about the cipher is that $P(A \mid \lnot K) = 1/2^n$.  If we suppose the key is uniformly distributed among $2^t$ possibilities (e.g., $t = 256$ for Salsa20 or AES256-CTR) so that $P(K) = 1/2^t$, then this reduces to

\begin{equation*}
  P(K \mid A)
    = \frac{1}{1 + 2^{-n} (2^t - 1)}.
\end{equation*}

When $n &lt; t$, meaning the key is underdetermined, this is about $2^n\!/2^t$.  For example, if you have a 100-byte text and a 256-bit key, $P(K \mid A) \approx 1/2^{156}$.  However, when $n &gt; t$ so that $2^n \gg 2^t - 1$, this is about $1 - 2^{-n}$ and rapidly converges to 1 as $n$ increases and overdetermines the key with more bytes of plaintext that must match the pattern.