How to check a file was encrypted (really & correctly) I would like to audit how an implementation of an encryption algorithm is really performed with the following given data of the problem:


the encryption mechanism is reversible (this isn’t a signature),
the algorithm is pretended to be AES, but might be implemented correctly or not, or worse something else,
the key is not known (in the case I am interested in, I want to check that all files are really and correctly encrypted, but I could play with specifically designed ones),
I don't have access to the source code.


At least I would like to be able to detect that the file is not encrypted.
Next I would like to be able to detect that the RNG is running in a tiny set or not really random.

In a first approach I thought to make an analysis of the randomness quality of the encrypted file: average value + standard deviation (with a tool like ent).
But I immediatly thought of artificial files with a perfect average value
and standard deviation which are perfectly regular and not the result of any encryption.
Then my first approach is wrong.

The environment in which I will make this audit is a Unix one.
( I cannot use tools or algorithms I cannot read, compile and check myself. )



Practical case:


I would like to check that my iPhone is correctly encrypted and what is the AES key derived from either a constant, my password, a hardware uniq identifier.
I would like to perform the same validation of an iPhone of any staff member which will ask me to make the same validation for his professionnal iPhone. This is a service toward users to provide them trust in what they think is encrypted and has to be.
If you can't get access to the key for at least some sample uses, there's no way to be sure. For example, it's impossible to distinguish AES-128 from AES-256 if you don't have access to the key. That's true of any encryption method: without knowing the key, you can't distinguish the ciphertext from random data of the same length.

A professional auditor would normally get access to some test keys, if they can't normally access those keys through some administration interface.

You can make a statistical test, but all this will tell you is that the encryption is not completely botched or skipped.

If your vendor is not completely dishonest and they claim to have used AES, they probably did use AES. A far more common problem than not using AES is using AES wrong. Here too, you can't be sure that they got it right, but you can at least check for some common problems.

Check how the length of the ciphertext varies depending on the length of the plaintext. The ciphertext should include an initialization vector and an authentication tag, each of which normally adds 16 bytes. If the ciphertext is not 32 bytes larger than the plaintext, something is probably wrong, but there are cases where it can be ok (e.g. for disk encryption where the sector number is used to build unique IVs and no threat requires authentication).

Pass the same inputs in different conditions and make sure that the resulting ciphertexts are completely different. If you can arrange to encrypt multiple messages with the same key, make sure that identical messages result in completely different ciphertexts. This validates that initialization vectors are generated, if not correctly, then at least non-stupidly.

If there's a way for you to submit modified ciphertexts, do that and check that they are rejected with a generic “invalid ciphertext” error, rather than a specific error due to invalid content. This validates that authenticated encryption is used. There are threat models where it's ok not to have authentication, but you need to tread very carefully.

Two things that you definitely cannot know by looking at the output is whether the keys are generated and stored securely. (As opposed to e.g. using a non-cryptographic random generator to generate keys, and writing a copy of the secret keys to an unprotected location.) You can only audit this by looking at the behavior of the system.