Determining the encryption method I am trying to reverse engineer a program that encrypts a file with a password.

So far i figured out that the start of the file is always the string "DecryptOK" encrypted with the password of course. 

With a password of "one" an "empty" file turns into 

ED5BDE97BB90DE1377948A31B0EECB4B4D8B


which i assume would be 

ED5B DE97 BB90 DE13 7794 8A31 B0EE CB4B 4D8B


aka "DecryptOK". 

So my question is how do i continue from here? How do i figure out what encryption method is being used? 

Here are some texts i encrypted with the password "one":


123 to 3366 7FAD CAF5 
abc to 3336 7FFD CAA5
cookies to 3334 7FF0 CAA9 33F7 D2D2 60AB 86EB
So my question is how do I continue from here? How do I figure out what encryption method is being used?


It sounds like you, in essence, have black box access to something that allows you to give it a key and a file, and it returns an encryption of that file (after adding some file format around it) using the supplied key.

Your goal is to figure out how the encryption works. There may be legal issues you want to consider before actually doing this, but legal issues aside, you have many options.

Your best option would be to reverse engineer the code that is doing the encryption. Is this software running on your computer or some sort of hardware device? In either case, you should be able to get the code and reverse engineer it by putting it in a disassembler and doing static analysis, or putting it in a debugger and doing dynamic analysis.

Let's pretend, however, that you do not have access in any way to the code. Maybe this black box is running on a remote server that you do not have access to. Your goal then is to figure out what cipher are they using (e.g., AES, DES, 3DES, RC4, etc) and what mode of encryption is being used.

You can start to do this by asking a few questions and using your black box access to start answering the questions.


Is the encryption deterministic? Or in other words, given the same plaintext (or file) and key, does it always return the same result?
If I flip a single bit in the plaintext, how does the resulting ciphertext change?
What happens if you feed a ciphertext in as the plaintext, do you get the original plaintext back? In other words, is the encryption mode the inverse of itself?
By flipping bits at different places, can you figure out the block size? Depending on the mode used, where you flip bits can change what blocks actually change?
Something else to investigate is why does the ciphertext appear to always be double the size of the plaintext? When you pass in 123, which as ASCII would have 3 bytes, why do you get 6 bytes back? Is there some sort of encoding going on?


This is how I would suggest continuing from where you are.