How to detect duplicate values of sensitive data that needs to be encrypted This was a question that was posted on the information security exchange but was suggested I may get a better answer here.

The link suggested in the original post is similar, but I think a search index for a simple comparison here is potentially not efficient.

I have a scenario where text values that are considered sensitive (not passwords) need to be encrypted for persistence onto a database.

The key requirement here is being able to identify if a value being received has been received in the past.

One solution that is being considered is to Encrypt the each value using the AES encryption algorithm with the same key and IV to generate the same cipher for each value received making it easy to detect if this value has been received in the past. However I have concerns regarding the strength and security of this solution.

The amount of data to be encrypted is quite small, &lt; 50 characters

So the question is, what other options are available to meet these requirements in a better way?
First, it is important to note that one important property of encryption is that you cannot determine if the same ciphertext is encrypted twice. However, assuming that you wish to give up on this property but you want it to be completely secure otherwise, then you still have to be careful.

Using regular encryption with the same IV is problematic. If you are using CTR mode then this completely breaks the encryption. If you are using CBC mode then if the first block is the same but the second is different, then you will be able to detect this.

The best solution is to use the AES-SIV or AES-GCM-SIV modes of operation with a fixed IV/nonce. These modes have the property that when encrypting different messages with the same IV, then you cannot learn anything apart from the fact that the plaintexts are different. Of course, if you encrypt the same message with the same IV then you get the same result.