I have an input of about 2-5 millions strings of about 400 characters each, coming from a stored text file.
I need to check for duplicates before adding them to the list that I check (doesn't have to be a list, can be any other data type, the list is technically a set since all items are unique).

I can expect about 0.01% at max of my data to be non-unique and I need to filter them out.

I'm wondering if there is any faster way for me to check if the item exists in the list rather than:

```
a=[]
for item in data:
    if item not in a:
        a.add(item)

```

I do not want to lose the order.

Would hashing be faster (I don't need encryption)? But then I'd have to maintain a hash table for all the values to check first.
Is there any way I'm missing?

I'm on python 2, can at max go upto python 3.5.