Let's say I have a huge list containing random numbers for example 

```
L = [random.randrange(0,25000000000) for _ in range(1000000000)]

```

I need to get rid of the duplicates in this list 

I wrote this code for lists containing a smaller number of elements 

```
def remove_duplicates(list_to_deduplicate):
seen = set()
result=[]
for i in list_to_deduplicate:
    if i not in seen:
        result.append(i)
        seen.add(i)
return result

```

In the code above I create a set so I can memorize what numbers have already appeared in the list I'm working on if the number is not in the set then I add it to the result list I need to return and save it in the set so it won't be added again in the result list

Now for 1000000 number in a list all is good I can get a result fast but for numbers superior to let's say 1000000000 problems arise I need to use the different cores on my machine to try and break up the problem and then combine the results from multiple processes 

My first guess was to make a set accessible to all processes but many complications will arise
How can a process read while maybe another one is adding to the set and I don't even know if it is possible to share a set between processes I know we can use a Queue or a pipe but I'm not sure on how to use it 

Can someone give me an advice on what is the best way to solve this problem
I am open to any new idea