I have a lot of txt files, around 10GB. What should I use in my program to merge them into one file without duplicates? I want to make sure each line in my output file will be unique.

I was thinking about making some kind of hash tree and use MPI. I want it to be effective.