Indexing a huge dataset (that does not fit into central memory)
The other approach that I can think of is the obvious one: process the file one line at a time, and consult a dictionary of &quot;seen&quot; lines. What may not be obvious is how to represent the dictionary.
There are plenty of on-disk data structures that would work, and this is a well-researched area given that this is what databases do for a living. Some variant of B-trees, or a hash table (e.g. extensible hashing) would work. Consider Berkeley DB if you need something off the shelf.
If that is too slow, you could use a Bloom filter as an in-memory cache. Then, you need only consult the on-disk data structure if you get a hit in the Bloom filter.