Removing entries with duplicate columns from large .vcf files
Some notes:


Run it through pycodestyle to learn how your style differs (at least in machine detectable ways) from idiomatic Python. This makes code not only easier to review, but also easier for seasoned Python developers to read and maintain. I would recommend not silencing any of the complaints, and instead fixing every one of them. It quickly becomes second nature to write idiomatic Python that way.
Abbr. var nams hindr rdability, makin it hardr to rd. Unless D1 is absolutely clear from context to anyone in your field I would rename it until it is. The same of course goes for f, which could be semi-tolerable if your script only ever processed one file, but is baffling if you're processing several. "More typing" is not an argument against this, because there are editors which will happily auto-complete any identifier for you.
You have a nested loop where both loops use the same instance variable. That means that the outer loop variable is useless.
You have huge files you want to process, so you really, really don't want to read whole files before starting processing. The way to avoid that is to process each line as you read it, as in for line in file_pointer rather than lines = file.readlines() + a loop.
The main parts of this should be split into functions, called by a short main function which is triggered only if __name__ == '__main__': - this rather unfortunate pattern is idiomatic Python, and allows reuse of the functions and objects in your module.
You have completely open ended except clauses, which is almost universally a bad idea. Exception handling is expensive, and you'll be much better off checking for the existence of an index than throwing an exception and catching it again immediately. In general, the anti-pattern is to use exceptions for flow control.
There are several patterns in use which may be fine for most processing, but which are not optimal for large amounts of data, such as splitting and joining of strings, regular expressions, and accumulating strings and then writing once rather than just writing every substring. From a cursory look it seems that your processing would be very much quicker if done with a database, if at all possible.
You are taking care to close some files by using with open(), but the second D2 is not closed.
Reusing variables for different things also makes code really hard to read, and it a common misconception that it is more efficient. Maybe in some niche corner cases, but for this kind of processing you will be much better off with evidence-based optimisation - prove where the code is slow with a profiler and fix those parts.