Efficiently filter a large (100gb+) csv file (v1)
Write code that does the conversion with a small portion of your data defined in the python module. That makes you focus on the data structures and if they fit to the problem. If you remove the code for file handling, logging - all that is not specific to your conversion problem - only a couple lines of code remain. That is what needs your attention and ours attention.

If that simple conversion work, add code that reads the data from external sources to populate the functions for your conversion code. Instead of splitting strings prefer code that is made for this common format: use built-in csv module. Separate file handling code from any algorithms. 

Next create code that writes your intermediate output to external sources - again use csv module.

Separate any part/idea into their own place and give them names that makes readers understand their purpose. This is actually the main problem for any reader of your code. 

What will make your code faster?


use generators to keep memory usage low and list comprehension for beauty and awesome filtering


What will make you solve it right?


If you split the problem into many parts, you can test any part itself. What makes you think your conversion is done properly at a scale of 2GB input files and large output files? Did you verified every line in output?
Apply testing with smaller data, gives you confidence for larger data.


What will make you giant leaps?


Go get IPython notebook. Write smaller parts and test execution time.


If you have understood your problem, use python libraries made for your problem and write the necessary glue code.


csvkit, pandas


Learn from similar questions/answers on CodeReview


Find the peak stock price for each company from CSV data