Need fast csv parser for Python to parse 80GB csv file
I would suggest you to create a thread pool and do line.split() in separate thread. Playing with thread pool size can give you balance between processing speed and consumed resources.