Parallel processing of a large .csv file in Python
<p>I'm processing large CSV files (on the order of several GBs with 10M lines) using a Python script.</p>

<p>The files have different row lengths, and cannot be loaded fully into memory for analysis.</p>

<p>Each line is handled separately by a function in my script. It takes about 20 minutes to analyze one file, and it appears disk access speed is not an issue, but rather processing/function calls.</p>

<p>The code looks something like this (very straightforward). The actual code uses a Class structure, but this is similar:</p>

<pre><code>csvReader = csv.reader(open("file","r")
for row in csvReader:
   handleRow(row, dataStructure)
</code></pre>

<p>Given the calculation requires a shared data structure, what would be the best way to run the analysis in parallel in Python utilizing multiple cores?</p>

<p>In general, how do I read multiple lines at once from a .csv in Python to transfer to a thread/process? Looping with <code>for</code> over the rows doesn't sound very efficient.</p>

<p>Thanks!</p>
