title: Parallel processing of a large .csv file in Python
body: <p>I'm processing large CSV files (on the order of several GBs with 10M lines) using a Python script.</p>
  
  <p>The files have different row lengths, and cannot be loaded fully into memory for analysis.</p>
  
  <p>Each line is handled separately by a function in my script. It takes about 20 minutes to analyze one file, and it appears disk access speed is not an issue, but rather processing/function calls.</p>
  
  <p>The code looks something like this (very straightforward). The actual code uses a Class structure, but this is similar:</p>
  
  <pre><code>csvReader = csv.reader(open("file","r")
  for row in csvReader:
     handleRow(row, dataStructure)
  </code></pre>
  
  <p>Given the calculation requires a shared data structure, what would be the best way to run the analysis in parallel in Python utilizing multiple cores?</p>
  
  <p>In general, how do I read multiple lines at once from a .csv in Python to transfer to a thread/process? Looping with <code>for</code> over the rows doesn't sound very efficient.</p>
  
  <p>Thanks!</p>