title: Reading a huge .csv file
body: <p>I'm currently trying to read data from .csv files in Python 2.7 with up to 1 million rows, and 200 columns (files range from 100mb to 1.6gb). I can do this (very slowly) for the files with under 300,000 rows, but once I go above that I get memory errors. My code looks like this:</p>
  
  <pre><code>def getdata(filename, criteria):
      data=[]
      for criterion in criteria:
          data.append(getstuff(filename, criteron))
      return data
  
  def getstuff(filename, criterion):
      import csv
      data=[]
      with open(filename, "rb") as csvfile:
          datareader=csv.reader(csvfile)
          for row in datareader: 
              if row[3]=="column header":
                  data.append(row)
              elif len(data)&lt;2 and row[3]!=criterion:
                  pass
              elif row[3]==criterion:
                  data.append(row)
              else:
                  return data
  </code></pre>
  
  <p>The reason for the else clause in the getstuff function is that all the elements which fit the criterion will be listed together in the csv file, so I leave the loop when I get past them to save time.</p>
  
  <p>My questions are:</p>
  
  <ol>
  <li><p>How can I manage to get this to work with the bigger files?</p></li>
  <li><p>Is there any way I can make it faster?</p></li>
  </ol>
  
  <p>My computer has 8gb RAM, running 64bit Windows 7, and the processor is 3.40 GHz (not certain what information you need).</p>