title: Pandas Chunksize iterator
body: <p>I have a 1GB, 70M row file which anytime I load it all it runs out of memory. I have read in 1000 rows and been able to prototype what I'd like it to do.</p>
  <p>My problem is not knowing how to get the next 1000 rows and apply my logic and then continue to run through my file until it finishes the last rows. I've read about chunksizing, although I can't figure out how to continue the iteration of the chunksizing.</p>
  <p>Ideally, it would flow like such:</p>
  <p>1)read in first 1000 rows
  2)filter data based on criteria
  3)write to csv
  4)repeat until no more rows</p>
  <p>Here's what i have so far:</p>
  <pre><code>import pandas as pd
  data=pd.read_table('datafile.txt',sep='\t',chunksize=1000, iterator=True)
  data=data[data['visits']&gt;10]
  with open('data.csv', 'a') as f:
      data.to_csv(f,sep = ',', index=False, header=False)
  </code></pre>