title: Do aggregations and statistics on a very large Pandas Data frame
body: <p>I have a very large (larger than cheaper RAM) CSV data set that I'm operating with pandas. Now I'm doing something like</p>
  
  <pre><code>df = pd.read_csv('verylargefile.csv', chunksize=10000)
  for df_chunk in df:
      df_chunk['new_column'] = df_chunk['old_column'].apply(my_func)
      # do other operations and filters...
      df_chunk.to_csv('processed.csv', mode='a')
  </code></pre>
  
  <p>so I can do what I need to operate on the data set and save the output to another file. </p>
  
  <p>The problem starts with doing some groupings and statistics on this data set: I need to calculate the mean, standard deviation and histograms of the entire data set and also graph the results, tendencies, use statsmodels etc. As the samples are not going to be homogeneous I cannot calculate the statistics. </p>
  
  <pre><code>df.groupby('column_1').sum()
  TypeError: Cannot groupby on a TextFileReader
  </code></pre>
  
  <p>I don't have the usual options of selecting only a few columns and I don't see how storing my data on HDF would help. Is there a way?</p>