title: Pandas GroupBy Mean of Large DataSet in CSV
body: <p>A common SQLism is "Select A, mean(X) from table group by A" and I would like to replicate this in pandas. Suppose that the data is stored in something like a CSV file and is too big to be loaded into memory.</p>
  
  <p>If the CSV could fit in memory a simple two-liner would suffice:</p>
  
  <pre><code>data=pandas.read_csv("report.csv")
  mean=data.groupby(data.A).mean()
  </code></pre>
  
  <p>When the CSV cannot be read into memory one might try:</p>
  
  <pre><code>chunks=pandas.read_csv("report.csv",chunksize=whatever)
  cmeans=pandas.concat([chunk.groupby(data.A).mean() for chunk in chunks])
  badMeans=cmeans.groupby(cmeans.A).mean()
  </code></pre>
  
  <p>Except that the resulting cmeans table contains repeated entries for each distinct value of A, one for each appearance of that value of A in distinct chunks (since read_csv's chunksize knows nothing about the grouping fields). As a result the final badMeans table has the wrong answer... it needs to compute a weighted average mean.</p>
  
  <p>So a working approach seems to be something like:</p>
  
  <pre><code>final=pandas.DataFrame({"A":[],"mean":[],"cnt":[]})
  for chunk in chunks:
      t=chunk.groupby(chunk.A).sum()
      c=chunk.groupby(chunk.A).count()
      cmean=pandas.DataFrame({"tot":t,"cnt":c}).reset_index()
      joined=pandas.concat(final,cmean)
      final=joined.groupby(joined.A).sum().reset_indeX()
  
  mean=final.tot/final.cnt
  </code></pre>
  
  <p>Am I missing something? This seems insanely complicated... I would rather write a for loop that processes a CSV line by line than deal with this. There has to be a better way.</p>