title: Pandas GroupBy memory deallocation
body: <h3>Problem</h3>
  <p>I noticed that memory allocated while iterating through a Pandas <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html" rel="noreferrer">GroupBy</a> object is not deallocated after iteration. I use <code>resource.getrusage(resource.RUSAGE_SELF).ru_maxrss</code> (<a href="https://stackoverflow.com/questions/938733/total-memory-used-by-python-process">second answer in this post for details</a>) to measure the total amount of active memory used by the Python process.</p>
  <pre><code>import resource
  import gc
  
  import pandas as pd
  import numpy as np
  
  i = np.random.choice(list(range(100)), 4000)
  cols = list(range(int(2e4)))
  
  df = pd.DataFrame(1, index=i, columns=cols)
  
  gb = df.groupby(level=0)
  # gb = list(gb)
  for i in range(3):
      print(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1e6)
      for idx, x in enumerate(gb):
          if idx == 0:
              print(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1e6)
      # del idx, x
      # gc.collect()
  </code></pre>
  <p>prints the following total active memory (in gb)</p>
  <pre><code>0.671732
  1.297424
  1.297952
  1.923288
  1.923288
  2.548624
  </code></pre>
  <h3>Solutions</h3>
  <p>Uncommenting <code>del idx, x</code> and <code>gc.collect()</code> fixes the problem. I do however have to <code>del</code> all variables that reference the DataFrames returned by iterating over the groupby (which can be a pain depending on the code in the inner for loop). The new printed memory usages become:</p>
  <pre><code>0.671768
  1.297412
  1.297992
  1.297992
  1.297992
  1.297992
  </code></pre>
  <p>Alternatively I can uncomment <code>gb = list(gb)</code>. The resulting memory usages are roughly the same as those from the previous solution:</p>
  <pre><code>1.32874
  1.32874
  1.32874
  1.32874
  1.32874
  1.32874
  </code></pre>
  <h3>Questions</h3>
  <ol>
  <li>Why is memory for DataFrames resulting from iteration through the groupby not deallocated after iteration is completed?</li>
  <li>Is there a better solution than the two above? If not, which of these two solutions is &quot;better&quot;?</li>
  </ol>