title: Accurate memory usage estimate of a pandas dataframe
body: <p>I discovered a mismatch between memory usage reported by pandas and python, and actual memory usage of a python process reported by the OS (Windows, in my case).</p>
  
  <p>After I import numpy, pandas and some other standard libs (like os, sys), RAM consumption of the own process, reported psutil and consistent with Task Manger, equals <strong>89 020 K</strong>.</p>
  
  <p>Then I create example dataframe consisting of both object and numerical dtypes like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>    l=5_000_000
      df = pd.DataFrame(np.random.uniform(-100,100,size=l), columns=list('A'))
      df['B']='dsgdb9nsdalWofsT'
      df['C']=np.random.randint(0,130000,size=l).astype(np.int64)
      df['D']=np.random.randint(0,130000,size=l).astype(np.int64)
      df['E']='2JHH'
  </code></pre>
  
  <p>After creating the dataframe psutil reports <strong>286 696 K</strong> (again, consistent with the Windows Task Manager).</p>
  
  <p>When running df.info() I get</p>
  
  <pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;
  
  RangeIndex: 5000000 entries, 0 to 4999999
  Data columns (total 5 columns):
  A    float64
  B    object
  C    int64
  D    int64
  E    object
  dtypes: float64(1), int64(2), object(2)
  memory usage: 190.7+ MB
  </code></pre>
  
  <p>which is very logical so far.</p>
  
  <p>Now, let's get</p>
  
  <pre class="lang-py prettyprint-override"><code>    df.memory_usage(index=True,deep=False).sum()/1024
  </code></pre>
  
  <blockquote>
    <p>195 312.6</p>
  </blockquote>
  
  <p>And what about</p>
  
  <pre class="lang-py prettyprint-override"><code>    df.memory_usage(index=True,deep=True).sum()/1024 ?
  </code></pre>
  
  <p>Terrific </p>
  
  <blockquote>
    <p>771 484.5</p>
  </blockquote>
  
  <p>which I can not explain.</p>
  
  <p>According to pandas' docs,</p>
  
  <blockquote>
    <p>Without deep introspection a memory estimation is made based in column
    dtype and number of rows assuming values consume the same memory
    amount for corresponding dtypes. With deep memory introspection, a
    real memory usage calculation is performed at the cost of
    computational resources.</p>
  </blockquote>
  
  <p>So we could expect <strong>deep</strong> option to be more precise. <strong>How is it not? What am I missing? Why is it giving wrong results? Is this a feature or a bug?</strong></p>
  
  <p>I also have to say that python itself gives wrong results when using <strong>sys</strong> module:</p>
  
  <pre class="lang-py prettyprint-override"><code>    sys.getsizeof(df)/1024
  </code></pre>
  
  <blockquote>
    <p>771 484.52</p>
  </blockquote>
  
  <p>I am testing using Jupyter notebook server 5.7.4, my specs are</p>
  
  <p><em>Python version: 3.7.2 (default, Feb 21 2019, 17:35:59) [MSC v.1915 64 bit (AMD64)]</em></p>
  
  <p><em>platform: AMD64</em></p>
  
  <p><em>system: Windows-10-10.0.17763-SP0</em></p>
  
  <p><em>pandas: 0.25.0</em></p>
  
  <p><em>numpy: 1.16.2</em></p>