title: Why does Python memory usage goes down after while?
body: <p>Specifically, I recently had to work with a large dataset (~3 GB) and to get a sense of the speed of the loading process (i.e. running <code>df = pd.read_csv(file)</code>), I opened a task manager.</p>
  
  <p>As I thought, I saw my Python process' memory usage going constantly up. And around the time when it reached approximately 3 GB, it was fully loaded. Logical.</p>
  
  <p>Then, after doing some other tasks not involving Python, let's say answering emails or browsing online, I noticed the Python memory usage in my task manager was way down, as if I hadn't loading any data yet. But I still had my notebook's kernel running (no interruption there).</p>
  
  <p>Strangely, even after this lowering of memory usage, my code is still functional: I can show information from my dataframe by running for example:</p>
  
  <p><code>df.loc[100000:101000,['col1','col2','col3']]</code>,</p>
  
  <p>which shows these 1000 lines instantly.</p>
  
  <p>Or if I run:</p>
  
  <p><code>df.info()</code>,</p>
  
  <p>which gives me this:</p>
  
  <pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;
  RangeIndex: 3228691 entries, 0 to 3228690
  Columns: 117 entries, first_column to last_column
  dtypes: category(49), float64(51), object(17)
  memory usage: 1.8+ GB
  </code></pre>
  
  <p>Which is strange, because when I run <code>df.info(memory_usage='deep')</code> instead of just <code>df.info()</code>, then it takes much longer and I can see in my task manager that the Python process' memory usage goes up until it reaches ~1.5 GB. Here is the output then:</p>
  
  <pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;
  RangeIndex: 3228691 entries, 0 to 3228690
  Columns: 117 entries, first_column to last_column
  dtypes: category(49), float64(51), object(17)
  memory usage: 4.9 GB
  </code></pre>
  
  <p>Again, this is strange because it reaches ~1.5 GB in memory usage, but then the output gives 4.9 GB as memory usage for a file that's ~3 GB.</p>
  
  <p>I guess Python doesn't keep in memory what has not been used for some time, but I'm curious what are the rules here? How does this work? Is this a memory management process from Pandas, or is the the one from Python?</p>