ipython notebook - memories disappear after running some function
Question: After runing some (imported) function that returns a variable, say X. The
memory usage increases much more than the size of X. And even I `del X` or all
variables by `%reest`. The memory usage is still much larger than before
running the function. And I can only clear all memory by restarting the
notebook.

I am confused, it seems the notebook saves the local variables used only
inside a (imported) function? Which seems unnecessary for me. And if this is
not a bug, how can I trace those large (local) variables used only in the
called function, and clear those memories?


Answer: As you said in your comment above, you are wondering why the iPython notebook
process doesn't release memory back to the OS. The answer is somewhat
complicated, and goes to the heart of how memory is managed in both the OS and
in the process.

When a process asks for more memory, the OS has to find an appropriately sized
chunk to map into the process's address space -- let's say 8KB (although it is
probably some large multiple of that). This is always done on page-boundaries
because of a number of factors, including how virtual memory works. TLDP has a
fairly decent description of how and why this is done: [Chapter 3 Memory
Management](http://www.tldp.org/LDP/tlk/mm/memory.html)

Meanwhile, the process didn't actually want 8KB for a new object, it only
wanted 128B. So it gets the 8KB chunk and adds it to the in-process pool of
available memory. Note that this is normally a portion of the process's
address space that is contiguous so that the 8KB boundaries of the OS page are
invisible to the process.

The in-process memory manager then allocates 128B to be used by whoever asked
for it. This happens over and over as objects are created. When they are
destroyed, the memory is returned _to the in-memory pool of available memory._

So if it's free, why not give it back to the OS? Well, for one thing it's
almost certainly not on a page boundary, so the OS wouldn't know what to do
with it. It's also almost certain that the freed memory is not at _the end_ of
the in-memory pool. So releasing it would create a discontinuity in the pool's
address space, which could then become a major PITA for the in-process memory
manager to tip-toe around.

In certain circumstances, where you _know_ you need some big chunk of memory,
that it will be used in a way that is independent from the rest of the in-
process memory management, and that when you're done with it you _can_ give it
back to the OS, then it is possible to reduce the process's memory size. This
is most often seen when a file is
[mmap](https://en.wikipedia.org/wiki/Mmap)'ed into the process image, worked
on, and then unmapped. But that's special case stuff.

