title: Python memory consumption on Linux: physical and virtual memory are growing while the heap size remains the same
body: <p>I'm working on the some kind of a system service (actually it's just a log parser) written in Python. This program should work continuously for a long time (hopefully I mean days and weeks without failures and needs of restart). That's why I am concerned about memory consumption.</p>
  
  <p>I put together different information about process memory usage from different sites into one simple function:</p>
  
  <pre><code>#!/usr/bin/env python
  from pprint import pprint
  from guppy import hpy
  from datetime import datetime
  import sys
  import os
  import resource
  import re
  
  def debug_memory_leak():
      #Getting virtual memory size 
      pid = os.getpid()
      with open(os.path.join("/proc", str(pid), "status")) as f:
          lines = f.readlines()
      _vmsize = [l for l in lines if l.startswith("VmSize")][0]
      vmsize = int(_vmsize.split()[1])
  
      #Getting physical memory size  
      pmsize = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
  
      #Analyzing the dynamical memory segment - total number of objects in memory and heap size
      h = hpy().heap()
      if __debug__:
          print str(h)
      m = re.match(
          "Partition of a set of ([0-9]+) objects. Total size = ([0-9]+) bytes(.*)", str(h))
      objects = m.group(1)
      heap = int(m.group(2))/1024 #to Kb
  
      current_time = datetime.now().strftime("%H:%M:%S")
      data = (current_time, objects, heap, pmsize, vmsize)
      print("\t".join([str(d) for d in data]))
  </code></pre>
  
  <p>This function has been used to study the dynamics of the memory consumption of my long-playing process, and I still cannot explain its behavior. You can see that the heap size and total amount of the objects did not changed while the physical and virtual memory increased by 11% and 1% during these twenty minutes.</p>
  
  <p><strong>UPD:</strong> The process has been working for almost 15 hours by this moment. The heap is still the same, but the physical memory increased sixfold and the virtual memory increased by 50%. The curve is seem to be linear excepting the strange outliers at 3:00 AM: </p>
  
  <blockquote>
    <p>Time Obj      Heap    PhM    VM</p>
    
    <p>19:04:19  31424   3928    5460    143732</p>
    
    <p>19:04:29  30582   3704    10276   158240</p>
    
    <p>19:04:39  30582   3704    10372   157772</p>
    
    <p>19:04:50  30582   3709    10372   157772</p>
    
    <p>19:05:00  30582   3704    10372   157772</p>
    
    <p>(...)</p>
    
    <p>19:25:00  30583   3704    11524   159900</p>
    
    <p>09:53:23  30581   3704    62380   210756</p>
  </blockquote>
  
  <p>I wonder what is going on with the address space of my process. The constant size of heap suggests that all of the dynamical objects are deallocated correctly. But I have no doubt that growing memory consumption will affect the sustainability of this life-critical process in the long run.</p>
  
  <p><img src="https://i.stack.imgur.com/9gsGB.png" alt="enter image description here"></p>
  
  <p>Could anyone clarify this issue please? Thank you.</p>
  
  <p>(I use RHEL 6.4, kernel 2.6.32-358 with Python 2.6.6)</p>