title: Python: subprocess memory leak
body: <p>I want to run a serial program on multiple cores at the same time and I need to do that multiple time (in a loop).
  I use subprocess.Popen to distribute the jobs on the processors by limiting the number of jobs to the number of available processors. I add the jobs to a list and then I check with poll() if the jobs are done, I remove them from the list and continue the submission until the total number of jobs are completed.</p>
  
  <p>I have been looking on the web and found a couple of interesting scripts to do that and came out with my adapted version:</p>
  
  <pre><code>nextProc = 0
  processes = []
  while (len(processes) &lt; limitProc):     # Here I assume that limitProc &lt; ncores
    input = filelist[nextProc]+'.in'      # filelist: list of input file
    output = filelist[nextProc]+'.out'    # list of output file
    cwd = pathlist[nextProc]              # list of paths
    processes.append(subprocess.Popen(['myProgram','-i',input,'-screen',output],cwd=cwd,bufsize=-1))
    nextProc += 1
    time.sleep(wait)
  
  while (len(processes) &gt; 0):                     # Loop until all processes are done
    time.sleep(wait)
    for i in xrange(len(processes)-1, -1, -1):    # Remove processes done (traverse backward) 
      if processes[i].poll() is not None:
        del processes[i]
      time.sleep(wait) 
    while (len(processes) &lt; limitProc) and (nextProc &lt; maxProcesses):    # Submit new processes
      output = filelist[nextProc]+'.out'
      input = filelist[nextProc]+'.in'
      cwd = pathlist[nextProc]
      processes.append(subprocess.Popen(['myProgram','-i',input,'-screen',output],cwd=cwd,bufsize=-1))
      nextProc += 1
      time.sleep(wait)
  
  print 'Jobs Done'
  </code></pre>
  
  <p>I run this script in a loop and the problem is that the execution time increases from one step to another. Here is the graph: <a href="http://i62.tinypic.com/2lk8f41.png" rel="nofollow">http://i62.tinypic.com/2lk8f41.png</a></p>
  
  <p>myProgram time execution is constant.
  I'd be so glad if someone could explain me what is causing this leak.</p>
  
  <p>Thanks a lot,
  Begbi</p>