title: Memory usage keep growing with Python's multiprocessing.pool
body: <p>Here's the program:</p>
  
  <pre><code>#!/usr/bin/python
  
  import multiprocessing
  
  def dummy_func(r):
      pass
  
  def worker():
      pass
  
  if __name__ == '__main__':
      pool = multiprocessing.Pool(processes=16)
      for index in range(0,100000):
          pool.apply_async(worker, callback=dummy_func)
  
      # clean up
      pool.close()
      pool.join()
  </code></pre>
  
  <p>I found memory usage (both VIRT and RES) kept growing up till close()/join(), is there any solution to get rid of this? I tried maxtasksperchild with 2.7 but it didn't help either.</p>
  
  <p>I have a more complicated program that calles apply_async() ~6M times, and at ~1.5M point I've already got 6G+ RES, to avoid all other factors, I simplified the program to above version.</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>Turned out this version works better, thanks for everyone's input:</p>
  
  <pre><code>#!/usr/bin/python
  
  import multiprocessing
  
  ready_list = []
  def dummy_func(index):
      global ready_list
      ready_list.append(index)
  
  def worker(index):
      return index
  
  if __name__ == '__main__':
      pool = multiprocessing.Pool(processes=16)
      result = {}
      for index in range(0,1000000):
          result[index] = (pool.apply_async(worker, (index,), callback=dummy_func))
          for ready in ready_list:
              result[ready].wait()
              del result[ready]
          ready_list = []
  
      # clean up
      pool.close()
      pool.join()
  </code></pre>
  
  <p>I didn't put any lock there as I believe main process is single threaded (callback is more or less like a event-driven thing per docs I read).</p>
  
  <p>I changed v1's index range to 1,000,000, same as v2 and did some tests - it's weird to me v2 is even ~10% faster than v1 (33s vs 37s), maybe v1 was doing too many internal list maintenance jobs. v2 is definitely a winner on memory usage, it never went over 300M (VIRT) and 50M (RES), while v1 used to be 370M/120M, the best was 330M/85M. All numbers were just 3~4 times testing, reference only.</p>