title: multiprocessing.Pool spawning more processes than requested only on Google Cloud
body: <p>I am using Python's multiprocessing.Pool class to distribute tasks among processes. </p>
  
  <p>The simple case works as expected:</p>
  
  <pre><code>from multiprocessing import Pool
  
  def evaluate:
      do_something()
  
  pool = Pool(processes=N)
  for task in tasks:
      pool.apply_async(evaluate, (data,))
  </code></pre>
  
  <p>N processes are spawned, and they continually work through the tasks that I pass into apply_async. Now, I have another case where I have many different very complex objects which each need to do computationally heavy activity. I initially let each object create its own multiprocessing.Pool on demand <em>at the time it was completing work</em>, but I eventually ran into OSError for having too many files open, even though I would have assumed that the pools would get garbage collected after use.</p>
  
  <p>At any rate, I decided it would be preferable anyway for each of these complex objects to share the same Pool for computations:</p>
  
  <pre><code>from multiprocessing import Pool
  
  def evaluate:
      do_something()
  
  pool = Pool(processes=N)
  
  class ComplexClass:
  
      def work:
          for task in tasks:
              self.pool.apply_async(evaluate, (data,))
  
  objects = [ComplexClass() for i in range(50)]
  
  for complex in objects:
      complex.pool = pool
  
  
  while True:
      for complex in objects:
          complex.work()
  </code></pre>
  
  <p>Now, when I run this on one of my computers (OS X, Python=3.4), it works just as expected. N processes are spawned, and each complex object distributes their tasks among each of them. However, when I ran it on another machine (Google Cloud instance running Ubuntu, Python=3.5), it spawns an enormous number of processes (>> N) and the entire program grinds to a halt due to contention.</p>
  
  <p>If I check the pool for more information:</p>
  
  <pre><code>import random
  random_object = random.sample(objects, 1)
  print (random_object.pool.processes)
  
  &gt;&gt;&gt; N
  </code></pre>
  
  <p>Everything looks correct. But it's clearly not. Any ideas what may be going on?</p>
  
  <p><strong>UPDATE</strong> </p>
  
  <p>I added some additional logging. I set the pool size to 1 for simplicity. Within the pool, as a task is being completed, I print the current_process() from the multiprocessing module, as well as the pid of the task using os.getpid(). It results in something like this:</p>
  
  <pre><code>&lt;ForkProcess(ForkPoolWorker-1, started daemon)&gt;, PID: 5122
  &lt;ForkProcess(ForkPoolWorker-1, started daemon)&gt;, PID: 5122
  &lt;ForkProcess(ForkPoolWorker-1, started daemon)&gt;, PID: 5122
  &lt;ForkProcess(ForkPoolWorker-1, started daemon)&gt;, PID: 5122
  ...
  </code></pre>
  
  <p>Again, looking at actually activity using htop, I'm seeing many processes (one per object sharing the multiprocessing pool) all consuming CPU cycles as this is happening, resulting in so much OS contention that progress is very slow. 5122 appears to be the parent process.</p>