title: python process pool with timeout on each process not all of the pool
body: <p>I need to run many processes, but not all together, for example 4 processes at same time. <code>multiprocessing.Pool</code> is exactly what I need. But the problem is that I need to terminate a process if it lasts more than a timeout (e.g. 3 seconds). <code>Pool</code> just supports wait for a timeout for all processes not each of them. This is what I need:</p>
  
  <pre><code>def f():
      process_but_kill_if_it_takes_more_than_3_sec()
  pool.map(f, inputs)
  </code></pre>
  
  <p>I couldn't find a simple way to use <code>Pool</code> with timeouts. There is <a href="http://eli.thegreenplace.net/2011/08/22/how-not-to-set-a-timeout-on-a-computation-in-python" rel="noreferrer">a solution</a> from Eli Bendersky. It's a function that limits execution time of an arbitrary function via <code>Thread.join(timeout)</code>. It works, (although it's stop method doesn't work well). But this method runs a new unnecessary thread while main thread of process is just waiting, because we need a timeout controller. It should be possible to control all timeouts from a single point, something like this:</p>
  
  <pre><code>import time
  from multiprocessing import Process
  
  
  def f(n):
      time.sleep(n)
  
  timeout = 3
  tasks = [1, 2, 4, 1, 8, 2]
  
  procs = []
  pool_len = 4
  while len(tasks) &gt; 0 or len(procs) &gt; 0:
      if len(tasks) &gt; 0 and len(procs) &lt; pool_len:
          n = tasks.pop(0)
          p = Process(target=f, args=(n,))
          p.start()
          procs.append({'n': n, 'p': p, 't': time.time() + timeout})
      for d in procs:
          if not d['p'].is_alive():
              procs.remove(d)
              print '%s finished' % d['n']
          elif d['t'] &lt; time.time():
              d['p'].terminate()
              procs.remove(d)
              print '%s killed' % d['n']
      time.sleep(0.05)
  </code></pre>
  
  <p>The output should be:</p>
  
  <pre><code>1 finished
  1 finished
  2 finished
  4 killed
  2 finished
  8 killed
  </code></pre>
  
  <p>Question: Is there a way to use Pool to solve this?</p>