title: Determining when a ThreadPool has finished processing a queue
body: <p>I am trying to implement a thread pool that processes a task queue using <code>ThreadPool</code> and <code>Queue</code>. It begins with an initial queue of tasks, and then each of the tasks may also push additional tasks onto the task queue. The problem is I don't know how to block until the queue is empty and the thread pool has finished processing, but still check the queue and submit any new tasks to the thread pool that were pushed onto the queue. I can't simply call <code>ThreadPool.join()</code>, because I need to keep the pool open for new tasks.</p>
  
  <p>For example:</p>
  
  <pre><code>from multiprocessing.pool import ThreadPool
  from Queue import Queue
  from random import random
  import time
  import threading
  
  queue = Queue()
  pool = ThreadPool()
  stdout_lock = threading.Lock()
  
  def foobar_task():
      with stdout_lock: print "task called" 
      if random() &gt; .25:
          with stdout_lock: print "task appended to queue"
          queue.append(foobar_task)
      time.sleep(1)
  
  # set up initial queue
  for n in range(5):
      queue.put(foobar_task)
  
  # run the thread pool
  while not queue.empty():
      task = queue.get() 
      pool.apply_async(task)
  
  with stdout_lock: print "pool is closed"
  pool.close()
  pool.join()
  </code></pre>
  
  <p>This outputs:</p>
  
  <pre><code>pool is closed
  task called
  task appended to queue
  task called
  task appended to queue
  task called
  task appended to queue
  task called
  task appended to queue
  task called
  task appended to queue
  </code></pre>
  
  <p>This exits the while loop before the foobar_tasks have appended to the queue, so the appended tasks are never submitted to the thread pool. I can't find any way to determine if the thread pool still has any active worker threads. I tried the following:</p>
  
  <pre><code>while not queue.empty() or any(worker.is_alive() for worker in pool._pool):
      if not queue.empty():
          task = queue.get() 
          pool.apply_async(task)
      else:   
          with stdout_lock: print "waiting for worker threads to complete..."
          time.sleep(1)
  </code></pre>
  
  <p>But it seems that <code>worker.is_alive()</code> always returns true, so this goes into an infinite loop.</p>
  
  <p>Is there a better way to do this?</p>