title: multiprocessing: pool: wait for all results, but process individual results immediately
body: <p>I have a list of tasks I would like to distribute to workers in a pool. I would like to achieve two things:</p>
  
  <ol>
  <li>Whenever a worker is finished, process the result immediately</li>
  <li>Have a simple way to wait for all workers to finish.</li>
  </ol>
  
  <p>Using fapply_async, I can easily achieve the first goal. Whenever a worker has finished, the callback gets invoked. However, to achieve the second goal, the only solution I could come up with basically just polls the AsyncResults until they are all ready().</p>
  
  <p>Using map_async, I can easily achieve the second goal. However, the callback gets invoked only once, when all workers have finished. I believe I understand the reason for this (the order of the results is relevant).</p>
  
  <p>Is there some solution I'm missing that would achieve both goal 1 and 2?</p>
  
  <p>Here's my test code:</p>
  
  <pre><code>#!/usr/bin/python3
  
  import multiprocessing
  import time
  import random
  
  def worker(src):
      time.sleep(0.2)
      # src is apply_async or map_async
      return (src, random.randint(1, 100))
  
  def map_async_example():
      tasks = ['map_async'] * 20
      with multiprocessing.Pool(processes=4) as pool:
          r = pool.map_async(worker, tasks, callback=print)
          r.wait()
  
  def fapply_async_example():
      tasks = [('fapply_async',)] * 20
      with multiprocessing.Pool(processes=4) as pool:
          ars = []
          for t in tasks:
              ar = pool.apply_async(worker, t, callback=print)
              ars.append(ar)
          # Wait for all AsyncResults to become ready()
          while len(ars) &gt; 0:
              time.sleep(0.5)
              # Keep only the not-ready results
              ars = [ar for ar in ars if not ar.ready()]
  
  def main():
      # One list of 20 results
      print('===============')
      print('Using map_async')
      print('===============')
      map_async_example()
  
      # 20 results
      print('==================')
      print('Using fapply_async')
      print('==================')
      fapply_async_example()
  
  if __name__ == '__main__':
      main()
  </code></pre>