title: Correct way to implement producer consumer pattern in python multiprocessing Pool
body: <p>I want to implement <a href="https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem" rel="nofollow noreferrer">producer-consumer pattern</a> by using <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool" rel="nofollow noreferrer"><code>multiprocessing.pool.Pool</code></a></p>
  <p>Since the <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.JoinableQueue" rel="nofollow noreferrer"><code>JoinableQueue</code></a> cannot be used in <code>Pool</code> (would claim <code>RuntimeError: JoinableQueue objects should only be shared between processes through inheritance</code>), I have to use <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.sharedctypes.multiprocessing.Manager" rel="nofollow noreferrer"><code>multiprocessing.Manager()</code></a> inspired by <a href="https://stackoverflow.com/a/9928191/">this answer</a>.</p>
  <p>The question is: now the program may hang when consumer jobs are larger than producer jobs.</p>
  <pre class="lang-py prettyprint-override"><code>import queue
  import random
  from multiprocessing import Manager, Pool
  
  
  def consumer(q):
      while True:
          try:
              res = q.get(block=False)
              if res is None:
                  break
              print(f'Consume {res}')
          except queue.Empty:
              pass
  
  
  def producer(q, food):
      for i in range(2):
          res = f'{food} {i}'
          print(f'Produce {res}')
          q.put(res)
      q.put(None) # sentinel
  
  
  if __name__ == &quot;__main__&quot;:
      with Pool() as pool:
          jobs = 2
          foods = ['apple', 'banana', 'melon', 'salad']
          q = Manager().Queue()
          [pool.apply_async(func=consumer, args=(q, )) for _ in range(jobs + 1)]  # would hang
  
          # would not hang only when the consumer jobs is equal or less than the producer jobs
          # [pool.apply_async(func=consumer, args=(q, )) for _ in range(jobs)] 
  
          [
              pool.apply_async(func=producer, args=(q, random.choice(foods)))
              for _ in range(jobs)
          ]
  
          pool.close()
          pool.join()
  </code></pre>
  <p>Seems like those <em>extra</em> consumers can't get the <em>sentinel</em> and just wait there forever.</p>
  <p>So what's the elegant way to implement the producer-consumer pattern in <code>multiprocessing.pool.Pool</code>?</p>
  <p>Or is it only possible with <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process" rel="nofollow noreferrer"><code>multiprocessing.Process</code></a> + <code>JoinableQueue</code>?</p>