title: Subprocess completes but still doesn't terminate, causing deadlock
body: <p>Ok, since there are currently no answer's I don't feel too bad doing this.
  While I'm still interested in what is actually happening behind the scenes to cause this problem, my most urgent questions are those specified in update 2. Those being,</p>
  
  <p>What are the differences between a <code>JoinableQueue</code> and a <code>Manager().Queue()</code> (and when should you use one over the other?). And importantly, is it safe to replace one for the other, in this example?</p>
  
  <hr>
  
  <p>In the following code, I have a simple process pool. Each process is passed the process queue (<code>pq</code>) to pull data to be processed from, and a return-value queue (<code>rq</code>) to pass the returned values of the processing back to the main thread. If I don't append to the return-value queue it works, but as soon as I do, for some reason the processes are blocked from stopping. In both cases the processes <code>run</code> methods return, so it's not <code>put</code> on the return-queue blocking, but in the second case the processes themselves do not terminate, so the program deadlocks when I <code>join</code> on the processes. Why would this be?</p>
  
  <p><strong>Updates:</strong></p>
  
  <ol>
  <li><p><strong>It seems to have something to with the number of items in the queue.</strong><br/><br/>
  On my machine at least, I can have up to 6570 items in the queue and it actually works, but any more than this and it deadlocks.</p></li>
  <li><p><strong>It seems to work with <code>Manager().Queue()</code>.</strong><br/><br/>Whether it's a limitation of <code>JoinableQueue</code> or just me misunderstanding the differences between the two objects, I've found that if I replace the return queue with a <code>Manager().Queue()</code>, it works as expected. What are the differences between them, and when should you use one over the other?</p></li>
  <li><p><strong>The error does not occur if I'm consuming from <code>rq</code></strong><br/><br/>Oop. There was an answer here for a moment, and as I was commenting on it, it disappeared. Anyway one of the things it said was questioning whether, if I add a consumer this error still occurs. I have tried this, and the answer is, no it doesn't.</p>
  
  <p>The other thing it mentioned was this quote from <a href="http://docs.python.org/py3k/library/multiprocessing.html#pipes-and-queues" rel="noreferrer">the multiprocessing docs</a> as a possible key to the problem. Referring to <code>JoinableQueue</code>'s, it says:</p>
  
  <blockquote>
    <p>... the semaphore used to count the number of unfinished tasks may
    eventually overflow raising an exception.</p>
  </blockquote></li>
  </ol>
  
  <hr>
  
  <pre><code>import multiprocessing
  
  class _ProcSTOP:
      pass
  
  class Proc(multiprocessing.Process):
  
      def __init__(self, pq, rq):
          self._pq = pq
          self._rq = rq
          super().__init__()
          print('++', self.name)
  
      def run(self):
          dat = self._pq.get()
  
          while not dat is _ProcSTOP:
  #            self._rq.put(dat)        # uncomment me for deadlock
              self._pq.task_done()
              dat = self._pq.get()
  
          self._pq.task_done() 
          print('==', self.name)
  
      def __del__(self):
          print('--', self.name)
  
  if __name__ == '__main__':
  
      pq = multiprocessing.JoinableQueue()
      rq = multiprocessing.JoinableQueue()
      pool = []
  
      for i in range(4):
          p = Proc(pq, rq) 
          p.start()
          pool.append(p)
  
      for i in range(10000):
          pq.put(i)
  
      pq.join()
  
      for i in range(4):
         pq.put(_ProcSTOP)
  
      pq.join()
  
      while len(pool) &gt; 0:
          print('??', pool)
          pool.pop().join()    # hangs here (if using rq)
  
      print('** complete')
  </code></pre>
  
  <hr>
  
  <p>Sample output, not using return-queue:</p>
  
  <pre><code>++ Proc-1
  ++ Proc-2
  ++ Proc-3
  ++ Proc-4
  == Proc-4
  == Proc-3
  == Proc-1
  ?? [&lt;Proc(Proc-1, started)&gt;, &lt;Proc(Proc-2, started)&gt;, &lt;Proc(Proc-3, started)&gt;, &lt;Proc(Proc-4, started)&gt;]
  == Proc-2
  ?? [&lt;Proc(Proc-1, stopped)&gt;, &lt;Proc(Proc-2, started)&gt;, &lt;Proc(Proc-3, stopped)&gt;]
  -- Proc-3
  ?? [&lt;Proc(Proc-1, stopped)&gt;, &lt;Proc(Proc-2, started)&gt;]
  -- Proc-2
  ?? [&lt;Proc(Proc-1, stopped)&gt;]
  -- Proc-1
  ** complete
  -- Proc-4
  </code></pre>
  
  <hr>
  
  <p>Sample output, using return queue:</p>
  
  <pre><code>++ Proc-1
  ++ Proc-2
  ++ Proc-3
  ++ Proc-4
  == Proc-2
  == Proc-4
  == Proc-1
  ?? [&lt;Proc(Proc-1, started)&gt;, &lt;Proc(Proc-2, started)&gt;, &lt;Proc(Proc-3, started)&gt;, &lt;Proc(Proc-4, started)&gt;]
  == Proc-3
  # here it hangs
  </code></pre>