title: Python multiprocessing - watchdog process?
body: <p>I have a set of long-running process in a typical "pub/sub" setup with queues for communication.</p>
  
  <p>I would like to do two things, and I can't figure out how to accomplish both simultaneously:</p>
  
  <ol>
  <li>Addition/removal of workers. For example, I want to be able to add extra consumers if I see that my pending queue size has grown too large.</li>
  <li>Watchdog for my processes - I want to be notified if any of my producers or consumers crashes.</li>
  </ol>
  
  <p>I can do (2) in isolation:</p>
  
  <pre><code>try:
      while True:
          for process in workers + consumers:
              if not process.is_alive():
                  logger.critical("%-8s%s died!", process.pid, process.name)
          sleep(3)
  except KeyboardInterrupt:
      # Python propagates CTRL+C to all workers, no need to terminate them
      logger.warn('Received CTR+C, shutting down')
  </code></pre>
  
  <p>The above blocks, which prevents me from doing (1).</p>
  
  <p>So I decided to move the code into its own process. </p>
  
  <p>This doesn't work, because <code>process.is_alive()</code> only works for a parent checking the status of its children. In this case, the processes I want to check would be siblings instead of children.</p>
  
  <p>I'm a bit stumped on how to proceed. How can my main process support changes to subprocesses while also monitoring subprocesses?</p>