title: Filling a queue and managing multiprocessing in python
body: <p>I'm having this problem in python:</p>
  
  <ul>
  <li>I have a queue of URLs that I need to check from time to time</li>
  <li>if the queue is filled up, I need to process each item in the queue</li>
  <li>Each item in the queue must be processed by a single process (multiprocessing)</li>
  </ul>
  
  <p>So far I managed to achieve this "manually" like this:</p>
  
  <pre><code>while 1:
          self.updateQueue()
  
          while not self.mainUrlQueue.empty():
              domain = self.mainUrlQueue.get()
  
              # if we didn't launched any process yet, we need to do so
              if len(self.jobs) &lt; maxprocess:
                  self.startJob(domain)
                  #time.sleep(1)
              else:
                  # If we already have process started we need to clear the old process in our pool and start new ones
                  jobdone = 0
  
                  # We circle through each of the process, until we find one free ; only then leave the loop 
                  while jobdone == 0:
                      for p in self.jobs :
                          #print "entering loop"
                          # if the process finished
                          if not p.is_alive() and jobdone == 0:
                              #print str(p.pid) + " job dead, starting new one"
                              self.jobs.remove(p)
                              self.startJob(domain)
                              jobdone = 1
  </code></pre>
  
  <p>However that leads to tons of problems and errors. I wondered if I was not better suited using a Pool of process. What would be the right way to do this?</p>
  
  <p>However, a lot of times my queue is empty, and it can be filled by 300 items in a second, so I'm not too sure how to do things here.</p>