title: Limiting the number of processes running at a time from a Python script
body: <p>I'm running a backup script that launches child processes to perform backups by rsync.  However I have no way to limit the number of rsyncs it launches at a time.</p>
  
  <p>Here's the code I'm working on at the moment:</p>
  
  <pre><code>print "active_children: ", multiprocessing.active_children()
  print "active_children len: ", len(multiprocessing.active_children())
  while len(multiprocessing.active_children()) &gt; 49:
     sleep(2)
  p = multiprocessing.Process(target=do_backup, args=(shash["NAME"],ip,shash["buTYPE"], ))
  jobs.append(p)
  p.start()
  </code></pre>
  
  <p>This is showing a maximum of one child when I have hundreds of rsyncs running.  Here's the code that actually launches the rsync (from inside the do_backup function), with <code>command</code> being a variable containing the rsync line:</p>
  
  <pre><code>print command
  subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
  return 1
  </code></pre>
  
  <p>If I add a sleep(x) to the do_backup function it will show up as an active child while it's sleeping.  Also the process table is showing the rsync processes as having a PPID of 1.  I'm assuming from this that the rsync splits off and is no longer a child of python which allows my child process to die so I can't count it anymore.  Does anyone know how to keep the python child alive and being counted until the rsync is complete?</p>