title: Queuing systems - what is a good way to start up multiple workers?
body: <ul>
  <li>How have you set-up one or more worker scripts for queue-oriented systems?</li>
  <li>How do you arrange to startup - and restart if necessary - worker scripts as required? (I'm thinking about such tools as init.d/, Ruby-based 'god', DJB's Daemontools, etc, etc)</li>
  </ul>
  
  <p>I'm developing an asynchronous queue/worker system, in this case using PHP &amp; <a href="http://xph.us/software/beanstalkd/" rel="noreferrer">BeanstalkdD</a> (though the actual language and daemon isn't important). The tasks themselves are not too hard - encoding an array with the commands and parameters into JSON for transport through the Beanstalkd daemon, picking them up in a worker script to action them as required.</p>
  
  <p>There are a number of other similar queue/worker setups out there, such as <a href="http://rubyforge.org/projects/starling/" rel="noreferrer">Starling</a>, <a href="http://www.danga.com/gearman/" rel="noreferrer">Gearman</a>, <a href="http://aws.amazon.com/sqs/" rel="noreferrer">Amazon's SQS</a> and other more 'enterprise' oriented systems like IBM's MQ and RabbitMQ. If you run something like Gearman, or SQS - how do <em>you</em> start and control the worker pool?  The questions is on the initial worker startup, and then being able to add additional extra workers, shutting them down at will (though I can send a message through the queue to shut them down - as long as some 'watcher' won't automatically restart them). This is not a PHP problem, it's about straight Unix processes of setting up one or more processes to run on startup, or adding more workers to the pool.</p>
  
  <p>A <a href="http://pastebin.ca/1331063" rel="noreferrer">bash script to loop a script </a> is already in place - this calls the PHP script which then collects and runs tasks from the queue, occasionally exiting to be able to clean itself up (it can also pause a few seconds on failure, or via a planned event).  This works fine, and building the worker processes on top of that won't be very hard at all.</p>
  
  <p>Getting a good worker controller system is about flexibility, starting one or two automatically on a machine start, and being able to add a couple more from the command line when the queue is busy, shutting down the extras when no longer required.</p>