title: Race condition in ZooKeeper and Python based message queue
body: <p>I've been evaluating ZooKeeper as a simple message queue and I've written two very simple scripts: mq feeder and mq consumer. The feeder, below, is inly pushing 20 jobs to the queue and then is monitoring the queue status (jobs being consumed):</p>
  
  <pre><code>from kazoo.client import KazooClient
  
  zk = KazooClient(hosts='xxx')
  zk.start()
  
  for i in xrange(20):
    zk.create("/queue/%s" % i, b"%s" % i)
  
  while 1:
    print zk.get_children('/queue')
  </code></pre>
  
  <p>The consumer, below, is being launched few times (up to 3 concurrent processes in my tests) and it takes the job list, iterates over it to find unlocked job, processes it (sleeps for random number of seconds to simulate some work) and once done, deletes the job and then deletes the lock:</p>
  
  <pre><code>from kazoo.client import KazooClient
  from kazoo.exceptions import NodeExistsError
  from time import sleep
  import random
  
  zk = KazooClient(hosts='xxx')
  zk.start()
  zk.ensure_path("/locks")
  zk.ensure_path("/queue")
  
  while 1:
    jobs = sorted(zk.get_children('/queue'))
    if jobs:
      for i in jobs:
        print "Checking job: %s" % i
        try:
          zk.create("/locks/%s" % i)
        except NodeExistsError:
          print "Job is locked, skipping!"
          pass
        else:
          print "Job is unlocked, processing."
          sleep(random.randrange(5))
          zk.delete("/queue/%s" % i)
          print "Deleted processed job, deleting the lock."
          zk.delete("/locks/%s" % i)
          pass
    else:
      print "There's no locks in the queue."
      pass
  </code></pre>
  
  <p>The problem I am seeing, that I am unable to track is that the consumer processes are exiting with:</p>
  
  <pre><code>Traceback (most recent call last):
    File "zk_consumer.py", line 24, in &lt;module&gt;
      zk.delete("/queue/%s" % i)
    File "/Library/Python/2.7/site-packages/kazoo/client.py", line 1055, in delete
      return self.delete_async(path, version).get()
    File "/Library/Python/2.7/site-packages/kazoo/handlers/threading.py", line 107, in get
      raise self._exception
  kazoo.exceptions.NoNodeError: ((), {})
  </code></pre>
  
  <p>while the last process remains forever checking a single job, that remains in queue, but always locked. Obviously, I've some logical error here that I think leads to race condition, but I've spent some time on it, and I cant seem to spot it. Am I doing something wrong here, or is ZooKeeper not a viable solution for simple job queues?</p>