title: Concerns about zookeeper's lock-recipe
body: <p>While reading the ZooKeeper's <a href="http://zookeeper.apache.org/doc/trunk/recipes.html#sc_recipes_Locks" rel="noreferrer">recipe for lock</a>, I got confused. It seems that this recipe for distributed locks can not guarantee <strong>"any snapshot in time no two clients think they hold the same lock"</strong>. But since ZooKeeper is so widely adopted, if there were such mistakes in the reference documentation, someone should have pointed it out long ago, so what did I misunderstand?</p>
  
  <p>Quoting <a href="http://zookeeper.apache.org/doc/trunk/recipes.html#sc_recipes_Locks" rel="noreferrer">the recipe for distributed locks</a>:</p>
  
  <blockquote>
    <p>Locks</p>
    
    <p>Fully distributed locks that are globally synchronous, <strong>meaning at any snapshot in time no two clients think they hold the same lock</strong>. These can be implemented using ZooKeeeper. As with priority queues, first define a lock node.</p>
    
    <ol>
    <li>Call create( ) with a pathname of "<em>locknode</em>/guid-lock-" and the sequence and ephemeral flags set. </li>
    <li>Call getChildren( ) on the lock node without setting the watch flag (this is important to avoid the herd effect).</li>
    <li>If the pathname created in step 1 has the lowest sequence number suffix, the client has the lock and the client exits the protocol.</li>
    <li>The client calls exists( ) with the watch flag set on the path in the lock directory with the next lowest sequence number.</li>
    <li>if exists( ) returns false, go to step 2. Otherwise, wait for a notification for the pathname from the previous step before going to step 2.</li>
    </ol>
  </blockquote>
  
  <p>Consider the following case:</p>
  
  <ul>
  <li>Client1 successfully acquired the lock (in step 3), with ZooKeeper node "locknode/guid-lock-0";</li>
  <li>Client2 created node "locknode/guid-lock-1", failed to acquire the lock, and is now watching "locknode/guid-lock-0";</li>
  <li>Later, for some reason (say, network congestion), Client1 fails to send a heartbeat message to the ZooKeeper cluster on time, but Client1 is still working away, mistakenly assuming that it still holds the lock.</li>
  <li><p>But, ZooKeeper may think Client1's session is timed out, and then</p>
  
  <ol>
  <li>delete "locknode/guid-lock-0",</li>
  <li>send a notification to Client2 (or maybe send the notification first?),    </li>
  <li>but can not send a "session timeout" notification to Client1 in time (say, due to network congestion).</li>
  </ol></li>
  <li>Client2 gets the notification, goes to step 2, gets the only node ""locknode/guid-lock-1", which it created itself; thus, Client2 assumes it hold the lock.</li>
  <li>But at the same time,  Client1 assumes it holds the lock.</li>
  </ul>
  
  <p>Is this a valid scenario?</p>