title: Cassandra lightweight transaction based sequence generator consistency failure?
body: <p>Summary</p>
  <p>Our team inherited this sequence generator implemented using Cassandra;</p>
  <p>Table</p>
  <pre><code>CREATE TABLE IF NOT EXISTS sequences (
    id_name varchar,
    next_id bigint,
    instance_name varchar,
    PRIMARY KEY (id_name)
  )WITH COMPRESSION = { ... };
  
  </code></pre>
  <pre><code>GET_LOCK(&quot;UPDATE sequences USING TTL 10 set instance_name = ?  where id_name = ? IF instance_name = null&quot;, ConsistencyLevel.LOCAL_QUORUM), 
  
  SELECT_SEQUENCE(&quot;SELECT next_id from sequences where id_name = ?&quot;,
              ConsistencyLevel.LOCAL_QUORUM)
  
  UPDATE_SEQUENCE(&quot;UPDATE sequences SET next_id= ? where id_name= ? IF next_id= ?&quot;,ConsistencyLevel.LOCAL_QUORUM), 
  
  REMOVE_LOCK(&quot;UPDATE sequences set instance_name = null where id_name = ? IF instance_name = ?&quot;, ConsistencyLevel.LOCAL_QUORUM);
  
  (note: ConsistencyLevel was set to LOCAL_SERIAL in Java)
  </code></pre>
  <p>it was running fine until yesterday, we found two different java App nodes got the same sequence number</p>
  <p>Time stamps when this happened</p>
  <pre><code>AppNode 1
  
  getlock:           4:25:14.480 
  UpdateSequence:    4:25:14.486 
  
  AppNode 2
  
  getlock:           4:25:14,489
  UpdateSequence:    4:25:14,496
  </code></pre>
  <p>How can this happen? How can we find out what exactly happened?</p>