title: Cassandra delete operation is sometimes not working able to select the data after delete
body: <p>I have two tables </p>
  
  <pre><code>CREATE TABLE IF NOT EXISTS QueueBucket (
      queueName   text,
      bucketId    int,
      scheduledMinute timestamp,
      scheduledTime timestamp,
      messageId   uuid,
      PRIMARY KEY ((queueName, bucketId, scheduledMinute), scheduledTime, messageId)
  )  WITH compaction = { 'class' :  'LeveledCompactionStrategy'  } AND speculative_retry='NONE' ;
  
  CREATE TABLE IF NOT EXISTS InDelivery (
      queueName       text,
      nodeId        uuid,
      dequeuedMinute    timestamp,
      messageId       uuid,
      bucketid        int,
      dequeuedTime    timestamp,
      PRIMARY KEY ((queueName, nodeId,bucketId, dequeuedMinute),dequeuedTime, messageId)
  );
  </code></pre>
  
  <p>In the code I perform insert into QueueBucket and delete from indelivery in batch (logged). But during load testing some how the delete from indelivery sometimes doesn't work although insert into QueueBucket works. To confirm this is applied a read from indelivery check immediately after  which read the deleted messageId if the messageId still exist prints WARN log .</p>
  
  <pre><code>    queueDao.insertMsgInfo(queueName, bucketId, QueueUtils.getMinute(scheduledTime), scheduledTime, messageId);
      queuDao.deleteInDelivery(queueName, nodeId, bucketId, bucketMinute, dequeuedTime, messageId);
      if(queueServiceMetaDao.hasIndeliveryMessage(inDeliveryPK)) {
          log.warn("messageId  {} of queue {} bucket {} with node {} dequuedTime {} dequeud minute {} could not get deleted from indelivery.",
                  messageId,queueName,bucketId, nodeId,QueueUtils.dateToString(dequeuedTime),QueueUtils.dateToString(bucketMinute));
          }
  </code></pre>
  
  <p>in insertMsgInfo and deleteInDelivery methods i am reusing prepared statement .             </p>
  
  <pre><code>"INSERT INTO queuebucket (queuename, bucketid , scheduledminute, scheduledtime, messageid ) VALUES ( ? , ? , ? , ? , ? );"
  "DELETE FROM indelivery WHERE queuename = ? AND nodeId = ? AND bucketId=? AND dequeuedMinute=? AND dequeuedTime =? AND messageId=? ;"
  </code></pre>
  
  <p>in hasIndeliveryMessage I am passing the same values wrapping into inDeliveryPrimaryKey as I passed for deleting indelivery data in  moveBackToQueueBucket method.</p>
  
  <pre><code>"SELECT messageId FROM indelivery WHERE queuename = ? AND nodeId = ? AND bucketId=? AND dequeuedMinute=? AND dequeuedTime=? AND messageId=? ;"
  </code></pre>
  
  <p>I am no clue why i see multple warn message "could not get deleted from indelivery." . Please help</p>
  
  <blockquote>
    <p>I am using cassandra version 2.2.7 it is 6 node cassandra cluster with
    replication factor 5 and read and write consistency used is QUORUM.</p>
  </blockquote>
  
  <p>I also gone through link <a href="https://stackoverflow.com/questions/26040374/cassandra-deleted-data-still-there">Cassandra - deleted data still there</a>  and <a href="https://issues.apache.org/jira/browse/CASSANDRA-7810" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/CASSANDRA-7810</a> 
  but this issue is fixed long time ago.in 2.0.11.</p>
  
  <p><strong>Futher update</strong> As per <a href="https://stackoverflow.com/questions/35392430/cassandra-delete-not-working">Cassandra - Delete not working</a> i also ran nodetool repair  but the problem still persist.
  Should i run compact as well ?</p>
  
  <p><strong>Futher update:</strong>
  I am no more using batch i do simple insert into queuebucket and  delete for indelivery and then read the data but still the problem persist</p>
  
  <p><strong>Adding some logs:</strong></p>
  
  <pre><code>2016-07-19 20:39:42,440[http-nio-8014-exec-12]INFO  QueueDaoImpl -deleting from indelivery queueName pac01_deferred nodeid 1349d57f-28f5-37d4-9fe1-dfa14dba4a9f bucketId 382 dequeuedMinute 20160719203900000 dequeuedTime 20160719203942310 messageId cc4fb158-f61e-345b-8dcf-3f842fe52d50:
  2016-07-19 20:39:42,442[http-nio-8014-exec-12]INFO  QueueDaoImpl -Reading from indelivery : queue pac01_deferred nodeId 1349d57f-28f5-37d4-9fe1-dfa14dba4a9f dequeueMinute 20160719203900000 dequeueTime 20160719203942310 messageid cc4fb158-f61e-345b-8dcf-3f842fe52d50 bucketId 382 indeliveryRow Row[cc4fb158-f61e-345b-8dcf-3f842fe52d50]
  2016-07-19 20:39:42,442[http-nio-8014-exec-12]WARN  QueueImpl -messageId  cc4fb158-f61e-345b-8dcf-3f842fe52d50 of queue pac01_deferred bucket 382 with node 1349d57f-28f5-37d4-9fe1-dfa14dba4a9f dequuedTime 20160719203942310 dequeud minute 20160719203900000 could not get deleted from indelivery .
  </code></pre>
  
  <p>Should i try will cosnsistency ALL ???</p>