title: Lost updates in Cassandra
body: <p>I am facing problems of lost writes while updating a row in Cassandra. Here's my schema:</p>
  
  <pre><code>create table balances(
  id bigint,
  balance decimal,
  last_transaction_id bigint,
  update_timestamp timestamp,
  type varchar,
  is_balance_valid boolean, 
  primary key (wallet_id)
  ) 
  </code></pre>
  
  <p>Total nodes in cluster: 3 in Local DC
  Replication factor: 2
  Cassandra Version: 2.1.8</p>
  
  <p>I update the value of column "<strong>balance</strong>" every time user does a transaction by reading the previously set value, adding the transaction amount and issuing the update. I am using Java, Datastax driver (2.1.5).</p>
  
  <p>Once out of about half a million transactions, one particular update would fail. This happens usually when the user has done two transactions in quick succession, down to few milliseconds. Here are logs:</p>
  
  <blockquote>
    <p>Transaction #1</p>
    
    <p>10 Feb 2016 18:15:16,984 -[pool-11-thread-1]-  INFO -
    ScratchpadMasterStreamProcessor.processMessage(62) - Printing str id:
    1466140282Scratchpad id: 9127013322</p>
    
    <p>10 Feb 2016 18:15:16,986 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(43) - <strong>Current balance: 0.0</strong></p>
    
    <p>10 Feb 2016 18:15:16,986 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(44) - <strong>Deviation : 200.0</strong></p>
    
    <p>10 Feb 2016 18:15:16,986 -[pool-11-thread-1]- DEBUG -
    UserBalanceManager.updateWalletBalance(70) - Updating user..510978682</p>
    
    <p>10 Feb 2016 18:15:16,987 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(51) - <strong>Final Balance: 200.0</strong></p>
    
    <p>10 Feb 2016 18:15:16,987 -[pool-11-thread-1]- DEBUG -
    ScratchpadMasterStreamProcessor.processMessage(79) - Balance Update
    was successful for wallet 510978682</p>
    
    <p>Transaction #2 </p>
    
    <p>10 Feb 2016 18:18:19,157 -[pool-11-thread-1]-  INFO -
    ConsumerThread.run(82) - Event Recieved</p>
    
    <p>10 Feb 2016 18:18:19,159 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(43) - <strong>Current balance: 200.0</strong></p>
    
    <p>10 Feb 2016 18:18:19,159 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(44) - <strong>Deviation : 50.0</strong></p>
    
    <p>10 Feb 2016 18:18:19,159 -[pool-11-thread-1]- DEBUG -
    UserBalanceManager.updateWalletBalance(70) - Updating user..510978682</p>
    
    <p>10 Feb 2016 18:18:19,160 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(51) - <strong>Final Balance: 250.0</strong></p>
    
    <p>10 Feb 2016 18:18:19,160 -[pool-11-thread-1]- DEBUG -
    ScratchpadMasterStreamProcessor.processMessage(79) - Balance Update
    was successful for wallet 510978682</p>
    
    <p>Transaction #3 (This is lost)</p>
    
    <p>10 Feb 2016 18:18:19,160 -[pool-11-thread-1]-  INFO -
    ScratchpadMasterStreamProcessor.processMessage(62) - Printing str id:
    1466162182Scratchpad id: 9127117934</p>
    
    <p>10 Feb 2016 18:18:19,161 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(43) - <strong>Current balance: 250.0</strong></p>
    
    <p>10 Feb 2016 18:18:19,161 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(44) - <strong>Deviation : -250.0</strong></p>
    
    <p>10 Feb 2016 18:18:19,161 -[pool-11-thread-1]- DEBUG -
    UserBalanceManager.updateWalletBalance(70) - Updating user..510978682</p>
    
    <p>10 Feb 2016 18:18:19,162 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(51) - <strong>Final Balance: 0.0</strong></p>
    
    <p>10 Feb 2016 18:18:19,162 -[pool-11-thread-1]- DEBUG -
    ScratchpadMasterStreamProcessor.processMessage(79) - Balance Update
    was successful for wallet 510978682</p>
    
    <p>Transaction #4 Read stale balance, oops</p>
    
    <p>10 Feb 2016 18:18:23,140 -[pool-11-thread-1]-  INFO -
    ConsumerThread.run(82) - Event Recieved</p>
    
    <p>10 Feb 2016 18:18:23,140 -[pool-11-thread-1]-  INFO -
    ScratchpadMasterStreamProcessor.processMessage(62) - Printing str id:
    1466162730Scratchpad id: 9127120830</p>
    
    <p>10 Feb 2016 18:18:23,141 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(43) - <strong>Current balance: 250.0</strong></p>
    
    <p>10 Feb 2016 18:18:23,141 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(44) - <strong>Deviation : 200.0</strong></p>
    
    <p>10 Feb 2016 18:18:23,141 -[pool-11-thread-1]- DEBUG -
    UserBalanceManager.updateWalletBalance(70) - Updating user..510978682</p>
    
    <p>10 Feb 2016 18:18:23,142 -[pool-11-thread-1]- DEBUG -
    SclwBalanceUpdater.updateBalance(51) - <strong>Final Balance: 450.0</strong></p>
    
    <p>10 Feb 2016 18:18:23,142 -[pool-11-thread-1]- DEBUG -
    ScratchpadMasterStreamProcessor.processMessage(79) - Balance Update
    was successful for wallet 510978682</p>
  </blockquote>
  
  <p>I have set consistency level to LOCAL_QUORUM for both reads and writes, and the three cassandra node servers have same time (using NTP). What could be the problem?</p>