title: Timeout errors in cassandra
body: <p>Read Queries Are timing out in cassandra whenever secondary indexes are involved...
  Getting the following errors in cassandra logs</p>
  
  <pre><code>DEBUG [Thrift:275] 2013-11-15 02:30:56,565 ClientState.java (line 130) logged out: #&lt;User allow_all groups=[]&gt;
  DEBUG [Thrift:280] 2013-11-15 02:30:56,764 CassandraServer.java (line 588) batch_mutate
  DEBUG [Thrift:280] 2013-11-15 02:30:56,765 StorageProxy.java (line 175) Mutations/ConsistencyLevel are [RowMutation(keyspace='Checkout', key='37396539383637612d33616433
  2d343636662d383135612d303163386562343839616437', modifications=[ColumnFamily(User [content:false:294@1384504256763002,email_address:false:13@1384504256763001,user_id:fa
  lse:36@1384504256763000,])])]/QUORUM
  DEBUG [Thrift:280] 2013-11-15 02:30:56,765 StorageProxy.java (line 448) insert writing local RowMutation(keyspace='Checkout', key='37396539383637612d336164332d343636662
  d383135612d303163386562343839616437', modifications=[User])
  DEBUG [Thrift:280] 2013-11-15 02:30:56,765 StorageProxy.java (line 212) Write timeout java.util.concurrent.TimeoutException for one (or more) of:
  DEBUG [Thrift:280] 2013-11-15 02:30:56,765 CassandraServer.java (line 641) ... timed out
  DEBUG [Thrift:280] 2013-11-15 02:30:56,765 CustomTThreadPoolServer.java (line 196) Thrift transport error occurred during processing of message.
  org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an inter
  nal error on the server side. Please check your server logs.)
          at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
          at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
          at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
          at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
          at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
          at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)
  DEBUG [Thrift:280] 2013-11-15 02:30:56,765 ClientState.java (line 130) logged out: #&lt;User allow_all groups=[]&gt;
  DEBUG [Thrift:204] 2013-11-15 02:30:56,768 CassandraServer.java (line 588) batch_mutate
  DEBUG [Thrift:204] 2013-11-15 02:30:56,768 StorageProxy.java (line 175) Mutations/ConsistencyLevel are [RowMutation(keyspace='Checkout', key='37396539383637612d33616433
  2d343636662d383135612d303163386562343839616437', modifications=[ColumnFamily(User [content:false:294@1384504256763002,email_address:false:13@1384504256763001,user_id:fa
  lse:36@1384504256763000,])])]/QUORUM
  </code></pre>
  
  <p>Also getting this sometimes</p>
  
  <pre><code>ERROR 13:54:12,177 Exception in thread Thread[SSTableBatchOpen:7,5,main]
  java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.nio.ByteBuffer
          at org.apache.cassandra.db.marshal.UTF8Type.compare(UTF8Type.java:27)
          at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:45)
          at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:89)
          at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:38)
          at java.util.TreeMap.getEntry(TreeMap.java:328)
          at java.util.TreeMap.containsKey(TreeMap.java:209)
          at java.util.TreeSet.contains(TreeSet.java:217)
          at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:396)
          at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:187)
          at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:225)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)
   INFO 13:54:12,181 reading saved cache /opt/cassandra/saved_caches/OpsCenter-pdps-KeyCache
  </code></pre>