title: Cassandra connection issue during compaction
body: <p>As soon as cassandra starts a compaction on one of the nodes, we see our application is not able to connect to that node. what i noticed is that when compaction starts in one of the nodes, that node CPU load becomes 100% and as the compaction progresses and cpu load becomes low, our application is able to connect back to cassandra.</p>
  
  <p>We are running both our java application and cassandra on the same node using m3.xlarge Amazon EC2 instance. 
  Both our application and cassandra has the following jvm settings -Xms2G -Xmx2G -Xmn512M. Cassandra has hardly 300MB of data, but still we see so many compaction happening. 
  Here is how our one of cassandra column family settings looks like , we have around 20 column family with pretty much same settings. each column family has one secondary index.</p>
  
  <pre><code>bloom_filter_fp_chance=0.010000 AND
    caching='ROWS_ONLY' AND
    comment='' AND
    dclocal_read_repair_chance=0.000000 AND
    gc_grace_seconds=864000 AND
    index_interval=128 AND
    read_repair_chance=0.100000 AND
    replicate_on_write='true' AND
    populate_io_cache_on_flush='false' AND
    default_time_to_live=0 AND
    speculative_retry='99.0PERCENTILE' AND
    memtable_flush_period_in_ms=0 AND
    compaction={'min_threshold': '6', 'class': 'SizeTieredCompactionStrategy'} AND
    compression={'sstable_compression': 'LZ4Compressor'};
  </code></pre>
  
  <p>We are running cassandra community version 2.0.5 and data mapping library <a href="https://github.com/valchkou/cassandra-driver-mapping" rel="nofollow">https://github.com/valchkou/cassandra-driver-mapping</a>.</p>
  
  <p>I looked around ,didn't see anyone facing this sort of issue. Any help would be greatly appreciated.</p>
  
  <p><strong>Error Stacktrace</strong></p>
  
  <pre><code>2015-07-29 18:22:50.341 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 234 : [] [] [] Defuncting connection to /172.31.22.125:9042
  com.datastax.driver.core.TransportException: [/172.31.22.125:9042] Channel has been closed
      at com.datastax.driver.core.Connection$Dispatcher.channelClosed(Connection.java:636) [cassandra-driver-core-2.1.0.jar!/:na]
      at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:375) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.0.Final.jar!/:na]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
  2015-07-29 18:22:50.341 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Cluster 1370 : [] [] [] Host /172.31.22.125:9042 is Suspected
  2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-171] closing connection
  2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-172] closing connection
  2015-07-29 18:22:50.342 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Connection 103 : [] [] [] [/172.31.22.125:9042-173] Error connecting to /172.31.22.125:9042 (Connection refused: /172.31.22.125:9042)
  2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #6] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-173] closing connection
  2015-07-29 18:22:50.343 DEBUG 29056 --- [New I/O boss #9] com.datastax.driver.core.Connection 608 : [] [] [] [/172.31.22.125:9042-173] connection error
  java.net.ConnectException: Connection refused: /172.31.22.125:9042
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_75]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.0.Final.jar!/:na]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
  2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Connection 234 : [] [] [] Defuncting connection to /172.31.22.125:9042
  com.datastax.driver.core.TransportException: [/172.31.22.125:9042] Cannot connect
      at com.datastax.driver.core.Connection.&lt;init&gt;(Connection.java:104) [cassandra-driver-core-2.1.0.jar!/:na]
      at com.datastax.driver.core.Connection$Factory.open(Connection.java:445) [cassandra-driver-core-2.1.0.jar!/:na]
      at com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1405) [cassandra-driver-core-2.1.0.jar!/:na]
      at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) [cassandra-driver-core-2.1.0.jar!/:na]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
      at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
  Caused by: java.net.ConnectException: Connection refused: /172.31.22.125:9042
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_75]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar!/:na]
      at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.0.Final.jar!/:na]
      ... 3 common frames omitted
  2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Cluster 1428 : [] [] [] Host /172.31.22.125:9042 is DOWN
  2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] c.d.driver.core.ControlConnection 592 : [] [] [] [Control connection] /172.31.22.125:9042 is down, currently connected to /172.31.27.168:9042
  2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-37] com.datastax.driver.core.Cluster 1428 : [] [] [] Host /172.31.22.125:9042 is DOWN
  2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Cluster 1473 : [] [] [] /172.31.22.125:9042 is down, scheduling connection retries
  2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] c.d.driver.core.ControlConnection 592 : [] [] [] [Control connection] /172.31.22.125:9042 is down, currently connected to /172.31.27.168:9042
  2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-36] c.d.d.core.AbstractReconnectionHandler 60 : [] [] [] First reconnection scheduled in 1000ms
  2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] com.datastax.driver.core.Cluster 1473 : [] [] [] /172.31.22.125:9042 is down, scheduling connection retries
  2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] c.d.d.core.AbstractReconnectionHandler 60 : [] [] [] First reconnection scheduled in 1000ms
  2015-07-29 18:22:51.345 DEBUG 29056 --- [Reconnection-0] com.datastax.driver.core.Connection 103 : [] [] [] [/172.31.22.125:9042-174] Error connecting to /172.31.22.125:9042 (Connection refused: /172.31.22.125:9042)
  2015-07-29 18:22:51.345 DEBUG 29056 --- [New I/O worker #7] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-174] closing connection
  2015-07-29 18:22:51.346 DEBUG 29056 --- [New I/O boss #9] com.datastax.driver.core.Connection 608 : [] [] [] [/172.31.22.125:9042-174] connection error
  java.net.ConnectException: Connection refused: /172.31.22.125:9042
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
  </code></pre>