title: Cassandra running compact indefinitely - High CPU usage
body: <p><strong>Context</strong></p>
  
  <p>We have 6 instances of Cassandra hosted on AWS, separated into 3 different regions, 2 per regions (2 in eu-west, 2 in us-west, 2 in ap-southeast).</p>
  
  <p>2 days ago, we moved 2 of our EC2 Cassandra instances from us-west-1 to us-east-1. When I say "move" I mean that we decommissioned them and added 2 new instances on our cluster.</p>
  
  <p>We ran <code>nodetool repair</code> which didn't do anything, and <code>nodetool rebuild</code> which synchronized our data from the eu-west data centre. Following that change we noticed that multiple instances on our Cassandra cluster were using over 70% CPU and had incoming traffic.</p>
  
  <p>At first, we thought it was the replication taking place, but considering that we only had 500MB of data, and that it is still running we are puzzled as to what is happening.</p>
  
  <hr>
  
  <p><strong>Instances hardware:</strong></p>
  
  <p>All of our instances are running on m3.medium which means that we are on:</p>
  
  <ul>
  <li>1 CPU, 2.5 GHz</li>
  <li>3.75 GB of RAM</li>
  <li>4GB SSD</li>
  </ul>
  
  <p>Also we have mounted an EBS volume for <code>/var/lib/cassandra</code> which is actually a RAID0 of 6 SSDs on EBS:</p>
  
  <ul>
  <li>EBS volume 300GB SSD, RAID0</li>
  </ul>
  
  <p>Ref: <a href="http://aws.amazon.com/ec2/instance-types/" rel="nofollow noreferrer">Amazon Instances Types</a></p>
  
  <hr>
  
  <p><strong>Software Version:</strong></p>
  
  <p>Cassandra Version: 2.0.12</p>
  
  <hr>
  
  <p><strong>Thoughts:</strong></p>
  
  <p>After analysing our data we thought this was caused by Cassandra data compaction.</p>
  
  <p>There is another stackoverflow question around the same subject: <a href="https://stackoverflow.com/questions/28236638/cassandra-compaction-tasks-stuck">Cassandra compaction tasks stuck</a>.</p>
  
  <p>However, this was solved by going for a single SSD (Azure Premium Storage - still in preview) and no RAID0 configured for Cassandra, and as the author said, there is no reason for this to fix the underlying problem (why would removing the RAID0 part from the equation fix this?).</p>
  
  <p>We are not keen yet to move to a local storage as AWS pricing is a lot higher than what we have now. Even though, if it really is the cause of our problem, we will try it.</p>
  
  <p>Another reason why this sounds like a deeper problem is that we have data showing that these EBS volumes have been writing/reading a lot of data in the last 3 days.</p>
  
  <p>Since we moved instances, we get something around 300-400KB of written data per second on each EBS volume, so since we have a RAID0, 6 times this amount per second = 1.8-2.4MB/s. This amounts to ~450GB of data written PER instance over the last 3 days. And we have basically the same value for READ operation too.</p>
  
  <p>We are only running tests on them at the moment, so the only traffic we are getting is coming from our CI server and eventually from the communication that Gossip is doing between the instances.</p>
  
  <hr>
  
  <p><strong>Debug notes</strong></p>
  
  <p>Output of <code>nodetool status</code>:</p>
  
  <pre><code>Datacenter: cassandra-eu-west-1-A
  =================================
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address         Load       Tokens  Owns   Host ID                               Rack
  UN  xxx.xxx.xxx.xxx 539.5 MB   256     17.3%  12341234-1234-1234-1234-12341234123412340cd7  eu-west-1c
  UN  xxx.xxx.xxx.xxx 539.8 MB   256     14.4%  30ff8d00-1ab6-4538-9c67-a49e9ad34672  eu-west-1b
  Datacenter: cassandra-ap-southeast-1-A
  ======================================
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address         Load       Tokens  Owns   Host ID                               Rack
  UN  xxx.xxx.xxx.xxx 585.13 MB  256     16.9%  a0c45f3f-8479-4046-b3c0-b2dd19f07b87  ap-southeast-1a
  UN  xxx.xxx.xxx.xxx 588.66 MB  256     17.8%  b91c5863-e1e1-4cb6-b9c1-0f24a33b4baf  ap-southeast-1b
  Datacenter: cassandra-us-east-1-A
  =================================
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address         Load       Tokens  Owns   Host ID                               Rack
  UN  xxx.xxx.xxx.xxx 545.56 MB  256     15.2%  ab049390-f5a1-49a9-bb58-b8402b0d99af  us-east-1d
  UN  xxx.xxx.xxx.xxx 545.53 MB  256     18.3%  39c698ea-2793-4aa0-a28d-c286969febc4  us-east-1e
  </code></pre>
  
  <p>Output of <code>nodetool compactionstats</code>:</p>
  
  <pre><code>pending tasks: 64
            compaction type        keyspace           table       completed           total      unit  progress
                 Compaction         staging    stats_hourly       418858165      1295820033     bytes    32.32%
  Active compaction remaining time :   0h00m52s
  </code></pre>
  
  <p>Running <code>dstat</code> on unhealthy instance:</p>
  
  <p><img src="https://i.stack.imgur.com/gtfcF.png" alt="dstat on unhealthy instance"></p>
  
  <p>Compaction history in graph form (average of 300 times per hour starting the 16th):</p>
  
  <p><img src="https://i.stack.imgur.com/YdEOV.png" alt="Compaction history graph"></p>
  
  <p>EBS Volumes usage:</p>
  
  <p><img src="https://i.stack.imgur.com/6wuH6.png" alt="EBS Volume 1"></p>
  
  <p><img src="https://i.stack.imgur.com/xTyJ1.png" alt="EBS Volume 2"></p>
  
  <p>Running df -h:</p>
  
  <pre><code>Filesystem      Size  Used Avail Use% Mounted on
  /dev/xvda1       33G   11G   21G  34% /
  none            4.0K     0  4.0K   0% /sys/fs/cgroup
  udev            1.9G   12K  1.9G   1% /dev
  tmpfs           377M  424K  377M   1% /run
  none            5.0M     0  5.0M   0% /run/lock
  none            1.9G  4.0K  1.9G   1% /run/shm
  none            100M     0  100M   0% /run/user
  /dev/xvdb       3.9G  8.1M  3.7G   1% /mnt
  /dev/md0        300G  2.5G  298G   1% /var/lib/cassandra
  </code></pre>
  
  <p>Running <code>nodetool tpstats</code>:</p>
  
  <pre><code>Pool Name                    Active   Pending      Completed   Blocked  All time blocked
  MutationStage                     0         0        3191689         0                 0
  ReadStage                         0         0         574633         0                 0
  RequestResponseStage              0         0        2698972         0                 0
  ReadRepairStage                   0         0           2721         0                 0
  ReplicateOnWriteStage             0         0              0         0                 0
  MiscStage                         0         0          62601         0                 0
  HintedHandoff                     0         1            443         0                 0
  FlushWriter                       0         0          88811         0                 0
  MemoryMeter                       0         0           1472         0                 0
  GossipStage                       0         0         979483         0                 0
  CacheCleanupExecutor              0         0              0         0                 0
  InternalResponseStage             0         0             25         0                 0
  CompactionExecutor                1        39          99881         0                 0
  ValidationExecutor                0         0          62599         0                 0
  MigrationStage                    0         0             40         0                 0
  commitlog_archiver                0         0              0         0                 0
  AntiEntropyStage                  0         0         149095         0                 0
  PendingRangeCalculator            0         0             23         0                 0
  MemtablePostFlusher               0         0         173847         0                 0
  
  Message type           Dropped
  READ                         0
  RANGE_SLICE                  0
  _TRACE                       0
  MUTATION                     0
  COUNTER_MUTATION             0
  BINARY                       0
  REQUEST_RESPONSE             0
  PAGED_RANGE                  0
  READ_REPAIR                  0
  </code></pre>
  
  <p>Running iptraf, sorted by bytes:</p>
  
  <p><img src="https://i.stack.imgur.com/UsDN2.png" alt="iptraf sorted by bytes"></p>