title: Cassandra Tombstoning warning and failure thresholds breached
body: <p>We are running a Titan Graph DB server backed by Cassandra as a persistent store and are running into an issue with reaching the limit on Cassandra tombstone thresholds that is causing our queries to fail / timeout periodically as data accumulates. It seems like the compaction is unable to keep up with the number of tombstones being added.</p>
  
  <p>Our use case supports:</p>
  
  <ol>
  <li>High read / write throughputs.</li>
  <li>High sensitivity to reads.</li>
  <li>Frequent updates to node values in Titan. causing rows to be updated in Cassandra.</li>
  </ol>
  
  <p>Given the above use cases, we are already optimizing Cassandra to aggressively do the following:</p>
  
  <ol>
  <li>Aggressive compaction by using the levelled compaction strategies</li>
  <li>Using tombstone_compaction_interval as 60 seconds.</li>
  <li>Using tombstone_threshold to be 0.01</li>
  <li>Setting gc_grace_seconds to be 1800</li>
  </ol>
  
  <p>Despite the following optimizations, we are still seeing warnings in the Cassandra logs similar to:
  <strong>[WARN] (ReadStage:7510) org.apache.cassandra.db.filter.SliceQueryFilter: Read 0 live and 10350 tombstoned cells in .graphindex (see tombstone_warn_threshold). 8001 columns was requested, slices=[00-ff], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}</strong></p>
  
  <p>Occasionally as time progresses, we also see the failure threshold breached and causes errors.</p>
  
  <p>Our cassandra.yaml file has the tombstone_warn_threshold to be 10000, and the tombstone_failure_threshold to be much higher than recommended at 250000, with no real noticeable benefits. </p>
  
  <p>Any help that can point us to the correct configurations would be greatly appreciated if there is room for further optimizations. Thanks in advance for your time and help.</p>