title: Cassandra - enable row cache then got a lot of "GC Pauses greater than 200 ms"
body: <p>I have a table with about 1 million record, most reads (over 95%), table schema :</p>
  
  <pre><code>CREATE TABLE ams_table (
      projectid text,
      tagk text,
      tagv text,
      metricid bigint,
      PRIMARY KEY (projectid, tagk, tagv, metricid)
  ) WITH CLUSTERING ORDER BY (tagk ASC, tagv ASC, metricid ASC)
      AND bloom_filter_fp_chance = 0.01
      AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
      AND comment = ''
      AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
      AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
      AND crc_check_chance = 1.0
      AND dclocal_read_repair_chance = 0.1
      AND default_time_to_live = 0
      AND gc_grace_seconds = 864000
      AND max_index_interval = 2048
      AND memtable_flush_period_in_ms = 0
      AND min_index_interval = 128
      AND read_repair_chance = 0.0
      AND speculative_retry = '99PERCENTILE';
  </code></pre>
  
  <p>read the table is very frequent, the number of requests in half, duration time for 70%, causing a high CPU usage. so I thought that it's perfect for use row cache.</p>
  
  <p>one record have 4 field, less than 100 byte, so I thought that the entire table does not occupy more than 100MB memory.</p>
  
  <p>so, alter table, rows_per_partition to 'ALL', cache all record. </p>
  
  <pre><code>ALTER TABLE ams.tbl_tags_with_metricid WITH caching = {'keys': 'ALL', 'rows_per_partition': 'ALL' } ;
  </code></pre>
  
  <p>modify cassandra.yaml, set row_cache_size_in_mb to 128M.</p>
  
  <pre><code>row_cache_size_in_mb : 128
  </code></pre>
  
  <p>nodetool info, hit rate almost 0.945, until now everything's just fine.
  but soon, I saw that the CPU usage is higher.
  check system.log, get a lot of GC Pauses greater than 200 ms. like this.</p>
  
  <pre><code>WARN  [Native-Transport-Requests-10] 2017-11-22 08:55:39,793 SelectStatement.java:377 - Aggregation query used without partition key
  INFO  [Service Thread] 2017-11-22 08:55:40,553 GCInspector.java:284 - ParNew GC in 204ms.  CMS Old Gen: 11955784424 -&gt; 12032443672; Par Eden Space: 671088640 -&gt; 0; Par Survivor Space: 77459760 -&gt; 83886080
  INFO  [Service Thread] 2017-11-22 08:55:43,551 GCInspector.java:284 - ParNew GC in 213ms.  CMS Old Gen: 12264670920 -&gt; 12341517200; Par Eden Space: 671088640 -&gt; 0; 
  INFO  [Service Thread] 2017-11-22 08:55:45,673 GCInspector.java:284 - ParNew GC in 221ms.  CMS Old Gen: 12354581144 -&gt; 12432569616; Par Eden Space: 671088640 -&gt; 0; 
  INFO  [Service Thread] 2017-11-22 08:55:46,186 GCInspector.java:284 - ParNew GC in 217ms.  CMS Old Gen: 12248728592 -&gt; 12343417080; Par Eden Space: 671088640 -&gt; 0; 
  INFO  [Service Thread] 2017-11-22 08:55:47,799 GCInspector.java:284 - ParNew GC in 354ms.  CMS Old Gen: 11967866640 -&gt; 12058730544; Par Eden Space: 671088640 -&gt; 0; 
  INFO  [Service Thread] 2017-11-22 08:55:48,242 GCInspector.java:284 - ParNew GC in 204ms.  CMS Old Gen: 11940028704 -&gt; 11987653704; Par Eden Space: 671088640 -&gt; 0; 
  </code></pre>
  
  <p>cassandra 3.9, deployed in kubernetes container(cpu 8 core, memory 32G), free memory about 10G.</p>
  
  <p>And I try to alter table, 'rows_per_partition' to '3000', so "GC Pauses greater than 200 ms" was disappear, but hit rate is lower and lower, and CPU usage become the same as before.</p>