title: DCE Cassandra 3.9 slow secondary index creation during joining existing cluster
body: <p>We have cassandra cluster with 32 nodes, average node size is about 1TB. Node configuration 1xIntel Xeon E3-1271v3, 32GB ram, 2x3TB HDD.
  We have one DB with some small tables and one big table, that holds is about 90-95% of total cluster size.</p>
  
  <p>I try to add additional nodes to this cluster, but suddenly find out, that adding one node to existing cluster take is about 13-14 days for joining to cluster. Build secondary indexes take most of this time and all this time i see that all compactor threads take all available CPU.</p>
  
  <p>I have changed cassandra config to extends limits:</p>
  
  <ul>
  <li>concurrent_compactors: 4</li>
  <li>compaction_throughput_mb_per_sec: 0</li>
  </ul>
  
  <p><a href="https://pastebin.com/raw/6gCingWK" rel="nofollow noreferrer">Cassandra full config</a></p>
  
  <p><a href="https://pastebin.com/raw/6Lc8hfmX" rel="nofollow noreferrer">Schema</a></p>
  
  <p>Is about 1 year ago we also add new nodes to this cluster and extend it from 16 nodes to 32 nodes cluster, average node size was 1TB before cluster extends. Cassandra version was 2.1. One node joining time was 1-1.5days.</p>
  
  <p>So the question how can we speed up this process ? Did we miss something ?</p>
  
  <p>Thanks.</p>