title: ElasticSearch: how to read output of _node/hot_threads
body: <p>I have a data node in ElasticSearch that is running high on CPU (99%) and searches are slow. Using <code>top</code> reveals it's elasticsearch's process that is using all of the CPU.</p>
  
  <p>I ran the <code>_nodes/hot_threads</code> API in that node and I got this output, but I don't know how to interpret it. Could somebody explain?</p>
  
  <pre><code>::: {warm-xxx}{XXXXXXX}{YYYYYYYY}{10.10.10.10}{10.10.10.10:9300}{aws_availability_zone=us-west-2b, data_type=warm, ml.machine_memory=64388997120, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
     Hot threads at 2019-12-30T23:22:24.304Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
  
     44.0% (220.2ms out of 500ms) cpu usage by thread 'elasticsearch[warm-xxx][management][T#1]'
       3/10 snapshots sharing following 57 elements
         org.elasticsearch.index.engine.Engine.segmentsStats(Engine.java:831)
         org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:1051)
         org.elasticsearch.action.admin.indices.stats.CommonStats.&lt;init&gt;(CommonStats.java:213)
         org.elasticsearch.indices.IndicesService.indexShardStats(IndicesService.java:403)
         org.elasticsearch.indices.IndicesService.statsByShard(IndicesService.java:357)
         org.elasticsearch.indices.IndicesService.stats(IndicesService.java:348)
  ...
  
     42.7% (213.4ms out of 500ms) cpu usage by thread 'elasticsearch[warm-xxx][search][T#2]'                                                                                             
       10/10 snapshots sharing following 21 elements
         org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:263)
         org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:214)
         org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
         org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:670)
         org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
         org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
  ...
     41.8% (208.9ms out of 500ms) cpu usage by thread 'elasticsearch[warm-xxx][search][T#7]'
       10/10 snapshots sharing following 21 elements
         org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:263)
         org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:214)
         org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
         org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:670)
         org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
         org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
  
  </code></pre>
  
  <p>I'm running ElasticSearch 6.8</p>