title: Understanding the search thread pool of elasticsearch
body: <h2>Background:</h2>
  
  <p>I am using Datadog integration with elasticsearch to monitor the ES clusters, one important metric which it shows on its dashboard is the no of active and waiting for search threads. Referring to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html" rel="nofollow noreferrer">this</a> ES docs, I understand that search threads work on a request queue in ES which is of the fixed size of 1000.</p>
  
  <h2>Problem</h2>
  
  <p>I am seeing a lot of waiting for threads as shown in the image, but there is no rejected queue exception explained <a href="https://discuss.elastic.co/t/rejected-execution-queue-capacity-1000/89954/2" rel="nofollow noreferrer">here</a>. So it means ES is not rejecting the requests but still search threads are not able to execute the request fast enough hence ended up in waiting status for a long time. </p>
  
  <p><strong>Questions</strong></p>
  
  <ol>
  <li>How Search request queue works exactly, is new request comes to this queue and removed as soon as it's picked by a thread?</li>
  <li>I Know ES rejecting the request is definitely explains that ES is under-pressure but is there is any way to show that metric in Datadog dashboard, I couldn't find the relevant metric mentioned in <a href="https://docs.datadoghq.com/integrations/elastic/" rel="nofollow noreferrer">Datadog site</a> if not is there is any API which shows the historical count of these.</li>
  <li><strong>Our ES cluster CPU usage is below 45 even during peak time%</strong>, Still, we see a lot of waiting for search threads, so is it possible that our ES configuration isn't optimized? if yes, what are the ways to improve it.</li>
  </ol>
  
  <p>I know its a board question, hence let me know if any additional information is required.</p>
  
  <p><a href="https://i.stack.imgur.com/eYDZT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eYDZT.png" alt="enter image description here"></a></p>