title: ElasticSearch BulkShardRequest failed due to org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor
body: <p>I am storing logs into elastic search from my reactive spring application. I am getting the following error in elastic search:</p>
  <blockquote>
  <p>Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of processing of [129010665][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[logs-dev-2020.11.05][1]] containing [index {[logs-dev-2020.11.05][_doc][0d1478f0-6367-4228-9553-7d16d2993bc2], source[n/a, actual length: [4.1kb], max length: 2kb]}] and a refresh, target allocation id: WwkZtUbPSAapC3C-Jg2z2g, primary term: 1 on EsThreadPoolExecutor[name = 10-110-23-125-common-elasticsearch-apps-dev-v1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6599247a[Running, pool size = 2, active threads = 2, queued tasks = 221, completed tasks = 689547]]]</p>
  </blockquote>
  <p>My index settings:</p>
  <pre><code>{
          &quot;logs-dev-2020.11.05&quot;: {
          &quot;settings&quot;: {
              &quot;index&quot;: {
                  &quot;highlight&quot;: {
                      &quot;max_analyzed_offset&quot;: &quot;5000000&quot;
                  },
                  &quot;number_of_shards&quot;: &quot;3&quot;,
                  &quot;provided_name&quot;: &quot;logs-dev-2020.11.05&quot;,
                  &quot;creation_date&quot;: &quot;1604558592095&quot;,
                  &quot;number_of_replicas&quot;: &quot;2&quot;,
                  &quot;uuid&quot;: &quot;wjIOSfZOSLyBFTt1cT-whQ&quot;,
                  &quot;version&quot;: {
                  &quot;created&quot;: &quot;7020199&quot;
                  }
              }
          }
      }
  }
  </code></pre>
  <p>I have gone through this site:</p>
  <p><a href="https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster" rel="nofollow noreferrer">https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster</a></p>
  <p>I thought adjusting &quot;write&quot; size in thread-pool will resolve, but it is mentioned as not recommended in the site as below:</p>
  <blockquote>
  <p>Adjusting the queue sizes is therefore strongly discouraged, as it is like putting a temporary band-aid on the problem rather than actually fixing the underlying issue.</p>
  </blockquote>
  <p>So what else can we do improve the situation?</p>
  <p>Other info:</p>
  <ul>
  <li>Elastic Search version 7.2.1</li>
  <li>Cluster health is good and they are 3 nodes in cluster</li>
  <li>Index will be created on daily basis, there are 3 shards per index</li>
  </ul>