title: Elastic search could not write all entries: May be es was overloaded
body: <p>I have an application where I read csv files and do some transformations and then push them to elastic search from spark itself. Like this</p>
  
  <pre><code>input.write.format("org.elasticsearch.spark.sql")
                .mode(SaveMode.Append)
                .option("es.resource", "{date}/" + type).save()
  </code></pre>
  
  <p>I have several nodes and in each node, I run 5-6 <code>spark-submit</code> commands that push to <code>elasticsearch</code></p>
  
  <p>I am frequently getting Errors </p>
  
  <pre><code>Could not write all entries [13/128] (Maybe ES was overloaded?). Error sample (first [5] error messages):
          rejected execution of org.elasticsearch.transport.TransportService$7@32e6f8f8 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4448a084[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 451515]]
  </code></pre>
  
  <p>My Elasticsearch cluster has following stats - </p>
  
  <pre><code>Nodes - 9 (1TB space,
  Ram &gt;= 15GB ) More than 8 cores per node
  </code></pre>
  
  <p>I have modified following parameters for elasticseach </p>
  
  <pre><code>spark.es.batch.size.bytes=5000000
  spark.es.batch.size.entries=5000
  spark.es.batch.write.refresh=false
  </code></pre>
  
  <p>Could anyone suggest, What can I fix to get rid of these errors?</p>