title: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: search_context_missing_exception: No search context found for id
body: <p>Spark tasks are failing because of "No search context found for id". I tried a couple of options like </p>
  
  <blockquote>
    <p>spark.es.input.max.docs.per.partition 250</p>
    
    <p>spark.es.scroll.size 100</p>
    
    <p>spark.es.batch.size.bytes 32mb</p>
  </blockquote>
  
  <p>But tasks are still failing. we are using: 
      <strong>Spark version: 2.3.1,
      Elasticsearh Cluster version: 6.5.4,
      elasticsearch-spark version: 6.5.4</strong></p>
  
  <pre><code>org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: search_context_missing_exception: No search context found for id [4872076]
  {"scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAABKV4wWRTJHQW9CdEZRTHVqMldEWnQxTUJnQQ=="}
      at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:443)
      at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:400)
      at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:382)
      at org.elasticsearch.hadoop.rest.RestClient.scroll(RestClient.java:458)
      at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:323)
      at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:115)
      at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
      at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:533)
      at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216)
      at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1092)
      at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083)
      at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018)
      at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083)
      at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809)
      at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:347)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:298)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:42)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:336)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:300)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      at org.apache.spark.scheduler.Task.run(Task.scala:112)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:384)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>Any help would be appriciated.</p>