title: scroll elastic search, call an API and store in elastic search asynchronous
body: <p>I have a elastic search index (30GB, 1M entries), I have to update each document with calling an API (1s execution) and do some calculus on result.</p>
  <p>I try many things like send request with aiohttp, create a Queue with asyncio, ect</p>
  <p>The only things who work is that:</p>
  <pre><code>def search_and_store():
      for i in range(1, count, CPU):
          hits = es.search(index=['index'], size=CPU, from_=i)['hits']['hits']
          data = [{&quot;title&quot;: hit['_source']['name'], &quot;body&quot;: hit['_source']['cleanText'],&quot;id&quot;: hit[&quot;_id&quot;]} for hit in
                  hits]
          print(i)
          with Pool(CPU) as p:
              results = p.map(getresult_store, data)
              p.close()
              p.join()
  
  def getresult_store(item):
      body = ScoreManager.get_article_score(item['title'], item['body'])
      elasticsearchManager.update_record(es, &quot;searchdata&quot;, body, item['id'])
  </code></pre>
  <p>But I sure that there is a best way to do this</p>