title: Elasticsearch drops too many requests -- would a buffer improve things?
body: <p>We have a cluster of workers that send indexing requests to a 4-node Elasticsearch cluster. The documents are indexed as they are generated, and since the workers have a high degree of concurrency, Elasticsearch is having trouble handling all the requests. To give some numbers, the workers process up to 3,200 tasks at the same time, and each task usually generates about 13 indexing requests. This generates an instantaneous rate that is between 60 and 250 indexing requests per second.</p>
  
  <p>From the start, Elasticsearch had problems and requests were timing out or returning 429. To get around this, we increased the timeout on our workers to 200 seconds and increased the <code>write</code> thread pool queue size on our nodes to 700.</p>
  
  <p>That's not a satisfactory long-term solution though, and I was looking for alternatives. I have noticed that when I copied an index within the same cluster with <code>elasticdump</code>, the <code>write</code> thread pool was almost empty and I attributed that to the fact that <code>elasticdump</code> batches indexing requests and (probably) uses the bulk API to communicate with Elasticsearch.</p>
  
  <p>That gave me the idea that I could write a buffer that receives requests from the workers, batches them in groups of 200-300 requests and then sends the bulk request to Elasticsearch for one group only.</p>
  
  <p>Does such a thing already exist, and does it sound like a good idea?</p>