title: How to improve percolator performance in ElasticSearch?
body: <p><strong>Summary</strong></p>
  
  <p>We need to increase percolator performance (throughput).</p>
  
  <p>Most likely approach is scaling out to multiple servers.</p>
  
  <p><strong>Questions</strong></p>
  
  <p>How to do scaling out right?</p>
  
  <p>1) Would increasing number of shards in underlying index allow running more percolate requests in parallel?</p>
  
  <p>2) How much memory does ElasticSearch server need if it does percolation only?</p>
  
  <p>Is it better to have 2 servers with 4GB RAM or one server with 16GB RAM?</p>
  
  <p>3) Would having SSD meaningfully help percolator's performance, or it is better to increase RAM and/or number of nodes?</p>
  
  <p><strong>Our current situation</strong></p>
  
  <p>We have 200,000 queries (job search alerts) in our job index.
  We are able to run 4 parallel queues that call percolator.
  Every query is able to percolate batch of 50 jobs in about 35 seconds, so we can percolate about:</p>
  
  <blockquote>
    <p>4 queues * 50 jobs per batch / 35 seconds * 60 seconds in minute = 343
    jobs per minute</p>
  </blockquote>
  
  <p>We need more.</p>
  
  <p>Our jobs index have 4 shards and we are using .percolator sitting on top of that jobs index.</p>
  
  <p>Hardware: 2 processors server with 32 cores total. 32GB RAM.
  We allocated 8GB RAM to ElasticSearch.</p>
  
  <p>When percolator is working, 4 percolation queues I mentioned above consume about 50% of CPU.</p>
  
  <p>When we tried to increase number of parallel percolation queues from 4 to 6, CPU utilization jumped to 75%+.
  What is worse, percolator started to fail with NoShardAvailableActionException:</p>
  
  <blockquote>
    <p>[2015-03-04 09:46:22,221][DEBUG][action.percolate         ] [Cletus
    Kasady] [jobs][3] Shard multi percolate failure
    org.elasticsearch.action.NoShardAvailableActionException: [jobs][3]
    null</p>
  </blockquote>
  
  <p>That error seems to suggest that we should increase number of shards and eventually add dedicated ElasticSearch server (+ later increase number of nodes).</p>
  
  <p>Related:
  <a href="https://stackoverflow.com/questions/28293715/how-to-optimize-elasticsearch-percolator-index-memory-performance">How to Optimize elasticsearch percolator index Memory Performance</a></p>