title: ElasticSearch/Logstash/Kibana How to deal with spikes in log traffic
body: <p>What is the best way to deal with a surge in log messages being written to an ElasticSearch cluster in a standard ELK setup?</p>
  
  <p>We use a standard ELK (ElasticSearch/Logstash/Kibana) set-up in AWS for our websites logging needs. </p>
  
  <p>We have an autoscaling group of Logstash instances behind a load balancer, that log to an autoscaling group of ElasticSearch instances behind another load balancer. We then have a single instance serving Kibana.</p>
  
  <p>For day to day business we run 2 Logstash instances and 2 ElasticSearch instances.</p>
  
  <p>Our website experiences short periods of high level traffic during events - our traffic increases by about 2000% during these events. We know about these occurring events well in advance.</p>
  
  <p>Currently we just increase the number of ElasticSearch instances temporarily during the event. However we have had issues where we have subsequently scaled down too quickly, meaning we have lost shards and corrupted our indexes.</p>
  
  <p>I've been thinking of setting the <code>auto_expand_replicas</code> setting to <code>"1-all"</code> to ensure each node has a copy of all the data, so we don't need to worry about how quickly we scale up or down. How significant would the overhead of transferring all the data to new nodes be? We currently only keep about 2 weeks of log data - this works out around 50gb in all.</p>
  
  <p>I've also seen people mention using a separate auto scaling group of non-data nodes to deal with increases of search traffic, while keep the number of data nodes the same. Would this help in a write heavy situation, such as the event I previously mentioned?</p>