title: How to delete older logs in ELK to give each application a certain disk quota
body: <p>I am trying to use the ELK (Elasticsearch+Logstash+Kibana) stack in the following scenario:</p>
  
  <p>I have about ten applications that send their logs, through Logstash, to a single Elasticsearch cluster.</p>
  
  <p>Some of these applications naturally generate more logs than others, and, sometimes, one of them can go 'crazy', because of a bug, for instance, and, thus, generate even more log entries than it normally does. As a result, the disk space available in the cluster can be unfairly 'taken' by the logs of a single application, leaving not enough room to others.</p>
  
  <p>I am currently managing the available disk space through Elasticsearch Curator. It runs periodically, as it is in the crontab, and deletes older indices based on a disk usage quota. When the disk space used by all indices exceeds a certain limit, the oldest indices are deleted, one by one, until the sum of the disk space used by them all is within the specified limit again.</p>
  
  <p>The first problem with this approach is that Elasticsearch Curator can only delete entire indices. Hence, I had to configure Logstash to create one different index per hour, and increase their granularity; thus, Curator deletes smaller chunks of logs at a time. In addition, it is very difficult to decide how often Curator should run. If applications are generating logs at a higher rate, not even one-hour indices may be enough. Secondly, there is no way to specify a disk usage quota for each different application.</p>
  
  <p>Ideally, Elasticsearch should be able to delete older log entries by itself whenever the indices reach a certain disk usage limit. This would eliminate the problem of defining how often Curator should run. However, I could not find any similar feature in the Elasticsearch manual. </p>
  
  <p>Would anybody recommend a different approach to address these issues?</p>
  
  <p>References:
  <a href="http://www.elasticsearch.org" rel="noreferrer">http://www.elasticsearch.org</a>
  <a href="https://github.com/elasticsearch/curator" rel="noreferrer">https://github.com/elasticsearch/curator</a></p>