title: Solr server memory and disk space
body: <p><strong>Context:</strong></p>
  
  <p>I have an AWS EC2 instance </p>
  
  <ul>
  <li>8Gb RAM </li>
  <li>8Gb of disk space</li>
  </ul>
  
  <p>It runs <strong>Solr 5.1.0</strong> with</p>
  
  <ul>
  <li>Java Heap of 2048Mb</li>
  <li><code>-Xms2048m -Xmx2048m</code></li>
  </ul>
  
  <p>Extra: (updated)</p>
  
  <ul>
  <li>Logs are generated on the server</li>
  <li>Imports happen in intervals of 10s (always delta)</li>
  <li>Importing from DB (<code>JdbcDataSource</code>)</li>
  <li>I don't think I have any optimization strategy configured right now</li>
  <li>GC profiling? I don't know.</li>
  <li>How can I find out how large the fields are .. and what is large?</li>
  </ul>
  
  <p><strong>Situation:</strong></p>
  
  <p>The index on Solr has 200.000 documents and is queried not more than once per second. However, in about 10 days, the memory <strong>and</strong> disk space of the server reaches 90% - 95% of the available space.</p>
  
  <p>When investigating the disk usage <code>sudo du -sh /</code> it only returns a total of <code>2.3G</code>. Not nearly as much as what <code>df -k</code> tells me (<code>Use% -&gt; 92%</code>).</p>
  
  <p>I can, sort of, resolve the situation by restarting the Solr service.</p>
  
  <p>What am i missing? How come Solr consumes all memory and disk space and how to prevent it?</p>
  
  <p><strong>Extra info for @TMBT</strong></p>
  
  <p>Sorry for the delay, but I’ve been monitoring the Solr production server for the last few days. You can see a roundup here:
  <a href="https://www.dropbox.com/s/x5diyanwszrpbav/screencapture-app-datadoghq-com-dash-162482-1468997479755.jpg?dl=0" rel="nofollow">https://www.dropbox.com/s/x5diyanwszrpbav/screencapture-app-datadoghq-com-dash-162482-1468997479755.jpg?dl=0</a>
  The current state of Solr: <a href="https://www.dropbox.com/s/q16dc5t5ctl32od/Screenshot%202016-07-21%2010.29.13.png?dl=0" rel="nofollow">https://www.dropbox.com/s/q16dc5t5ctl32od/Screenshot%202016-07-21%2010.29.13.png?dl=0</a>
  I restarted Solr at the beginning of the monitoring and now, 2 days later I see the disk space goes down at a rate of 1,5Gb per day.
  If you need more specifics, let me know.</p>
  
  <ul>
  <li>There are not so many deleted docs per day. We’re talking 50 - 250 per day max.</li>
  <li>The current logs directory of Solr: <code>ls -lh /var/solr/logs</code> -> <code>total 72M</code></li>
  <li>There is no master-slave setup</li>
  <li>The importer runs ever 10 seconds, but it imports no more than 10 - 20 docs each time. The large import of 3k-4k docs happens each night. There is not much action going on in Solr at that time.</li>
  <li>There are no large fields, the largest field can contain up to 255 chars.</li>
  </ul>
  
  <p>With the monitoring in place I tested the most common queries. It does contain faceting (field, queries), sorting, grouping, … But I doesn’t really affect the various metrics of heap and gc count.</p>