title: SOLR Out Of Memory Error when reading/indexing a large index
body: <p>I'm running into a OOM error with Solr, when indexing large amounts of data. I know the general advice would be to split the index into shards, but actually this is already the case.  I'm indexing shards, and further splitting is at this point not an option. I want to understand what is happening and why am I getting this error and if there is anything I can do about it other than splitting or giving more RAM. </p>
  
  <p>I would be sad if the RAM consumption was linear (or worse) in this case, I'd rather have it sub-linear. </p>
  
  <p>The case is I'm indexing documents with random strings (thus the dictionary is very very large). Each document has a couple of 20-30 chars fields and one field about 200-500 chars. The index size in every shard is about 250-260GB, each solr instance handling this index has about 4GB of memory. When the OOM occured, after restarting it the Solr HeapDump looked about the same, so it's probably not tied to indexing but the Solr Searcher. Just before the OOM the heapdump's largest objects look like the following:</p>
  
  <pre><code>&lt;tree type="Heap walker - Biggest objects"&gt;
    &lt;object leaf="false" class="org.apache.solr.core.SolrCore" objectId="0xf02c" type="instance" retainedBytes="120456864" retainedPercent="97.4"&gt;
      &lt;outgoing leaf="false" class="org.apache.solr.search.SolrIndexSearcher" objectId="0xfb52" type="instance" retainedBytes="120383232" retainedPercent="97.3" referenceType="not specified" referenceName="[transitive reference]"&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x1018e" type="instance" retainedBytes="8161688" retainedPercent="6.6" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10185" type="instance" retainedBytes="8148072" retainedPercent="6.6" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10188" type="instance" retainedBytes="8138232" retainedPercent="6.6" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10186" type="instance" retainedBytes="8129160" retainedPercent="6.6" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10191" type="instance" retainedBytes="8124608" retainedPercent="6.6" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x1018a" type="instance" retainedBytes="8123144" retainedPercent="6.6" referenceType="not specified" referenceName="[transitive reference]"/&gt;
  
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10192" type="instance" retainedBytes="8100904" retainedPercent="6.5" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10190" type="instance" retainedBytes="8097984" retainedPercent="6.5" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x1018b" type="instance" retainedBytes="8096160" retainedPercent="6.5" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x1018d" type="instance" retainedBytes="8081656" retainedPercent="6.5" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10187" type="instance" retainedBytes="8042504" retainedPercent="6.5" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x1018c" type="instance" retainedBytes="8039336" retainedPercent="6.5" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10189" type="instance" retainedBytes="8036952" retainedPercent="6.5" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x1018f" type="instance" retainedBytes="7948568" retainedPercent="6.4" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10195" type="instance" retainedBytes="832448" retainedPercent="0.7" referenceType="not specified" referenceName="[transitive reference]"/&gt;
  
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10196" type="instance" retainedBytes="830584" retainedPercent="0.7" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10194" type="instance" retainedBytes="829232" retainedPercent="0.7" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10197" type="instance" retainedBytes="828808" retainedPercent="0.7" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10198" type="instance" retainedBytes="827312" retainedPercent="0.7" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10199" type="instance" retainedBytes="824736" retainedPercent="0.7" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x1019a" type="instance" retainedBytes="822608" retainedPercent="0.7" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;outgoing leaf="false" class="org.apache.lucene.index.ReadOnlySegmentReader" objectId="0x10193" type="instance" retainedBytes="783424" retainedPercent="0.6" referenceType="not specified" referenceName="[transitive reference]"/&gt;
        &lt;cutoff objectCount="96" totalSizeBytes="534976" maximumSingleSizeBytes="87560"/&gt;
      &lt;/outgoing&gt;
  
      &lt;cutoff objectCount="53" totalSizeBytes="73496" maximumSingleSizeBytes="40992"/&gt;
    &lt;/object&gt;
    &lt;object leaf="false" class="org.mortbay.jetty.webapp.WebAppClassLoader" objectId="0xdf88" type="instance" retainedBytes="420208" retainedPercent="0.3"/&gt;
    &lt;object leaf="false" class="org.apache.solr.core.SolrConfig" objectId="0xe5f5" type="instance" retainedBytes="184976" retainedPercent="0.1"/&gt;
   ..... 
  </code></pre>
  
  <p>The jmap simple dump looks like this:</p>
  
  <pre><code>Attaching to process ID 27000, please wait...
  Debugger attached successfully.
  Server compiler detected.
  JVM version is 20.5-b03
  
  using thread-local object allocation.
  Parallel GC with 2 thread(s)
  
  Heap Configuration:
     MinHeapFreeRatio = 40
     MaxHeapFreeRatio = 70
     MaxHeapSize      = 268435456 (256.0MB)
     NewSize          = 1310720 (1.25MB)
     MaxNewSize       = 17592186044415 MB
     OldSize          = 5439488 (5.1875MB)
     NewRatio         = 2
     SurvivorRatio    = 8
     PermSize         = 21757952 (20.75MB)
     MaxPermSize      = 85983232 (82.0MB)
  
  Heap Usage:
  PS Young Generation
  Eden Space:
     capacity = 31719424 (30.25MB)
     used     = 17420488 (16.61347198486328MB)
     free     = 14298936 (13.636528015136719MB)
     54.92056854500258% used
  From Space:
     capacity = 26673152 (25.4375MB)
     used     = 10550856 (10.062080383300781MB)
     free     = 16122296 (15.375419616699219MB)
     39.55608995892199% used
  To Space:
     capacity = 27000832 (25.75MB)
     used     = 0 (0.0MB)
     free     = 27000832 (25.75MB)
     0.0% used
  PS Old Generation
     capacity = 178978816 (170.6875MB)
     used     = 168585552 (160.7757110595703MB)
     free     = 10393264 (9.911788940429688MB)
     94.19302002757689% used
  PS Perm Generation
     capacity = 42008576 (40.0625MB)
     used     = 41690016 (39.758697509765625MB)
     free     = 318560 (0.303802490234375MB)
     99.24167865152106% used
  </code></pre>
  
  <p>I don't see here anything that would give me any clues as to how to deal with it, other than just giving more RAM, which in the general case is not a solution, I'd like to know what's going on, why is the Searcher and it's ReadOnlySegmentReaders taking all the memory and do they really have to, can I do something about it?</p>
  
  <p>UPDATE:
  I've done a test with a much smaller dictionary of about 150 thousand words (not random words), I reached the index size of about 350GB and there are no OOME, so this is not directly connected to index size, probably has to do more with the term vector size (unique terms). But still I'd like to understand the limitations I have and how can I go around them.</p>