title: Lucene.net runs out of memory for large index
body: <p>I have a Lucene.net index with 10 fields, some stored and some indexed, with 460 million documents.  The index is about 250GB.  I'm using Lucene.net 3.0.3 and every time I do a search I easily eat up 2GB+ in RAM, which causes my 32 bit application to get out of memory exceptions.  I unfortunately cannot run the app as a 64 bit process due to other 32 bit dependencies.</p>
  
  <p>As far as I know I'm following Lucene best practices:</p>
  
  <ul>
  <li><p>One open index writer that writes documents in batches</p></li>
  <li><p>A shared reader that doesn't close and reopen itself across searches</p></li>
  <li><p>The index searcher has a <code>termInfosIndexDivisor</code> set to 4, which didn't seem to make a difference. I even tried setting it to something huge like 1000 but didn't notice any memory changes.</p></li>
  <li><p>Fields that do not need to be subsearched aren't analyzed (i.e. full string searching only) and fields that don't need to be retrieved back from the search aren't stored.  </p></li>
  <li><p>I'm  using the default <code>StandardAnalyzer</code> for both indexing and searching. </p></li>
  <li><p>If I prune the data and make a smaller index, then things do work.  When I have an index that is around 50GB in size I can search it with only about 600MB of RAM</p></li>
  </ul>
  
  <p>However, I do have a sort applied on one of the fields, but even without the sort the memory usage is huge for any search.  I don't particularly care about document score, more that the document exists in my index, but I'm not sure if somehow ignoring the score calculation will help with the memory usage.</p>
  
  <p>I recently upgraded from Lucene.net 2.9.4 to Lucene.net 3.0.3 thinking that that might help, but the memory usage looks about the same between the two versions.  </p>
  
  <p>Frankly I'm not sure if this index is just too large for a single machine to feasbily search or not. Most examples I find talk about indexes 20-30GB in size or less so maybe this isn't possible, but I wanted to at least ask.</p>
  
  <p>If anyone has any suggestions on what I can do to make this useable that would be great. I am willing to sacrifice search speed for memory usage if possible.</p>