title: Very slow Solr performance when highlighting
body: <p>I have a Solr 4.4.0 core configured that contains about 630k documents with an original size of about 10 GB. Each of the fields gets copied to the <strong>text</strong> field for purposes of queries and highlighting. When I execute a search without highlight, the results come back in about <strong>100 milliseconds</strong>, but when highlighting is turned on, the same query takes <strong>10-11 seconds</strong>. I also noticed that <strong>subsequent</strong> queries for the same terms continued to take about the same 10-11 seconds.</p>
  
  <p>My initial configuration of the field was as follows</p>
  
  <pre><code>&lt;field name="text" type="text_general" indexed="true" stored="true"
     multiValued="true"
     omitNorms="true"
     termPositions="true"
     termVectors="true"
     termOffsets="true" /&gt;
  </code></pre>
  
  <p>The query that is sent is similar to the following</p>
  
  <pre><code>http://solrtest:8983/solr/Incidents/select?q=error+code&amp;fl=id&amp;wt=json&amp;indent=true&amp;hl=true&amp;hl.useFastVectorHighlighter=true
  </code></pre>
  
  <p>All my research seems to provide no clue as to why the highlight performance is so bad. On a whim, I decided to see if the <strong>omitNorms=true</strong> attribute could have an effect, I modified the <strong>text</strong> field, wiped out the data, and reloaded from scratch.</p>
  
  <pre><code>&lt;field name="text" type="text_general" indexed="true" stored="true"
     multiValued="true"
     termPositions="true"
     termVectors="true"
     termOffsets="true" /&gt;
  </code></pre>
  
  <p>Oddly enough, this seemed to fix things. The initial query with highlighting took <strong>2-3 seconds</strong> with subsequent queries taking less than <strong>100 milliseconds</strong>.</p>
  
  <p>However, because we want the <strong>omitNorms=true</strong> in place, my permanent solution was to have <strong>two</strong> copies of the "text" field, one with the attribute and one without. The idea was to perform queries against one field and highlighting against the other. So now the schema looks like</p>
  
  <pre><code>&lt;field name="text" type="text_general" indexed="true" stored="true"
     multiValued="true"
     omitNorms="true"
     termPositions="true"
     termVectors="true"
     termOffsets="true" /&gt;
  
  &lt;field name="text2" type="text_general" indexed="true" stored="true"
     multiValued="true"
     termPositions="true"
     termVectors="true"
     termOffsets="true" /&gt;
  </code></pre>
  
  <p>And the query is as follows</p>
  
  <pre><code>http://solrtest:8983/solr/Incidents/select?q=error+code&amp;fl=id&amp;wt=json&amp;indent=true&amp;hl=true&amp;hl.fl=text2&amp;hl.useFastVectorHighlighter=true
  </code></pre>
  
  <p>Again, the data was cleared and reloaded with the same 630k documents but this time the index size is about 17 GB. (As expected since the contents on the "text" field is duplicated.)</p>
  
  <p>The problem is that the performance numbers are back to the original 10-11 seconds each run. Either the first removal of omitNorms was a fluke or there is something else is going on. I have no idea what...</p>
  
  <p>Using jVisualVM to capture a CPU sample shows the following two methods using most of the CPU</p>
  
  <pre><code>org.apache.lucene.search.vectorhighlight.FieldPhraseList.&lt;init&gt;()    8202 ms (72.6%)
  org.eclipse.jetty.util.BlockingArrayQueue.poll()                     1902 ms (16.8%)
  </code></pre>
  
  <p>I have seen the init method as low as 54% and the poll number as high as 30%.</p>
  
  <p>Any ideas? Any other places I can look to track down the bottleneck?</p>
  
  <p>Thanks</p>
  
  <p><strong>Update</strong></p>
  
  <p>I have done a bunch of testing with the same dataset but different configurations and here is what I have found...although I do not understand my findings.</p>
  
  <ul>
  <li>Speedy highlighting performance requires that <strong>omitNorms not be set to true</strong>. (Have no idea what omitNorms and highlighting has to do with one another.)</li>
  <li>However, this is only seems to be true if both the query and highlighting are executed against the <strong>same</strong> field (i.e. df = hl.fl). (Again, no idea why...)</li>
  <li>And another however, only if done against the default <strong>text</strong> field that exists in the schema.</li>
  </ul>
  
  <p>Here is how I tested --></p>
  
  <ul>
  <li>Test was against about 525,000 documents</li>
  <li>Almost all of the fields were copied to the multi-valued <strong>text</strong> field</li>
  <li>In some tests, almost all of the fields were <strong>also</strong> copied to a send multi-valued <strong>text2</strong> field (this field was identical to <strong>text</strong> except it had the opposite <strong>omitNorms</strong> setting</li>
  <li>Each time the configuration was changed, the Solr instance was stopped, the data folder was deleted, and the instance was started back up</li>
  </ul>
  
  <p>What I found --></p>
  
  <ul>
  <li>When just the <strong>text</strong> field was used and <strong>omitNorms = true</strong> was <strong>present</strong>, performance was bad (10 second response time)</li>
  <li>When just the <strong>text</strong> field was used and <strong>omitNorms = true</strong> was <strong>not present</strong>, performance was great (sub-second response times)</li>
  <li>When <strong>text</strong> did <strong>not</strong> have <strong>omitNorms = true</strong> and <strong>text2</strong> did, queries wit highlighting against <strong>text</strong> returned in sub-second times, <strong>all other</strong> combinations resulted in 10-30 second response times.</li>
  <li>When <strong>text</strong> did have <strong>omitNorms = true</strong> and <strong>text2</strong> did <strong>not</strong>, <strong>all</strong> combinations of queries with highlighting returned in 7-10 seconds.</li>
  </ul>
  
  <p>I am soooo confused....</p>