title: How to improve Lucene performance in a distributed environment?
body: <p>I'm facing long search times (order of 10s of seconds) while searching on a master -  shard implementation in a distributed environment. However the same query via Luke returns in milliseconds.</p>
  
  <p>The application is a distributed system. All the nodes share a common NFS mount where the indexes reside. For simplicity lets consider two nodes <code>Node1</code> and <code>Node2</code>. The <code>/etc/fstab</code> entries is as follows.</p>
  
  <pre><code>nfs:/vol/indexes /opt/indexes nfs rw,suid,nodev,rsize=32768,wsize=32768,soft,intr,tcp 0 0
  </code></pre>
  
  <p>There are multiple feeds (say <code>Feed1</code> and <code>Feed2</code> ) that hit the system and there is a shard for each of the feed per node and a master for each Feed. The indexes look like </p>
  
  <pre><code>Feed1-master
  Feed1-shard-Node1.com
  Feed1-shard-Node1.com0
  Feed1-shard-Node1.com1
  </code></pre>
  
  <p>The code that does the search is</p>
  
  <pre><code>FeedIndexManager fim = getManager(feedCode);
  searcher = fim.getSearcher();
  TopDocs docs = searcher.search(q, filter, start + max, sort);
  
  private FeedIndexManager getManager(String feedCode) throws IOException {
    if (!_managers.containsKey(feedCode)) {
      synchronized(_managers) {
        if (!_managers.containsKey(feedCode)) {
          File shard = getShardIndexFile(feedCode);
          File master = getMasterIndexFile(feedCode);
          _managers.put(feedCode, new FeedIndexManager(shard, master));
        }
      }
    }  
    return _managers.get(feedCode);
  }
  </code></pre>
  
  <p>The FeedIndexManager is as follows. </p>
  
  <pre><code>public class FeedIndexManager implements Closeable {
  
    private static final Analyzer WRITE_ANALYZER = makeWriterAnalyzer();
    private final Directory _master;
    private SearcherManager _searcherManager;
    private final IndexPair _pair;
  
    private int _numFailedMerges = 0;
    private DateTime _lastMergeTime = new DateTime();
  
    public FeedIndexManager(File shard, File master) throws IOException {
      _master = NIOFSDirectory.open(master, new SimpleFSLockFactory(master));
  
      IndexWriter writer = null;
      try {
        writer = new IndexWriter(_master,
                                 WRITE_ANALYZER, 
                                 MaxFieldLength.LIMITED);
      } finally {
        if (null != writer) {
          writer.close();
        }
        writer = null;
      }
  
      _searcherManager = new SearcherManager(_master);
      _pair = new IndexPair(_master,
                            shard, 
                            new IndexWriterBuilder(WRITE_ANALYZER));
    }
  
    public IndexPair getIndexWriter() {
      return _pair;
    }
  
    public IndexSearcher getSearcher() {
      try {
        return _searcherManager.get();
      }
      catch (IOException ioe) {
        throw new DatastoreRuntimeException(
          "When trying to get an IndexSearcher for " + _master, ioe);
      }
    }
  
    public void releaseSearcher(IndexSearcher searcher) {
      try {
        _searcherManager.release(searcher);
      }
      catch (IOException ioe) {
        throw new DatastoreRuntimeException(
          "When trying to release the IndexSearcher " + searcher
          + " for " + _master, ioe);
      }
    }
  
    /**
     * Merges the changes from the shard into the master.
     */
    public boolean tryFlush() throws IOException {
      LOG.debug("Trying to flush index manager at " + _master
                + " after " + _numFailedMerges + " failed merges.");
      if (_pair.tryFlush()) {
        LOG.debug("I succesfully flushed " + _master);
        _numFailedMerges = 0;
        _lastMergeTime = new DateTime();
        return true;
      }
      LOG.warn("I couldn't flush " + _master + " after " + _numFailedMerges
               + " failed merges.");
      _numFailedMerges++;
      return false;
    }
  
    public long getMillisSinceMerge() {
      return new DateTime().getMillis() - _lastMergeTime.getMillis();
    }
  
    public long getNumFailedMerges() {
      return _numFailedMerges;
    }
  
    public void close() throws IOException {
      _pair.close();
    }
  
    /**
     * Return the Analyzer used for writing to indexes.
     */
    private static Analyzer makeWriterAnalyzer() {
      PerFieldAnalyzerWrapper analyzer = 
        new PerFieldAnalyzerWrapper(new LowerCaseAnalyzer());
  
      analyzer.addAnalyzer(SingleFieldTag.ID.toString(), new KeywordAnalyzer());
      // we want tokenizing on the CITY_STATE field
      analyzer.addAnalyzer(AddressFieldTag.CITY_STATE.toString(),
              new StandardAnalyzer(Version.LUCENE_CURRENT));
      return analyzer;
    }
  }
  </code></pre>
  
  <p>The killer which consumes about 95-98% of the latency is this call, it takes about 20 seconds for a search, whereas if the index is opened via Luke it is in milliseconds. </p>
  
  <pre><code>TopDocs docs = searcher.search(q, filter, start + max, sort);
  </code></pre>
  
  <p>I've the following questions </p>
  
  <ol>
  <li><p>Is it sane to have multiple masters per feed or should I reduce it to only one master? The number of elements in the index is about 50 million.</p></li>
  <li><p>The latency is low on feeds where the number of entities is less than a million (sub second response). The feeds where the entities are over 2 million takes about 20 seconds. Should I maintain only 1 Shard per node against 1 Shard per node per feed?</p></li>
  <li><p>The merge from the Shard's to the master is attempted every every 15 seconds. Should this parameter be tweaked?</p></li>
  </ol>
  
  <p>I'm currently using Lucene 3.1.0 and JDK 1.6. The boxes are two 64-bit cores with 8&nbsp;GB of RAM. Currently the JVM runs with 4&nbsp;GB max.</p>
  
  <p>Any suggestion to improve the performance is highly appreciated. I've already carried out all standard performance tuning that is generally prescribed by Lucene. Thanks a lot for reading this lengthy post. </p>