title: TokenStream contract violation when using custom Analyzer with Lucene 4.9
body: <p>I have a few custom Analyzers like this one:</p>
  
  <pre><code>private static class ModelAnalyzer extends Analyzer
  {
      @Override
      protected TokenStreamComponents createComponents(String string, Reader reader)
      {
          return new TokenStreamComponents(
              new StandardTokenizer(Version.LUCENE_4_9, reader),
              new LowerCaseFilter(Version.LUCENE_4_9,
                  new NGramTokenFilter(Version.LUCENE_4_9,
                      new CharTokenizer(Version.LUCENE_4_9, reader)
                      {
                          @Override
                          protected boolean isTokenChar(int c)
                          {
                              return Character.isLetterOrDigit(c);
                          }
                      }, 3, 20)));
      }
  }
  </code></pre>
  
  <p>They are added to a PerFieldAnalyzerWrapper and added to my IndexWriterConfig.  When I try to rebuild my index I was always getting the error when adding the second document to my index:</p>
  
  <pre><code>java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
  </code></pre>
  
  <p>All I am doing is adding documents to my IndexWriter.  I am not in any way touching these Filters or Tokenizers so there is no clean way for me to call reset() on them.  Shouldn't the IndexWriter follow the "correct consuming workflow" without my help?  </p>
  
  <p>After 8 hours of reading everything on the web about this I gave up and just passed Version.LUCENE_4_5 to each of my tokenizers and filters so that the irritating state machine checks (that I understand were added in 4_6) are not run. This has fixed the problem, but I am at a loss as to the right way to make this work with 4.9. I have to assume I am making my Analyzers wrong or something, but I can't see how I could do it differently and it works just fine in the earlier version.</p>