title: What is the beginning and end of a Lucene token stream when parsing a query?
body: <p>We're trying to implement a custom filter that needs to remember past tokens that were processed before within the same query. We tried overriding the <code>end()</code> and/or <code>reset()</code> methods from <code>TokenFilter</code> but found out that these methods were called between each token. </p>
  
  <p>This was contrary to our expectation of the <code>end()</code> and/or <code>reset()</code> methods only being called at the beginning or end of the token stream representing the query. This unexpected behavior can be reproduced by the following example code below.</p>
  
  <p>(solr) schema snippet:</p>
  
  <pre><code>&lt;fieldType name="text_general" class="solr.TextField"&gt;
    &lt;analyzer type="index"&gt;
      ...
    &lt;/analyzer&gt;
    &lt;analyzer type="query"&gt;
      &lt;tokenizer class="solr.WhitespaceTokenizerFactory"/&gt;
      &lt;filter class="com.foobar.solr.CustomFilterFactory" /&gt;
    &lt;/analyzer&gt;
  &lt;/fieldType&gt;
  </code></pre>
  
  <p>Filter implementation:</p>
  
  <pre><code>public class CustomFilter extends TokenFilter {
      CharTermAttribute termAttribute = addAttribute(CharTermAttribute.class);
  
      public CustomFilter(TokenStream in) {
          super(in);
      }
  
      @Override
      public boolean incrementToken() throws IOException {
          System.out.println("### increment token pre loop: " + termAttribute.toString());
  
          while (input.incrementToken()) {
              System.out.println("### increment token looping through input: " + termAttribute.toString());
          }
  
          return false;
      }
  
      @Override
      public void end() throws IOException {
          System.out.println("### end");
          super.end();
      }
  
      @Override
      public void reset() throws IOException {
          System.out.println("### reset");
          super.reset();
      }
  }
  </code></pre>
  
  <p>Log output for the query "foo bar":</p>
  
  <pre><code>### reset
  ### increment token pre loop:
  ### increment token looping through input: foo
  ### end
  ### reset
  ### increment token pre loop:
  ### increment token looping through input: bar
  ### end
  </code></pre>
  
  <p>Why are the <code>end()</code> and <code>reset()</code> methods called for each token instead of for the complete query?</p>
  
  <p><strong>Edit:</strong> Or why is <code>input.incrementToken()</code> returning <code>false</code> after the first token was processed?</p>