title: Antlr get Sub-Tokens
body: <p>Forgive me if my terminology is off.</p>
  
  <p>Lets say I have this bit of simplified grammar:</p>
  
  <pre><code>// parser
  expr : COMPARATIVE;
  
  // lexer
  WS : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+; 
  COMPARATOR 
          : 'vs'
      | 'versus'
          ;
  ITEM 
          : 'boy'
          | 'girl'
          ;
  COMPARATIVE :ITEM WS* COMPARATOR WS* ITEM;
  </code></pre>
  
  <p>So this will of course match <code>'boy vs girl'</code> or <code>'girl vs boy'</code>, etc.
  But my question is that is when I create a Lexer, i.e.</p>
  
  <pre><code>CharStream stream = new ANTLRInputStream("boy vs girl");
  SearchLexer lex = new SearchLexer(stream);
  CommonTokenStream tokens = new CommonTokenStream(lex);
  tokens.fill();
  for(Token token : tokens) {
      System.out.print(token.getType() + " [" + token.getText() + "] ");
  }
  </code></pre>
  
  <p>This prints out something like this:
  9 [boy vs girl], i.e. it matches my query fine, but now I want to be able to do something like, get the sub tokens of this current token.</p>
  
  <p>My intuition tells me I need to use trees, but really don't know how to do this in Antlr4 for my example. Thanks</p>