title: ANTLR4 Lexer getTokens() returning 0 tokens
body: <p>I'm running code from here:  <a href="https://github.com/bkiers/antlr4-csv-demo" rel="noreferrer">https://github.com/bkiers/antlr4-csv-demo</a>.
  I want to view the tokens analyzed by the lexer by adding this line:</p>
  
  <pre><code>System.out.println("Number of tokens: " + tokens.getTokens().size())
  </code></pre>
  
  <p>to Main.java:</p>
  
  <pre><code>public static void main(String[] args) throws Exception {  
      // the input source  
      String source =   
          "aaa,bbb,ccc" + "\n" +   
          "\"d,\"\"d\",eee,fff";  
  
      // create an instance of the lexer  
      CSVLexer lexer = new CSVLexer(new ANTLRInputStream(source));  
  
      // wrap a token-stream around the lexer  
      CommonTokenStream tokens = new CommonTokenStream(lexer);  
  
      // look at tokens analyzed
      System.out.println("Number of tokens: " + tokens.getTokens().size())
  
      // create the parser  
      CSVParser parser = new CSVParser(tokens);  
  
      // invoke the entry point of our grammar  
      List&lt;List&lt;String&gt;&gt; data = parser.file().data;  
  
      // display the contents of the CSV source  
      for(int r = 0; r &lt; data.size(); r++) {  
        List&lt;String&gt; row = data.get(r);  
        for(int c = 0; c &lt; row.size(); c++) {  
          System.out.println("(row=" + (r+1) + ",col=" + (c+1) + ") = " + row.get(c));  
        }  
      }  
    }  
  </code></pre>
  
  <p>The result printed out is: <code>Number of tokens: 0</code>. Why is the list returned by <code>getTokens()</code> empty? The rest of the parser code returns the data completely fine.</p>
  
  <p>EDIT: So using <code>lexer.getAllTokens()</code> instead works, but why is the <code>CommonTokenStream</code> not returning the correct tokens?</p>
  
  <p>csv.g4:</p>
  
  <pre><code>grammar CSV;
  
  @header {
    package csv;
  }
  
  file returns [List&lt;List&lt;String&gt;&gt; data]  
  @init {$data = new ArrayList&lt;List&lt;String&gt;&gt;();}  
   : (row {$data.add($row.list);})+ EOF  
   ; 
  
  row returns [List&lt;String&gt; list]  
  @init {$list = new ArrayList&lt;String&gt;();}  
   : a=value {$list.add($a.val);} (Comma b=value {$list.add($b.val);})* (LineBreak | EOF)  
   ;
  
  value returns [String val]  
   : SimpleValue {$val = $SimpleValue.text;}  
   | QuotedValue   
     { 
       $val = $QuotedValue.text; 
       $val = $val.substring(1, $val.length()-1); // remove leading- and trailing quotes 
       $val = $val.replace("\"\"", "\""); // replace all `""` with `"` 
     }  
   ;  
  
  Comma  
   : ','  
   ;  
  
  LineBreak  
   : '\r'? '\n'  
   | '\r'  
   ;  
  
  SimpleValue  
   : ~[,\r\n"]+  
   ;  
  
  QuotedValue  
   : '"' ('""' | ~'"')* '"'  
   ;  
  </code></pre>