title: ANTLR4: Consuming all remaining input until EOF
body: <p>Using ANTLR version 4.3 here.</p>
  
  <p>This grammar accepts a variety of <code>EOF</code>-delimited expressions, one at a time. The expressions start with <code>key</code> and vary in syntax thereafter. Sample accepted strings:</p>
  
  <pre><code>"cycle in freerun" &lt;EOF&gt;
  "runtime &lt;= 20m" &lt;EOF&gt;
  "grab enabled" &lt;EOF&gt;
  </code></pre>
  
  <p>ANTLR merrily parses all expressions into components, the listener acts on the relevant components, life is happy. Here is a representative snippet of the grammar:</p>
  
  <pre><code>expr               // &lt;-- Start rule
    : freq_p EOF
    | cycle_p EOF
    ...              // Many more, ad nauseum
    ;
  
  freq_p  : FREQ '=' INT | FREQ '&lt;' INT ;
  
  cycle_p : CYCLE IN cycles ;
  cycles  : cycle (',' cycle)* ;
  cycle   : PHASELOCK | FREERUN ;
  
  // Keywords
  CYCLE     : 'cycle' ;
  FREERUN   : 'freerun' ;
  FREQ      : 'frequency' ;
  IN        : 'in' ;
  PHASELOCK : 'phaselock' ;
  
  INT       : '0'..'9'+ ;
  WS        : [ \n\t\r]+ -&gt; skip ;
  </code></pre>
  
  <p>But now, I need to extend the grammar to incorporate 2 new expressions, both of which end with the acceptance of any sequence of characters at all (including Unicode) through to the <code>EOF</code>. Sample input:</p>
  
  <pre><code>"echo = Confirm 'interlock' is clear,\n and actuate \"frequency\" button." &lt;EOF&gt;
  "report Process complete." &lt;EOF&gt;
  </code></pre>
  
  <p>I'm having great difficulty expressing the acceptance of all input to <code>EOF</code> in the grammar. The following changes lead to misery:</p>
  
  <pre><code>expr
    ...
    : echo_p EOF
    : report_p EOF
    ...
  
  echo_p   : ECHO   '=' REMAINING ; // Snarfs all remaining input until EOF
  report_p : REPORT     REMAINING ; // Ditto
  
  ECHO     : 'echo' ;
  REPORT   : 'report' ;
  
  REMAINING : &lt;WHAT_GOES_HERE?&gt; ;  // .* messes up everything else
  </code></pre>
  
  <p>How can this be accomplished? The desired outcome is that the parse tree listener will obtain the text value such as <code>REMAINING().getText()</code>.</p>
  
  <h2>Abandoned Approach: Lexer Grammar utilizing mode()</h2>
  
  <p>I tried writing <code>REMAINING</code> into a separate <code>lexer grammar</code> and <code>import</code>ing that from the combined grammar, but ran into <a href="https://github.com/antlr/antlr4/issues/160" rel="nofollow">https://github.com/antlr/antlr4/issues/160</a> and compile-time warnings. The IntelliJ ANTLR plugin also malfunctions which is a negative impact on productivity. I learned that including a lexer grammar that uses modes is unsupported, at least in ANTLR 4.3.</p>
  
  <pre><code>lexer grammar Remainder;
  
  @lexer::members {
  // Needed at least until ANTLR issue #160 is fixed.
  public static final int CONSUME_ALL = 123;
  }
  
  REMAINING : . -&gt; more, mode(CONSUME_ALL) ;
  
  mode CONSUME_ALL;
  
  TEXT : .* ; // Consume all remaining input
  </code></pre>