title: Why aren't defined tokens recognized?
body: <p>(using antlr-4.1-complete.jar)</p>
  
  <p>I'm a newbie to ANTLR4, and created a simple grammar to test out what I learned.  I have found the results to be baffling, and I feel that there must be some basic concept I have (through all my reading) managed to not grasp.</p>
  
  <p>Here is the grammar I defined:</p>
  
  <pre><code>grammar Gm;
  
  // parser RULES
  gmModel
      : obj+
      EOF
      ;
  
  obj : objItem nameItem begin pairs end ;
  
  objItem : OBJDECL ;
  nameItem : OBJNAME ;
  begin : OPEN ;
  end : CLOSE ;
  pairs : Pair+ ;
  
  // LEXER RULES
  OBJDECL : 'object' ;
  OBJNAME : 'anotherobj' | 'testobj';
  OPEN : '{' ;
  CLOSE : '}' ;
  
  fragment Var : ID_START (ID_CONT)+ ; // variable names start with an alpha
  fragment Val : .*? ';' ; // values can be anything; capture everything up to the semicolon
  
  Pair : Var Val ;
  
  // fragments
  fragment ID_START : [a-zA-Z] ;
  fragment ID_CONT : [-_a-zA-Z0-9] ;
  
  WS : [ \t]+ -&gt; skip ;
  </code></pre>
  
  <p>And here is the file I'm using as input. Each object has some number of name-value groups. Most of the time, it is a pair - there is only one value (space/tab delimited) but sometimes, there are two values following the name which is why it captures to the semi-colon</p>
  
  <pre><code>object testobj {
      testvar testval;
      v_B 1234.9876;
  }
  
  object anotherobj {
      name o611;
      phases "CN";
      v_A 2401.7771;
      groupid OBJTEST;
      timeslice 1 hour;
  }
  </code></pre>
  
  <p>Even though I define</p>
  
  <pre><code>OBJDECL : 'object' ;
  </code></pre>
  
  <p>as a Lexer rule, it is not recognized as a token (TestRig basically shows several globs of text).
  I even put it first, because I read that first-defined tokens take precedence.  I also read that the longest possible token is used, and maybe that's what's happening here, but in that case, I am failing to understand how to define useful tokens that can match strings of unpredictable length.</p>
  
  <p>Any help you could offer would be hugely appreciated!</p>
  
  <p>Also, a final comment after more tinkering: if I make the following two changes, the tokenizing is correct, EXCEPT that each pair only consists of a semicolon (where did the contents go?)</p>
  
  <pre><code>pairs : Pair+ ;
  -&gt;
  pairs : pair+ ;
  </code></pre>
  
  <p>and</p>
  
  <pre><code>Pair : Var Val ;
  -&gt;
  pair : ID_START (ID_CONT)+ .*? ';' ;
  </code></pre>
  
  <p>(in other words, make 'pair' a parser rule and replace the Var/Val fragments with an in-place definition)</p>
  
  <p>From my limited understanding, I think that this change "improves" things because I am effectively removing the longest-possible token that was matchable before (Pair+), but I am still confused about why the name/value(s) are now missing altogether and I'm also (obviously!) still not clear on the recommended way of creating tokens in order to ensure they are not too greedy.</p>
  
  <p>Why is only the semi-colon getting picked up now?  Not understanding that is one reason I feel I am still missing something basic.  (But in case you can't tell from my frequent edits, I have been thinking about this a lot and have read a lot of other previously-posted questions  and answers in order to not ask a dumb question, but I really am stumped.)</p>
  
  <p>Thank you for any guidance!</p>