Title: ANTLR tokenizing works with space, but not without
Tags: parsing;antlr;grammar;lexer
Question: I'm trying to figure out this bug. Currently, when I supply a few different inputs, I get either the correct result, or the ANTLRWorks debugger loops infinitely.
1) {v0, p1} = Infinite loop or mismatched token exception
2) {v0..v1} Works!
3) {v0,p1} Works? Why?! Notice the lack of space?
If anyone could help me understand why this is ambiguous, I'd really appreciate it. I've tried changing operator precedence, and still, I cannot get it to work the way I think it should. Thank you!
WHITESPACE    :  ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+ { $channel = HIDDEN; };
LEFTCURL      : '{';
RIGHTCURL     : '}';
REGISTER      : ('v'|'p') NUMBER+;
REGISTERLIST  : LEFTCURL REGISTER (',' REGISTER)* RIGHTCURL;
REGISTERANGE  : LEFTCURL REGISTER '..' REGISTER RIGHTCURL;

Here is the accepted answer: WHITESPACEs are only put on the HIDDEN channel inside parser rules. Since both REGISTERLIST and REGISTERANGE are lexer rules, the spaces are not hidden, as you noticed. Instead of accounting for spaces inside these rules, these look to be more suitable as parser rules:
registerList  : LEFTCURL REGISTER (',' REGISTER)* RIGHTCURL;
registerRange : LEFTCURL REGISTER '..' REGISTER RIGHTCURL;
Comment for this answer: I had this epiphany when I was riding my bike home yesterday and had not yet had a chance to try it out. Thanks so much for the advice!


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2013:00:09/18/13', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<parsing><antlr><grammar><lexer>', 'Title': 'ANTLR tokenizing works with space, but not without', 'ViewCount': 114, 'id': 18862325, 'user_AboutMe': '', 'user_AccountId': 2308083.0, 'user_DisplayName': 'Steve H.', 'user_Reputation': 233.0, 'user_Views': 34.0, 'user_id': 2026903.0}}