title: Token matching, but it shouldn't
body: <p>I am trying to parse Windows header files to extract function prototypes. Microsoft being Microsoft means that the function prototypes are not in a regular, easily parseable format. Routine arguments usually, but not always, are annotated with Microsoft's Structured Annotation Language, which starts with an identifier that begins and ends with an underscore, and may have an underscore in the middle. A SAL identifier may be followed by parentheses and contain a variety of compile-time checks, but I don't care about SAL stuff. Routines are generally annotated with an access specifier, which is usually something like WINAPI, APIENTRY, etc., but there may be more than one. There are cases where the arguments are specified only by their types, too. Sheesh!</p>
  
  <p>My grammar looks like this:</p>
  
  <pre><code>//
  // Parse C function declarations from a header file
  //
  
  grammar FuncDef;
  
  //
  // Parser rules
  //
  
  start :
                  func_def+
                  ;
  
  func_def :
                  'extern'? ret_type = IDENTIFIER access = access_spec routine = IDENTIFIER '(' arg_list* ')' ';'
                  ;
  
  sal_statement :
                  SAL_NAME SAL_EXPR?
                  ;
  
  access_spec :
                  ('FAR' | 'PASCAL' | 'WINAPI' | 'APIENTRY' | 'WSAAPI' | 'WSPAPI')?
                  ;
  
  argument :
                  sal_statement type = IDENTIFIER is_pointer = '*'? arg = IDENTIFIER
                  ;
  
  arg_list :
                  argument (',' argument)*
                  ;
  
  hex_number :
                  '0x' HEX_DIGIT+
                  ;
  
  //
  // Lexer rules
  //
  
  INTEGER :           Digit+;
  HEX_DIGIT :         [a-fA-F0-9];
  SAL_NAME :          '_' Capital (Letter | '_')+? '_';           // Restricted form of IDENTIFIER, so it must be first
  IDENTIFIER :        Id_chars+;
  SAL_EXPR :          '(' ( ~( '(' | ')' ) | SAL_EXPR )* ')';     // We don't care about anything within a SAL expression, so eat everything within matched and nested parentheses
  CPP_COMMENT :       '//' .*? '\r'? '\n'     -&gt; channel (HIDDEN);
  C_COMMENT :         '/*' .*? '*/'           -&gt; channel (HIDDEN);
  
  WS :                [ \t\r\n]+ -&gt; skip;                 // Ignore all whitespace
  
  fragment Id_chars : Letter | Digit | '_' | '$';
  fragment Capital :  [A-Z];
  fragment Letter :   [a-zA-Z];
  fragment Digit :    [0-9];
  </code></pre>
  
  <p>I am using the TestRig, and providing the following input:</p>
  
  <pre><code>PVOID WINAPI routine ();
  
  PVOID WINAPI routine (type param);
  
  extern int PASCAL FAR __WSAFDIsSet(SOCKET fd, fd_set FAR *);
  
  // comment
  /*
  Another comment*/
  
  int
  WSPAPI
  WSCSetApplicationCategory(
      _Out_writes_bytes_to_(nNumberOfBytesToRead, *lpNumberOfBytesRead) LPBYTE lpBuffer,
      _In_ DWORD PathLength,
      _In_reads_opt_(ExtraLength) LPCWSTR Extra,
      _When_(pbCancel != NULL, _Pre_satisfies_(*pbCancel == FALSE))
          DWORD ExtraLength,
      _In_ DWORD PermittedLspCategories,
      _Out_opt_ DWORD * pPrevPermLspCat,
      _Out_ LPINT lpErrno
      );
  </code></pre>
  
  <p>I'm getting this output:</p>
  
  <pre><code>[@0,0:4='PVOID',&lt;IDENTIFIER&gt;,1:0]
  [@1,6:11='WINAPI',&lt;'WINAPI'&gt;,1:6]
  [@2,13:19='routine',&lt;IDENTIFIER&gt;,1:13]
  [@3,21:22='()',&lt;SAL_EXPR&gt;,1:21]
  [@4,23:23=';',&lt;';'&gt;,1:23]
  [@5,28:32='PVOID',&lt;IDENTIFIER&gt;,3:0]
  [@6,34:39='WINAPI',&lt;'WINAPI'&gt;,3:6]
  [@7,41:47='routine',&lt;IDENTIFIER&gt;,3:13]
  [@8,49:60='(type param)',&lt;SAL_EXPR&gt;,3:21]
  [@9,61:61=';',&lt;';'&gt;,3:33]
  [@10,66:71='extern',&lt;'extern'&gt;,5:0]
  [@11,73:75='int',&lt;IDENTIFIER&gt;,5:7]
  [@12,77:82='PASCAL',&lt;'PASCAL'&gt;,5:11]
  [@13,84:86='FAR',&lt;'FAR'&gt;,5:18]
  [@14,88:99='__WSAFDIsSet',&lt;IDENTIFIER&gt;,5:22]
  [@15,100:124='(SOCKET fd, fd_set FAR *)',&lt;SAL_EXPR&gt;,5:34]
  [@16,125:125=';',&lt;';'&gt;,5:59]
  [@17,130:141='// comment\r\n',&lt;CPP_COMMENT&gt;,channel=1,7:0]
  [@18,142:162='/*\r\nAnother comment*/',&lt;C_COMMENT&gt;,channel=1,8:0]
  [@19,167:169='int',&lt;IDENTIFIER&gt;,11:0]
  [@20,172:177='WSPAPI',&lt;'WSPAPI'&gt;,12:0]
  [@21,180:204='WSCSetApplicationCategory',&lt;IDENTIFIER&gt;,13:0]
  [@22,205:568='(\r\n    _Out_writes_bytes_to_(nNumberOfBytesToRead, *lpNumberOfBytesRead) LPBYTE lpBuffer,\r\n    _In_ DWORD PathLength,\r\n    _In_reads_opt_(ExtraLength) LPCWSTR Extra,\r\n    _When_(pbCancel != NULL, _Pre_satisfies_(*pbCancel == FALSE))\r\nDWORD ExtraLength,\r\n    _In_ DWORD PermittedLspCategories,\r\n    _Out_opt_ DWORD * pPrevPermLspCat,\r\n    _Out_ LPINT lpErrno\r\n    )',&lt;SAL_EXPR&gt;,13:25]
  [@23,569:569=';',&lt;';'&gt;,22:5]
  [@24,572:571='&lt;EOF&gt;',&lt;EOF&gt;,23:0]
  line 1:21 mismatched input '()' expecting '('
  line 3:21 mismatched input '(type param)' expecting '('
  line 5:18 extraneous input 'FAR' expecting IDENTIFIER
  line 5:34 mismatched input '(SOCKET fd, fd_set FAR *)' expecting '('
  line 13:25 mismatched input '(\r\n    _Out_writes_bytes_to_(nNumberOfBytesToRead, *lpNumberOfBytesRead) LPBYTE lpBuffer,\r\n    _In_ DWORD PathLength,\r\n    _In_reads_opt_(ExtraLength) LPCWSTR Extra,\r\n    _When_(pbCancel != NULL, _Pre_satisfies_(*pbCancel == FALSE))\r\nDWORD ExtraLength,\r\n    _In_ DWORD PermittedLspCategories,\r\n    _Out_opt_ DWORD * pPrevPermLspCat,\r\n    _Out_ LPINT lpErrno\r\n    )' expecting '('
  </code></pre>
  
  <p>What I don't understand is why is SAL_EXPR matching on lines 3 and 8? It should only match something if SAL_NAME matches first.</p>
  
  <p>Why doesn't SAL_NAME match at line 22?</p>