title: Does PLY's lexer support "maximal munch"?
body: <p>The syntax of many programming languages requires that they be tokenized according to the <a href="https://en.wikipedia.org/wiki/Maximal_munch" rel="nofollow">"maximal munch"</a> principle. That is, that tokens be built from the maximum possible number of characters from the input stream.</p>
  
  <p>PLY's lexer does not seem to apply this principle. For example:</p>
  
  <pre><code>import ply.lex as lex
  
  tokens = ('ASSIGNMENT', 'EQUALITY')
  
  t_ASSIGNMENT = r'[+\-*/]?='
  t_EQUALITY   = r'=='
  
  lexer = lex.lex()
  
  lexer.input('==')
  for tok in lexer:
      print(tok)
  </code></pre>
  
  <p>According to "maximal munch", the output of this code should be <code>LexToken(EQUALITY,'==',1,0)</code>, but it is <code>LexToken(ASSIGNMENT,'=',1,0) LexToken(ASSIGNMENT,'=',1,1)</code>. This seems to be because the lexer prefers <code>ASSIGNMENT</code> over <code>EQUALITY</code> - prioritizing the longer regular expression rather than the longer match.</p>
  
  <p>Is it possible to get PLY's lexer to follow the "maximal munch" principle?</p>
  
  <p>If not, are there any guidelines for how tokens should be specified to avoid "less-than-maximal munch" situations such as the one above?</p>