Tokenizer and complex operators I'm trying to create simple tokenizer to transform following (only part shown) search expression to tokens

word1 near(1) word2


where word1, word2 are some words and near(1) is distance operator.
The question is how this expression should be tokenized. 
I see two ways

1. &lt;WORD, word1&gt; &lt;WORD, near&gt; &lt;LPAREN&gt; &lt;NUMBER,1&gt; &lt;RPAREN&gt; &lt;WORD, word2&gt;.
2. &lt;WORD, word1&gt; &lt;NEAROP, 1&gt; &lt;WORD, word2&gt;


But should I really try to tokenize NEAR(\d+) during tokenization, or should I go first way and handle NEAR operator at parser level, during building parse tree?
Since you indicate that the parameter of the near operator can be an arbitrary expression, it should be handled at the parser rather than at the lexer. Otherwise how would you handle things like near(x+y)?