Lexer performance is a concern, and optimizing is worthwhile, more so in stable languages where the lexer runs very often (such as C or HTML). lex/flex-generated lexers are reasonably fast, but improvements of two to three times are possible using more tuned generators. Hand-written lexers are sometimes used, but modern lexer generators produce faster lexers than most hand-coded ones. The lex/flex family of generators uses a table-driven approach which is much less efficient than the directly coded approach. With the latter approach the generator produces an engine that directly jumps to follow-up states via goto statements. Tools like re2c have proven to produce engines that are between two and three times faster than flex produced engines. It is in general difficult to hand-write analyzers that perform better than engines generated by these latter tools.
Lexer performance is a concern and optimizing is worthwhile more so in stable languages where the lexer runs very often (such as C or HTML) lex/flex-generated lexers are reasonably fast but improvements of two to three times are possible using more tuned generators Hand-written lexers are sometimes used but modern lexer generators produce faster lexers than most hand-coded ones The lex/flex family of generators uses a table-driven approach which is much less efficient than the directly coded approach With the latter approach the generator produces an engine that directly jumps to follow-up states via goto statements Tools like re2c have proven to produce engines that are between two and three times faster than flex produced engines It is in general difficult to hand-write analyzers that perform better than engines generated by these latter tools