The "ragel" branch is an experimental branch in which I'll be exploring
the feasibility of replacing the existing ANTLR lexer with a
Ragel-based scanner.

I suspect that Ragel may be faster, but a mere suspicion isn't really
very helpful: the difference between the two engines needs to be
profiled. To that end I am going to try making a Ragel-based scanner
which can effectively spit out a series of tokens (akin to the ANTLR
lexer's "nextToken" function), and just drop that in and measure the
performance.

One reason why I suspect the Ragel-based solution might be superior is
that it is relatively straightforward to do everything in UTF-8 instead
of the UCS-2 that the ANTLR C runtime defaults to. Doing everything in
UTF-8 should remove a lot of clutter form the code because it will no
longer be necessary to perform all those cumbersome conversions to and
from UCS-2, nor shadow the instance variables and maintain arrays of
16-bit chars. For 99% of expected input the UTF-8 solution will be
faster and less memory intensive (because 99% of input will be subset
of 7-bit ASCII). Additionally it will mean that we can support the
entire range of Unicode codepoints rather than the subset that can be
expressed in UCS-2.

Having said all that, you in theory can add support for UTF-8 input
streams to the ANTLR runtime, but my personally assessment of the
difficulty level of this is that it is a non-trivial task.

Other reasons why I'd like to move away from ANTLR if I can include:

  - break away from dependency on Java-based toolchain
  - generate code with zero runtime dependencies (no libraries)
  - pure state machines are easy to understand (no "black magic")

Add notes to "ragel" branch
Add notes to "ragel" branch

Add text file explaining what the "ragel" branch is and what it will be
used for.

Signed-off-by: Wincent Colaiuta <c76ac2e9cd86441713c7dfae92c451f3e6d17fdc@wincent.com>
