1 ) A method for performing lexical analysis, comprising the steps of: creating a first table, wherein the first table describes one or more single character transitions using records of type ET_onecat; creating a second table, wherein the second table is an ordered series of records of type ET_catrange; responsive to text input, searching through the records of the first table searching for each text input character; if a matching record is identified in the first table, providing the token associated with the matching record as output; responsive to a failure to locate a matching record in the first table, searching the records of the second table from the beginning for each input character, wherein a match is found when the current analyzer state lies between the upper state and lower state bounds and the input character lies between the upper character and lower character bounds, in each case as specified in the record being searched; if a matching record in the second table is found, assigning the current analyzer state the value of the ‘estat’ field in the matching record; if an exit condition is reached, terminating the loop.