1. In a computer system, a method for processing data stored as input strings, the input strings including a plurality of sub-strings, each sub-string associated with a value, the values of the sub-strings distributed over an interval, the interval of the values including a plurality of ranges of the values, comprising arranging the sub-strings into disjoint sets, each set including the sub-strings having values selected from a predetermined range of the interval of the values, a prefix of the sub-strings of each set being identical; and for each input string repeatedly truncating the input string, by deleting the prefix of a leading sub-string of the input string to produce a token in an output string, until the input strings have been parsed into the tokens of the output strings.
6. The method of claim 1 wherein the output strings include a plurality of encoded sub-strings, each encoded sub-string associated with an encoded value, the encoded values distributed over an other interval, the other interval of the encoded values including a plurality of other ranges of the encoded values, and further comprising arranging the encoded sub-strings into disjoint encoded sets, each encoded set including an other predetermined range of the encoded sub-strings, an encoded prefix of the encoded sub-strings of each encoded set being identical; and for each output string, repeatedly truncating the output string, by deleting the encoded prefix of a leading encoded sub-string of the output string to produce an encoded token in an input string, until the output strings have been parsed into the encoded tokens of the input strings.