8. The method of claim 3 further comprising representing each set and each corresponding encoding as an entry of a dictionary, each set of ordered sub-strings expressed in the entry as a lower bound and an upper bound of the range of the ordered sub-strings included in the set.
8. The method of claim 3 further comprising representing each set and each corresponding encoding as an entry of a dictionary, each set of ordered sub-strings expressed in the entry as a lower bound and an upper bound of the range of the ordered sub-strings included in the set. 9. The method of claim 8 further comprising for each leading sub-string of each input string, searching the entries of the dictionary for a particular entry expressing a particular range of the sub-strings containing the leading sub-string, and selecting the associated prefix to truncate the input string to produce the token, and selecting the encoding expressed in the particular entry to encode the token.