11. A computer program product comprising a non-transitory tangible computer usable storage medium having readable program code embodied in the non-transitory tangible computer usable storage medium, the computer program product includes at least one component operable to: determine an attribute of a current character in input text, the attribute of the current character indicating one or more classes of characters the current character is assigned thereto; determine one or more attributes of one or more next characters in the input text, the one or more attributes of the one or more next characters indicating the one or more classes the one or more next characters are assigned thereto; and construct a token of the input text that comprises the current character and the one or more next characters, the attribute of the current character and the one or more attributes of the one or more next characters intersecting with each other, and the attribute of the current character and the one or more attributes of the one or more next characters comprising an attribute data structure which comprises a one-byte array, wherein the one-byte array comprises a plurality of binary bits and each bit of the binary bits indicates a different class from remaining bits of the binary bits.
11. A computer program product comprising a non-transitory tangible computer usable storage medium having readable program code embodied in the non-transitory tangible computer usable storage medium, the computer program product includes at least one component operable to: determine an attribute of a current character in input text, the attribute of the current character indicating one or more classes of characters the current character is assigned thereto; determine one or more attributes of one or more next characters in the input text, the one or more attributes of the one or more next characters indicating the one or more classes the one or more next characters are assigned thereto; and construct a token of the input text that comprises the current character and the one or more next characters, the attribute of the current character and the one or more attributes of the one or more next characters intersecting with each other, and the attribute of the current character and the one or more attributes of the one or more next characters comprising an attribute data structure which comprises a one-byte array, wherein the one-byte array comprises a plurality of binary bits and each bit of the binary bits indicates a different class from remaining bits of the binary bits. 14. The computer program product of claim 11 , wherein the at least one component is further operable to receive the input text from at least one of a local component of the computer program product, a user device, and a third party server.