19. One or more computer-readable storage media comprising a plurality of instructions that in response to being executed cause a computing device to: update an index data structure based on an input data stream, wherein the index data structure includes index data associated with offsets in the input data stream; process a plurality of chunks of the input data stream in parallel to generate a plurality of token streams using the index data, wherein each chunk has a first length and each chunk overlaps a previous chunk by a second length, and wherein each token stream is generated from a corresponding disjoint subset of the plurality of chunks; and merge the plurality of token streams to generate an output token stream.
19. One or more computer-readable storage media comprising a plurality of instructions that in response to being executed cause a computing device to: update an index data structure based on an input data stream, wherein the index data structure includes index data associated with offsets in the input data stream; process a plurality of chunks of the input data stream in parallel to generate a plurality of token streams using the index data, wherein each chunk has a first length and each chunk overlaps a previous chunk by a second length, and wherein each token stream is generated from a corresponding disjoint subset of the plurality of chunks; and merge the plurality of token streams to generate an output token stream. 20. The one or more computer-readable storage media of claim 19 , wherein each chunk overlaps a previous chunk by a second length.