1. A method for partitioning a data stream into tokens, the method comprising steps of: using an input/output system receiving the data stream; using a processor device configured to perform: setting a partition scanner to a beginning point in the data stream; identifying likely token boundaries in the data stream using the partition scanner; partitioning the data stream according to the likely token boundaries as determined by the partition scanner, wherein each partition of the partitioned data stream bounded by the likely token boundaries comprises a chunk; passing the chunk to a next available token scanner, one chunk per token scanner, for: identifying at least one actual token within each chunk; and reporting the actual token to a client for downstream processing only when the token scanner is determined to be non-speculative.