16. The method of claim 15 , wherein merging the previous token and the next token further comprises: determining whether an end offset of the next token is greater than or equal to an end offset of the previous token; reading a next token from the plurality of token streams in response to determining that the end offset of the next token is not greater than or equal to the end offset of the previous token; and merging the previous token and the next token in response to determining that the end offset of the next token is greater than or equal to the end offset of the previous token.
16. The method of claim 15 , wherein merging the previous token and the next token further comprises: determining whether an end offset of the next token is greater than or equal to an end offset of the previous token; reading a next token from the plurality of token streams in response to determining that the end offset of the next token is not greater than or equal to the end offset of the previous token; and merging the previous token and the next token in response to determining that the end offset of the next token is greater than or equal to the end offset of the previous token. 17. The method of claim 16 , wherein merging the previous token and the next token further comprises: determining whether the end offset of the next token equals the end offset of the previous token; and dropping the next token in response to determining that the end offset of the next token equals the end offset of the previous token.