14. The system of claim 8 , wherein the set of tokenization rules defines how elements of the tokenized document are processed, the elements including punctuation, hyphenation, and whitespace.
8. A system, comprising: a memory having computer readable instructions; and a processor for executing the computer readable instructions, the computer readable instructions including: receiving a tokenized document and an offset annotation file at the processor, the tokenized document including a source document and corresponding tokens resulting from a low-level segmentation process; applying annotations from the offset annotation file, in conjunction with a set of tokenization rules, to the source document; determining an occurrence of a misalignment responsive to the applying, the misalignment indicating at least one of an annotation and associated position count in the annotation file does not coincide with a term at a corresponding position count in the source document, and the annotation file refers to a non-existent or inconsistent token in the source document; upon determining the misalignment is caused by an offset mismatch, calculating an offset number of characters between the position counts in the annotation file and the source document, and adjusting the position count in the annotation file to coincide with the position count in the source document; and upon determining the misalignment is not caused by an offset mismatch, resetting a current position count in the source document to at least one of a position count of a previous location in which a most recent alignment between the annotation file and the source document was ascertained, and a next identified calibration annotation in the annotation file.