20. The computer program product of claim 15 , wherein the program instructions executable by the processor further cause the computer processor to perform: storing one or more of the sets of the tokenization rules learned from previously processed annotation files.
15. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the computer processor to perform: receiving a tokenized document and an offset annotation file, the tokenized document including a source document and corresponding tokens resulting from a low-level segmentation process; applying annotations from the offset annotation file, in conjunction with a set of tokenization rules, to the source document; determining an occurrence of a misalignment responsive to the applying, the misalignment indicating at least one of an annotation and associated position count in the annotation file does not coincide with a term at a corresponding position count in the source document, and the annotation file refers to a non-existent or inconsistent token in the source document; upon determining the misalignment is caused by an offset mismatch, calculating an offset number of characters between the position counts in the annotation file and the source document, and adjusting the position count in the annotation file to coincide with the position count in the source document; and upon determining the misalignment is not caused by an offset mismatch, resetting a current position count in the source document to at least one of a position count of a previous location in which a most recent alignment between the annotation file and the source document was ascertained, and a next identified calibration annotation in the annotation file.