Successful methods for optical character recognition?
This is not meant to be a direct answer, as OCR is not an area I have experience in, but I want to present a deeper problem, that of mathematical notation.

Mathematics has "exploded" in the last few centuries, and notation is not standard across various sub-fields.  Check out this question on notational change with integrals. In some cases, individual sub-fields have alternate forms of notation.

My suspicion is that this task will be enormously difficult without a great deal of human guidance, so I'd point you towards AI methods that utilize "human-assisted machine learning".