What is the best approach for specified optical character recognition? I have a quite understandable request of extracting information (invoice number, invoice data, due date, total etc.) from scanned invoices (the digital format is image, not PDF), preferably in Python. The good thing is that the necessary information is more or less certain to exist on the page, and the (regexp-like) textual format of these is also tend to be consequent. The downside on the other hand is that the layout of the invoices are very diverse.

I have played with the following possible approaches:


Use character recognition to extract pure text and later try to
puzzle with the fragments. This method has some considerable
problems: the quality of the OCR (at least of the implemented one in
tesseract library) are quite mediocre and the output is hopelessly
unstructured (practically a big pile of words), it is very difficult
to come out any regexp or other rule even for regular phrases.
My other approach would be to apply some kind of deep learning either to
the raw image itself or the text pile where we leave the heavy
lifting to the network, but in this case I'm not sure what is
supposed to be the output? Is it some kind of a sequence to sequence
mapping?


Very unusual task, indeed.
Optical character recognition is a well-studied problem with many possible solutions (ressources). CNNs have proven to work extremely well even for hand-written character recognition. Take a look at this two papers:

Backpropagation Applied to Handwritten zip code
Comparaison of Classifier Methods: A Case Study in Handwritten Digit Recognition

Here is a beginner tutorial to do just that with Tensorflow.
If you need extra data to train your model, take a look at the MNIST dataset.