Recognize Street View House Numbers
Have you seen this blog post? It is doing exactly what you are trying to do.


Convolutional neural network
Use a fixed sliding window to crop plate number
Train on a fixed-length number plate


If you want to train on the digit-level, you should define a fixed rectangle size, large enough to fit a single digit in your data set. If your convolutional neural network is good, it should be able to detect your digit.

If you haven't you should take a look at the Google paper: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42241.pdf