How to train the machine so that it can give 'out of bound/classes' as an output for neural network
I think the best way would be to augment some data and have an additional output class "unknown". However, if that is not possible or the neural net can not be retrained I would compare the distribution of the outputs of a hidden layer. 

For the CNN architecture below, calculate the empirical distribution for the outputs of a hidden layer after the flatten layer for the training data (e.g.$n1$). Save this distribution and during deployment, calculate the output of $n1$ for the test instance. If the output of $n1$ corresponds to a highly unlikely value with respect to the empirical distribution of training examples, return "unknown". Else, predict the digit for this instance.



(image source)