Classifying with certainty
Softmax output in neural networks can be misleading - often the confidence provided is higher than is intuitive.  See e.g. here:

A Baseline for Detecting Misclassified and
Out-of-Distribution Examples in Neural Networks
https://pdfs.semanticscholar.org/c26e/1beaeaa55acae7336882de5df48716afb8bb.pdf

which suggests that in practice, softmax is not helpfully interpretable as a probability but should instead be used for ranking among class options.

If you want to have an accurate probability estimate, you might consider using a Bayesian approach in which you explicitly model your estimate of each of the input variances, the output variance, etc.

Failing that, having a second phase neural network that takes the input and predict correct or incorrect classification by the first network is an interesting idea - where incorrect classification is a proxy for 'low confidence' classification.  If you try it I'd be curious to know how it works.

Edit: As @Emre said the input to the softmax would be more informative than the softmax itself because it's pre-scaled (i.e. not forced to sum to 1). So it should reflect confidence better, with values further away from 0 indicating higher confidence.