What loss function avoids overconfidence? In the case of a neural net with a relatively small training data set, doing simple classification with categorical cross entropy (log loss), it is very easy for the results of the network to be "overconfident" because that improves log loss on the training data.  

What I mean by overconfident (this is partly intuition and partly something that can be derived from Bayesian stats): given 1000 training examples in 2 equal classes, it is unreasonable for any classifier to give a probability of a new input as being in one class or the other which is substantially higher than (1-1/1000), even if the classifier is perfect on the training data.  That is because if the training set was increased by just one more example, the 1001st example could break whatever rule seems to have been followed in the data set up to that point, and that would force all the predicted values/probability estimates to be updated accordingly.  

However, in practice trained neural nets with log loss function will output values which are arbitrarily close to 0 or 1 given a long enough training run, because that continues improving their performance on the training data way past the point that it makes sense to do so based on the size of the training data.

The question: how can the usual log loss function be modified so that making predictions extremely close to 0 or 1 does not hugely improve log loss compared to predictions which are only somewhat close?
You could add a penalty by adding a small "distance" to every loss calculation. They you would have a "floor" below which no comparison could be below. The tricky decision is how large to make the penalty. There is a large literature on how to choose the "right" penalty that keep your overconfidence low but maximizes the predictive ability.

If you have a copy of "The Elements of Statistical Learning", then look at page 61 and the 10 + pages following that. There are also other items in the index covering penalized methods. The Wikipedia article on the Lasso methods is also a possible starting point.

When I type this into an R console on a new Linux box I'm outfitting:

 ??penalized


I get:

Help pages:

brglm::profileObjectives        Objectives to be profiled
mda::gen.ridge      Penalized Regression
mgcv::bam       Generalized additive models for very large datasets
mgcv::bam.update        Update a strictly additive bam model for new data.
mgcv::gam       Generalized additive models with integrated smoothness estimation
mgcv::jagam     Just Another Gibbs Additive Modeller: JAGS support for mgcv.
mgcv::null.space.dimension      The basis of the space of un-penalized functions for a TPRS
mgcv::pcls      Penalized Constrained Least Squares Fitting
mgcv::smooth.construct.bs.smooth.spec       Penalized B-splines in GAMs
mgcv::smooth.construct.cr.smooth.spec       Penalized Cubic regression splines in GAMs
mgcv::smooth.construct.tp.smooth.spec       Penalized thin plate regression splines in GAMs
quantreg::rq.fit.lasso      Lasso Penalized Quantile Regression
quantreg::rq.fit.scad       SCADPenalized Quantile Regression
rms::pentrace       Trace AIC and BIC vs. Penalty
VGAM::sm.ps     Defining Penalized Spline Smooths in VGAM Formulas
VGAM::summarypvgam      Summarizing Penalized Vector Generalized Additive Model Fits


I would have gotten more if I were using my older machine ( a macpro that died recently) but I havent' installed an lasso-type packages yet. There are quite a few:

&gt; rownames(available.packages()[  grep("lasso", available.packages()[,1]) , ])
 [1] "biglasso"      "cglasso"       "CVglasso"      "DIFlasso"      "dpglasso"      "elasso"       
 [7] "extlasso"      "gglasso"       "glamlasso"     "glasso"        "glassoFast"    "GPCMlasso"    
[13] "grplasso"      "hglasso"       "iilasso"       "ipflasso"      "lasso2"        "lassopv"      
[19] "lassoscore"    "lassoshooting" "lmmlasso"      "nnlasso"       "palasso"       "ppmlasso"     
[25] "prioritylasso" "PUlasso"       "sealasso"      "sglasso"       "Tlasso"