What loss function avoids overconfidence?
You could add a penalty by adding a small "distance" to every loss calculation. They you would have a "floor" below which no comparison could be below. The tricky decision is how large to make the penalty. There is a large literature on how to choose the "right" penalty that keep your overconfidence low but maximizes the predictive ability.

If you have a copy of "The Elements of Statistical Learning", then look at page 61 and the 10 + pages following that. There are also other items in the index covering penalized methods. The Wikipedia article on the Lasso methods is also a possible starting point.

When I type this into an R console on a new Linux box I'm outfitting:

 ??penalized


I get:

Help pages:

brglm::profileObjectives        Objectives to be profiled
mda::gen.ridge      Penalized Regression
mgcv::bam       Generalized additive models for very large datasets
mgcv::bam.update        Update a strictly additive bam model for new data.
mgcv::gam       Generalized additive models with integrated smoothness estimation
mgcv::jagam     Just Another Gibbs Additive Modeller: JAGS support for mgcv.
mgcv::null.space.dimension      The basis of the space of un-penalized functions for a TPRS
mgcv::pcls      Penalized Constrained Least Squares Fitting
mgcv::smooth.construct.bs.smooth.spec       Penalized B-splines in GAMs
mgcv::smooth.construct.cr.smooth.spec       Penalized Cubic regression splines in GAMs
mgcv::smooth.construct.tp.smooth.spec       Penalized thin plate regression splines in GAMs
quantreg::rq.fit.lasso      Lasso Penalized Quantile Regression
quantreg::rq.fit.scad       SCADPenalized Quantile Regression
rms::pentrace       Trace AIC and BIC vs. Penalty
VGAM::sm.ps     Defining Penalized Spline Smooths in VGAM Formulas
VGAM::summarypvgam      Summarizing Penalized Vector Generalized Additive Model Fits


I would have gotten more if I were using my older machine ( a macpro that died recently) but I havent' installed an lasso-type packages yet. There are quite a few:

&gt; rownames(available.packages()[  grep("lasso", available.packages()[,1]) , ])
 [1] "biglasso"      "cglasso"       "CVglasso"      "DIFlasso"      "dpglasso"      "elasso"       
 [7] "extlasso"      "gglasso"       "glamlasso"     "glasso"        "glassoFast"    "GPCMlasso"    
[13] "grplasso"      "hglasso"       "iilasso"       "ipflasso"      "lasso2"        "lassopv"      
[19] "lassoscore"    "lassoshooting" "lmmlasso"      "nnlasso"       "palasso"       "ppmlasso"     
[25] "prioritylasso" "PUlasso"       "sealasso"      "sglasso"       "Tlasso"