Confidence curves are another popular choice of metric belonging to the ranking-based methods. The confidence curve shows the change in test set error as data points are excluded based on the predicted uncertainty (highest uncertainty points excluded first). One would then expect a decreasing curve. The observed confidence curve is often compared with what one would get for an "oracle", which represents the quite unrealistic scenario that the ranking of the errors and uncertainties are perfectly correlated (corresponding to ρ rank = 1), meaning that the uncertainty predictor is actually an error predictor. Here we will focus on ρ rank to represent the ranking-based metrics but refer the reader to work by Pernot on the use of confidence curves for UQ validation, which was published while preparing this manuscript . Similarly to how we propose the simulated ρ rank as a reference to Spearman's rank correlation coefficient, Pernot suggests changing the reference confidence curve from an "oracle" to a probabilistic one based on errors sampled from the predicted uncertainties assuming normally distributed errors (just like we do for ρ sim rank ).