While good uncertainty estimates can hold extremely useful information about the degree of trust one should put in a ML prediction, they have yet to become a standard part of chemical ML studies. We believe one of the reasons for that is the lack of consensus on how to best benchmark the uncertainty estimates. One part of reaching that consensus is using an appropriate metric as described above. Another part is figuring out how best to design test sets that actually test the performance of the uncertainties on the intended objective.