One of the important use-cases for uncertainty estimates is in sequential learning applications such as Bayesian optimization and active learning. While several studies on sequential learning applied to chemical datasets have emerged , it is often concluded that a greedy search strategy works best i.e. one that ignores the uncertainty estimates. One question to ask is then whether these observations are due to the search strategy itself or due to problems in the calibration of the uncertainty estimates. Consider using the uncertainty estimates presented in Figure for active learning. The highest uncertainties are greatly overestimated compared to the lower/medium uncertainties. These molecules would be over-sampled to a degree that is unwarranted when considering the actual expected error. The first step towards answering the question above is to have a proper way of evaluating ones uncertainty estimates. We believe the work presented herein will provide a starting point for chemical ML users for incorporating uncertainties in their work in an informed manner.