Calibration curves and the respective AUCEs for the considered models are shown in Fig. , where a very good calibration is observed for systems with low noise and bias (N = 711613, h ≥ 100, noise≤0.02 and mean aggregation). In fact, the artificial dataset employed in this study is an ideal test case for calibration, because it features a controlled amount of noise, and can be approximated with an arbitrary level of accuracy with a sufficient amount of datapoints and model degrees of freedom. We find that ensembling of d-MPNNs yields a well-calibrated measure of uncertainty for a prediction in this case. However, when adding larger errors from noise or bias, worse values for the AUCE are observed, since the total error of each prediction is now dominated by other contributions, thus impacting the correlation between ensemble uncertainty and true error. Model bias is often ignored as an error source, but can significantly impact the ability of ensemble uncertainties to depict true errors. Datasets with a low amount of datapoints and large noise have been shown to lead to ill-calibrated models in literature (for example the Lipophilicity dataset in Ref. 47), but the contribution of data coverage and other sources of bias is usually overlooked relative to the contribution of noise or variance. Even our artificial, noiseless and easy-tolearn dataset leads to severely ill-calibrated models if the amount of training data is low (for example for N = 711 or 7116). The bottom panels of Fig. explain this failure: A low amount of training data leads to a bias-dominated model, where the total error is nearly exclusively due to bias, not variance. Because many chemical datasets are made of only hundreds to thousands of datapoints, we expect many deep-learning models to suffer from calibration error caused by data coverage model bias and should not be assessed using ensemble variance alone.