Table shows results for the Var(Z) ? = 1 and µ(Z) ? = 0 tests. CI(Var(Z)) and CI(µ(Z)) are the 95% confidence interval for the variance and average of Z, respectively. The lack of average calibration is clear for the models trained on 100 and 500 data points (Var(Z)>1 meaning underestimated uncertainties) in agreement with our findings based on the error-based calibration plots, miscalibration areas and NLL vs. NLL sim . µ(Z) is above 0 for the three models trained on the least amount of data indicating biased models, the remaining models are in agreement with unbiased models. The uncertainties from the 5k, 10k, 20k and 50k models all fail the V ar(Z) ? = 1 test for average calibration. This explains the higher-than-expected NLL (by around 5 standard deviations) for the 5k, 10k and 20k models (Table ). The discrepancy between NLL and NLL sim is less significant for the 50k model and it is also closer to passing the V ar(Z) ? = 1 test. In this case, the fact that there is a problem with average calibration for these four models was not obvious from the miscalibration area, A mis . Almost all models show a bias, but whether the bias is positive or negative varies between models. For biased errors, the Var(Z) ? = 1 changes to the ⟨Z 2 ⟩ ? = 1 test for average calibration . The change from Var(Z) to ⟨Z 2 ⟩ amounts to the addition of ⟨Z⟩ 2 (zero for unbiased errors). The most biased errors based on Z-scores are those of LS-NN 150k for which ⟨Z 2 ⟩ = 1.02 and the 95% confidence interval is [0.96;1.16] consistent with average calibrated errors. Likewise for the remaining models, we see that all models passing the Var(Z) ? = 1 test also pass the ⟨Z 2 ⟩ ? = 1 test. Though there is some correlation between the uncertainty estimates showing highest miscalibration area (A mis = 0.07) and the 4 sets of uncertainty estimates failing the ⟨Z 2 ⟩ ? = 1 test (LS-NN a , LS-GCN a , LS-GCN b and LS-GCN d ), there generally is a lack of agreement between the two metrics on what uncertainties have better average calibration. For example, LS-NN d just passes the ⟨Z 2 ⟩ ? = 1 test but still has a miscalibration area of 0.07 which is similar to the RF uncertainties trained on 100 samples significantly for this 265-sample test set. The miscalibration area is calculated based on an assumption of Gaussian distributed errors, while the Var(Z) ? = 1 test does not assume that . Thus, the miscalibration area test evaluates the agreement of the Z-distribution being Gaussian with variance = 1, while the Var(Z) tests if the Z-distribution has variance = 1, but make no assumption on the distribution. seed Var(Z) CI(Var(Z)) µ(Z) CI(µ(Z)) : UQ evaluation metrics for the latent space uncertainties of a GCNN model trained on 9k+1k data points of Crippen's logP. For NN uncertainties, the latent space used is the very last layer of the NN. For GCN uncertainties, the latent space is the vector right after the pooling layer (LS-GCN). The problems with the average calibration for the model with random seed 1 can also be inferred by observing the error-based calibration plot; all but one of the binned RMSE values are below that predicted based on the RMV (black dashed line) pointing to a tendency of overestimated uncertainties leading to a both lower than expected NLL and uncertainties that are not average calibrated.