In this work, we produce ensembles of submodels by starting each training run from differently initialized model parameters. Many other techniques for generating randomly differentiated submodels exist, such as bootstrapping, Monte-Carlo-dropout or saving snapshots from different training epochs. Our reported model prediction is the average of the predictions of the submodels in the ensemble (Eq. ( )). Fig. depicts the observed mean absolute error of the test set of several d-MPNN models trained on the artificial enthalpy dataset as a function of the number of models in the ensemble. The different panels in Fig. refer to different situations in which additional model errors have been introduced with data coverage (top left), model size (top right), noise (bottom left) and architecture (bottom right). Regardless of the sources of additional error in the model, ensembling always improves model performance on average. The performance with increasingly large ensembles will approach an asymptote. This is because ensembling purely addresses variance error. A large ensemble can remove variance error, as the ensemble prediction y(X n ) will converge to the variance distribution mean µ j (X n ) (Eq. ( ), ( )). However, bias and noise errors remain even when an ensemble size is made very large. The lower the error from other sources, the larger the performance gains available from ensembling. This effect could already be anticipated from Fig. , where a larger contribution of variance error was observed for lower data coverage error. We further note that the performance gains for adding an additional submodel to an ensemble are diminishing, while the computational cost of training and saving models scales roughly linearly with the number of submodels. Adding a small number of additional submodels to improve the model performance may be justified against the costs while adding a large number of additional submodels may not be.