The comparisons between different MD engines and free energy estimators, made in this work, could only be made meaningfully when the uncertainty in the binding free energy is accounted for correctly. The results from one-off simulations are not reproducible and so only with the proper application of ensemble simulation could such good agreement between the MD engines and free energy estimations compared in this work be found. The application of multiple independent simulations was critical for our error control; similar ideas are found elsewhere in the literature . If only one-off simulations are performed errors are consistently underestimated in these calculations. Figure shows this explicitly by comparing the SEM computed by MBAR averaged over 5 replicas to the "TIES-like" error calculated by computing the SEM of the bootstrapping results from 5 replicas. It can be seen that, for the ligand simulations in figure , some systems (TYK2, CDK2 and thrombin) have errors correctly estimated by MBAR but for, PTP1B and MCl1, MBAR consistently underestimates the error. This underestimation is only exacerbated in complex simulation, Figure , where all systems have their error underestimated by MBAR. This is most likely due to the greater relevance of "rare events" in the complex simulation. Similar findings by Rizzi et al. have concluded "Nevertheless, when sampling is governed by rare events and systematically misses relevant areas of conformational space, data from a single trajectory simply cannot contain sufficient information to estimate the uncertainty accurately" 39 . It has often been argued that the time series of potentials fed to MBAR should be decorrelated to ensure reliable error estimation . Decorrelation of the time series of potentials, in this case, does not change any of the conclusions. For completeness, we provide an equivalent version of figure 10 using decorrelated data in the SI (see figure ) which demonstrates this conclusively.