Ensemble simulations are necessary to ensure a reproducible method and capture the aleatoric uncertainties associated with the predicted free energies. In future work, they could be used to bring systematic errors arising from the epistemic uncertainties under control. Ensemble methods show that the free energy distributions obtained from independent simulations are non-Gaussian. Moreover, an important requirement for the reliability of NEQ calculations is the overlap between the work distributions in both directions which also requires ensembles. Hitherto completely overlooked, we quantify the overlap requirements and systematically investigate the potential role of various parameters in ensuring that NEQ protocols fulfil these requirements in order to yield reliable results.