Our results demonstrate that small deviations in the reference configurations selected for the umbrella windows along the exact same reaction coordinate can significantly affect the computed free energies, leading to 2-20 kcal/mol differences in the obtained PMF profiles. Such discrepancies are in line with previous reports on hysteresis, and a dependence of the computed free energy on initial conditions. Because the lack of reproducibility can be attributed to small differences between the paths, we suggest that sampling multiple paths simultaneously is needed to obtain reliable free energy estimates.