We compared alchemical ABFE predictions obtained from equilibrium and non-equilibrium approaches. The headline finding is that the results obtained from the EQ approach are both more accurate and precise than those obtained from the NEQ approach. The MUEs for the full dataset are 0.93(0.05) and 1.15(0.07) kcal/mol for the EQ and the NEQ approaches, respectively. The corresponding ranges of RMSEs across protein systems for these methods are 0.39-1.83 kcal/mol and 0.60-2.23 kcal/mol and the number of complexes with error bars ≤ 0.5 kcal/mol are 94.98% and 46.12%, respectively. Moreover, the NEQ approach has more set up parameters to fine-tune which makes it more complicated to implement and automate as well as prone to risk of sub-optimal performance. For instance, around 54% complexes do not fulfil the first criterion of reliability whereas 49% complexes do not meet the second one when using our recommended standard protocol. Even on extending the transition length to 2 ns, these numbers only go down to 39% and 27% complexes, respectively, for our dataset. For the same reason, the NEQ approach exhibits a higher degree of system dependence which makes it largely impossible to apply a standard protocol. Due to these factors, the NEQ approach is plainly less preferable for blind and/or large scale applications.