On a set of 598 ligands each binding to one of 22 targets, we showed that the public force fields OpenFF-1.0 (Parsley), OpenFF-2.0 (Sage), GAFF2.1x, and CGenFF/MATCH* are performing comparably based on aggregated statistics across the whole dataset, both in terms of RMSE of relative binding free energies ∆∆G (perturbations) and the RMSE and Kendall's tau of binding free energies ∆G. The proprietary force field OPLS3e performs significantly better, but a Consensus approach based on Sage, GAFF2.1x, and CGenFF/MATCH* is similarly accurate. There is a clear target dependence which can be attributed to input protein (binding pocket) flexibility, chemistries of ligands, and difficulty of perturbations (in terms of heavy atom changes). While Parsley and Sage are performing comparable based on aggregated statistics across the whole dataset, there are differences in terms of outliers. A parameter analysis revealed that improved parameters lead to significant improvement in the accuracy of affinity predictions on subsets of the dataset involving those parameters. Thus, we can show that there is a considerable improvement of successive OpenFF forcefield versions.