In this paper, we test the non-equilibrium approach for absolute binding free energy calculations and compare it with equilibrium free energy calculations. We highlight challenges encountered applying the two methods and explore similarities and differences in handling these issues. Since the challenges we identify are likely to show up in other protein-ligand systems as well, we propose some strategies to assess sampling problems in alchemical free energy calculations (FEC). Although we find ways to work around some of the problems, our results highlight the large amount of work still to be done to make these calculations robust and to reliably detect problems in an automated way. Most literature studies report only successes, and calculations generally seem to work, though industry reports have highlighted the need for more careful analysis of failures. Here, we report on a relatively large number of "failures" with common protocols and highlight how they fail and why. We provide insights and directions for future method development and research in this area.