Why not always use the ADAM optimization technique?
Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM.
There is often a value to using more than one method (an ensemble), because every method has a weakness.