Policy gradient: why does this converge with Adam and not SGD? I am looking into policy gradient methods. I stumbled into this implementation:
https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a
It uses a Nesterov Adam optimizer.

If I run it, it converges and gets good scores on OpenAI Gym's CartPole-v0.

However, if I change the optimizer from Adam to stochastic gradient descent (SGD), it never converges and seems to act randomly. Why is this? Is there something about policy gradient methods which make SGD a poor choice?

NOTE: there is a bug in that code which only runs the episode for 100 time steps. The episode can run for up to 200 time steps. I fixed this when running it.
Stochastic gradient descent (SGD) can get stuck in saddle points which results in the model not converging. A saddle points gradient is where zero in many directions but not all directions.
Adam adaptively changes the learning rate based on past gradients. This additional information can allow the model to escape saddle points, thus the model is more likely to converge.