Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?
Without the code there's not much we can do but, I'd guess you need to significantly lower the learning rate. From my experience Adam requires a significantly lower learning rate compared to SGD.