Deep Reinforcement Learning: Rewards suddenly dip down
Without digging into your diagnostics more deeply, on it's face this seem like a local optima issue. Assuming you are optimizing via GD, there are many local optima that a network or agent in this case can converge to and stay at which will cause symptoms like seeing above.

With that being said, assuming this is our issue, here are some things you can try:


Regularization, try adding dropout or L2 and see how that affects convergence and learning.
Adjust network architecture, number of layers, nodes, etc. 
Try a different type of RL(Q learning for example), this will be dependent on your problem of course.
Adjust starting seed. Assuming you have a static seed used for weight initialization, you will always converge to the same solution. It could be as simple as adjusting the seed value.


If all these steps fail, you likely have a deeper issue at work, and I would suggest coming back after with some additional detail if this does not succeed.