Catastrophic forgetting in linear semi-gradient RL agent?
The same thing was happening to me with a deep Q Network on the cart-pole  problem. Having a "memory" with past (S,A,R,S) sequences and sampling it to form mini batches with the new observations helped a lot to reduce catastrophic forgetting. Reducing the step size once the agent has improved a certain amount also helped.