Agent Collapse / Overfitting during Training I'm new to reinforcement learning so please bear with me. I'm training an agent to play ms-Pacman using the actor-critic method.
Below are the results of a couple of runs, in both graphs the orange line in the average of the previous 100 values. The left graph is the episode duration and the right is the reward earnt in each episode.
In both the runs, the agent steadily learns and then collapses, and is unable to recover. Has the agent reached it's maximum potential with this method or is this a case of overfitting?
Hyperparameters for both the runs:

Batch Size: 128
Replay Memory Buffer: 500,000
Epsilon Minimum: 0.1
Epsilon Maximum: 1.0
Epsilon minimum is reached at around 2,000,000 steps (approx episode 3,000)
There are lots of things that might lead to this performance. With the very limited information provided I can only guess and suggest the reasons.
I have personally used lots of Pacman variations (including this one) with RL models successfully. My first initial guess is that you are stopping exploration too early. Pacman environments are very hard to solve (up to the last level). You can get a reasonable performance though within a couple of days training (depending on your system). I highly doubt that the agent underfits/overfits as it will take time for the agent to reach  a good score. Also underfitting/overfitting would be better observable when you test your agent (you do not update weights). I would let exploration to become minimum once the agent reaches the last episode. Then I would test the current state of the agent with very little or none amount of exploration (full exploitation).