Understanding the configuration of replay memory and epsilon in deep reinforcement learning
Replay memory is a recording of the games that the agents has played. It is this data that is used to train the neural network (or whatever machine learning method you are using).
The training is performed by looking at the games played and the resulting reward signal (in Pacman, this might be your score), and then learning which game strategies worked well and which did not.
It is difficult to say exactly how big the replay memory should be, this is something that you will need to experiment with. The larger it is, then the more examples will be available to be learned from during a particular training period, but of course it takes longer to acquire them.

Epsilon is a parameter that governs the trade-off between exploration and exploitation. By this, I mean that for each action that your agent takes, it needs to decide whether to do the best thing that it has already discovered to do (exploitation), or try something new (exploration). Usually, a value of epsilon near to 1 means that the agent will choose to explore more, and a value closer to 0 will mean that the agent will exploit more.

You can imagine therefore that at the beginning of an agent's training you might want epsilon ~1, because it has lots of new strategies to try to discover, but as it gets better you would reduce it to ~0 so that it can explore strategies that are only reachable after a long sequence of game steps.
Naturally, when the agent is fully trained and ready to be deployed then epsilon would = 0.
In the example that you posted, you can see that Pacman starts off doing nothing but exploring, but after 500000 games it exploits 95% of the time.