Is it necessary to clear the replay memory regularly in a DQN when an agent plays against itself?
The purpose of the replay memory in DQN and similar architectures is to ensure that the gradients of the deep net are stable and doesn't diverge. Limiting what memory to keep and how far into the past it reaches is a problem that arises in in practical AI deployments. Throwing it away usually isn't the best decision because the agent's gradients may become unstable. Rather, keeping a window of the last T timesteps/gameplays can keep enough information while ensuring that the data is relevant as the agent improves. There are some additional methods which can improve the results as well:


Only keeping relevant timesteps (relevant depends highly on the application)
Removing timesteps from memory with probability relative to their age (the oldest has the highest probability to be removed)


It should be noted, that a larger replay memory is usually better than a smaller one to ensure stability but although memory is cheap it isn't infinite.