How to avoid overfitting in Reinforcement Learning I have implemented a RL model based on Deep Q-Learning for learning how to play a 2D game, like the ones in the OpenAI Gym. For testing the model, unlike most people, I have chosen to evaluate its performance on different levels from the ones used for training. This way, I can assess if the knowledge learnt by the model generalizes well to previously unseen levels.

After doing some testing and plotting the results, I have noticed an issue. Although the model generalizes well, it ends up overfitting the training set (levels used for training) if the training process continues for too long. This makes sense since early stopping is a common technique used to prevent overfitting. The problem is that the longer the training lasts, the more samples the agent is trained on, which should improve its test performance. This doesn't happen, however, due to overfitting, as I have explained.

I would like to run the model and plot the performance (reward obtained) obtained on the test set as the number of samples in the experience replay increases. The plot I want to obtain should show how the performance increases as the number of samples grows, but overfitting causes that, after a point, the performance decreases, even if more samples are used for training. I want to avoid this. These are the solutions I can think of:


Use techniques to prevent overfitting. Right now I'm using Dropout but it still occurs.
Use an exponential decaying learning rate. Right now I'm using a constant one, since I don't want the newer samples to impact the training less than the older ones.
Give up and change the number of training steps depending on how many samples I want to use for training. Then, to obtain the plots, I will have to repeat the training once per each point (number of samples) of the plot, in order to obtain the best performance for each different size of the experience replay. Of course, this is my last resort.


What do you think I should do?

Edit: I have just tried L2 regularization. It works more or less the same as Dropout.
In RL this is known as the exploration-exploitation trade-off, so i don't think you can avoid this regularizing the neural network, but rather the action policy it learns. The agent you are training is learning to solve a specific task and its policy is "over-fitting". You could tackle this in several ways, one of which is to encourage exploration by using a distribution over the Q Values such as Softmax

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

or by applying a $\epsilon$-greedy action selection, where you choose a random action with probability $\epsilon$ and otherwise the max Q Value. There are a lot of other ways to achieve this, just to quick "exploration strategies for dqn" search on google scholar.