How to avoid overfitting in Reinforcement Learning
In RL this is known as the exploration-exploitation trade-off, so i don't think you can avoid this regularizing the neural network, but rather the action policy it learns. The agent you are training is learning to solve a specific task and its policy is "over-fitting". You could tackle this in several ways, one of which is to encourage exploration by using a distribution over the Q Values such as Softmax

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

or by applying a $\epsilon$-greedy action selection, where you choose a random action with probability $\epsilon$ and otherwise the max Q Value. There are a lot of other ways to achieve this, just to quick "exploration strategies for dqn" search on google scholar.