Why do I get the best policy before Q values converge using DQN?
Even if the mean of the maximum Q-value increases from episode 300 onwards, it doesn't mean that the relative order of the Q-values of the actions that you can take in the states change, which means that the policy may not change, even though the value function changes, assuming you're acting greedily with respect to the value function.
More concretely, suppose that you can take one of two actions $\{ a_1, a_2\} = \mathcal{A}$ in each state $s \in \mathcal{S}$. Let's say that you pick $s_1, s_2$ to calculate the average of the maximum Q-value. Without loss of generality, suppose that the action associated with the highest Q-value in these states is $a_2$. If your policy is the greedy policy with respect to the state-action value function, then your policy will choose $a_2$ in $s_1$ and $s_2$. If all Q-values $\hat{q}(s_1, a_1)$, $\hat{q}(s_1, a_2)$, $\hat{q}(s_2, a_1)$ and $\hat{q}(s_2, a_2)$ increase (or even decrease) but their relative order remains the same (i.e. $\hat{q}(s_1, a_2) &gt; \hat{q}(s_1, a_1)$ and $\hat{q}(s_2, a_2) &gt; \hat{q}(s_2, a_1)$, from episode 300 onwards), the greedy policy also remains the same.
So, I think that what you observed is theoretically possible, although I cannot guarantee that you don't have another issue.