Does convergence equal learning in Deep Q-learning?
Policies found by Deep Q-Learning, even after convergence, are not guaranteed to be optimal. The reason is that the neural networks that approximate the Q function in DQN inherently come with a statistical error (bias and variance), a pointer can be found here.
Furthermore, convergence to the optimal policy for tabular Q-learning is only guaranteed when every action is sampled infinitely often in every state. This may make using any 'converged and optimal policy' in experiments very hard, even when disregarding additional complexities from function approximation in DQN.
Please be aware, additionally, that generalization in supervised learning requires that the training and test data are sampled from the same distribution. Similarly for RL, the training and test environments are assumed to be sampled from the same 'distribution'. Generalization to environments not reflected in the training set is referred to as transfer learning in the RL literature.
This is an important and very interesting line of research though, so please be not be discouraged by this comment.