Why does ε-greedy $Q$-learning not oscillate?
The assumptions for Q-learning's convergence proof require that:


  [...] all state-action pairs be visited infinitely often


So the convergence is understood in a mathematical sense - the Q-function converges to its fixed-point as the number of steps approaches infinity.

In practice, even after your Q-function will be good enough to induce an optimal policy, you will observe "unnecessary" updates each time a non-greedy action is chosen. To avoid that, you can set the learning rate to 0 after the acceptable quality of the solution is achieved.

Also, your intuition about modifying the ϵ parameter, in case of ϵ-greedy policy, seems right. Setting ϵ to 0 will have practically the same effect as setting the learning rate to 0 in the situation described above - in case of a non-stochastic environment.