What is a good convergence criterion for Q-learning in a stochastic environment?
To obtain guarantees of convergence for Q table values, you need to decay the learning rate, $\alpha$, at a suitable rate. Too fast and convergence will be to inaccurate values. Too slow and convergence never happens.
For sticking with theoretical guarantees, the learning rate decay process should generally follow the rule that $\sum_t \alpha_t = \infty$ but $\sum_t \alpha_t^2 \ne \infty$ - an example of a learning rate schedule that does this is $\alpha_t = \frac{1}{t}$, although in practice that specific choice could lead to very slow convergence.
Choosing a good starting $\alpha$ and a good decay schedule will depend on the problem, and you may want to base it on experience with similar problems. However, it is not that common to need to gurantee convergence of action values in value-based reinforcement learning. In control problems you often care more about have finding an optimal policy than about perfectly accurate action values. Further to that, many interesting control problems are too complex to solve perfectly in tabular form, so you expect some approximation. It seems relatively common just to pick a learning rate for the problem and stick with it.
If you make your learning rate lower, the Q table will converge to more stable Q values, but possibly at the expense of taking longer to converge on the optimal policy.