How to determine convergence when using Q-learning? I'm using Q-Learning to find the values of states in on a gameboard. For example, something like:

-----------------
|   |   |   | 1 |
-----------------
|   | W |   | -1|
-----------------
|   | W |   |   |
-----------------
| S |   |   | 1 |
-----------------


I have the transition, and I can calculate the values of the states, but I can't tell when to stop computing. Normally I would just compute until a terminal state, for x-episodes; but this particular board has no terminal states. So I'm stuck to just calculating the values over say 1000 iterations (the more the more accurate obviously). I would like a way to stop though when I'm within an acceptable range.

How is this normally done for a non-terminal state space?
There are two ways that I know of:
First, based on how many iterations, like you would do.
Second, based on the difference between current Q value and previous Q value (i.e., insignificant difference means termination).