Q table not converging for an arbitrary experiment
If your intention is to learn make the agent learn which has the min arbitrary value, then you would need to modify your rewards a bit. 

The current reward structure provides the incentive to just move to a stage where it gets a reward. 

For example, if it is at state 0, it gets the same reward to go to either state 2 or state 3, since both of them have a higher inverse value.

To make the agent learn to move to state 2, you would have to provide it with more incentives to go to state 2.

def reward(s,a,s_dash):
    if s_dash == 2:
        return 5
    elif inverse_values_for_states[s]&lt;inverse_values_for_states[s_dash]: 
        return 1
    elif inverse_values_for_states[s]&gt;inverse_values_for_states[s_dash]:
        return -1
    else:
        return 0


I tried using this and it converges to 2. This is a hard-coded version, but I guess you get the idea.