Q-learning problem wrong policy
You have a simple mistake in your TD Error function:

TD_error = reward + gamma * max(Q[next_state]) - Q[state][action]


You have made Q[next_state] a Python dict, so this will take the maximum key which is 3 for all your Q table entries. This is why you end up with values very close to 3 at the end, which is impossible for your problem, the maximum return will be 1.0 when stepping from adjacent grid points to the positive terminal state.

The correct code is:

TD_error = reward + gamma * max(Q[next_state].values()) - Q[state][action]


Alternatively, since your actions are all numeric, you could use a list instead of a dict in your Q table.