Why is the update in-place faster than the out-of-place one in dynamic programming?
When you make updates in-place, then some of the entries in state_values[next_i, next_j] that you are referencing here
value += ACTION_PROB * (reward + discount * state_values[next_i, next_j])

will already be updated earlier in the same loop. Which means you get to use the latest and likely more accurate values earlier.
The strength of this effect varies depending on the order that states and actions are visited. If you can manage to visit them in reverse order that they would appear in natural trajectories, then the speed improvement will be very noticeable.