Another way of looking at this algorithm is thinking that we're doing value iteration, but on most rounds we just go with the last action that optimized for the state, rather than considering them all. 