Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?
You gave an exemple of a value function, but not an optimal value function, that would verify :
$$ V^{*}(s) =max_\pi(V^{\pi}(s)) $$
Such a value function would allow to use one-step look ahead approach to get the optimal policy. Because the optimal value function would be defined somehow iteratively to take neighbor value into account.
In other words, the problem with your value function is that it is not optimal, notably  because of the significant step from 0 to 20. If it was optimal you would have a value nearer to 20 near it and the look-ahead would follow this path.