def value_iteration(mdp, epsilon=<NUM_LIT>):
U1 = dict([(s, <NUM_LIT:0>) for s in mdp.states])<EOL>R, T, gamma = mdp.R, mdp.T, mdp.gamma<EOL>while True:<EOL><INDENT>U = U1.copy()<EOL>delta = <NUM_LIT:0><EOL>for s in mdp.states:<EOL><INDENT>U1[s] = R(s) + gamma * max([sum([p * U[s1] for (p, s1) in T(s, a)])<EOL>for a in mdp.actions(s)])<EOL>delta = max(delta, abs(U1[s] - U[s]))<EOL><DEDENT>if delta < epsilon * (<NUM_LIT:1> - gamma) / gamma:<EOL><INDENT>return U<EOL><DEDENT><DEDENT>
Solving an MDP by value iteration. [Fig. 17.4]