 To define optimality in a formal manner, define the value of a policy formula_28 by

where formula_30 stands for the return associated with following formula_28 from the initial state formula_3.Defining formula_40 as the maximum possible value of formula_41, where formula_28 is allowed to change,

A policy that achieves these optimal values in each state is called "optimal".Clearly, a policy that is optimal in this strong sense is also optimal in the sense that it maximizes the expected return formula_44, since formula_45, where formula_46 is a state randomly sampled from the distribution formula_47.