 Although state-values suffice to define optimality, it is useful to define action-values.Given a state formula_3, an action formula_5 and a policy formula_28, the action-value of the pair formula_51 under formula_28 is defined by

where formula_30 now stands for the random return associated with first taking action formula_5 in state formula_3 and following formula_28, thereafter.The theory of MDPs states that if formula_58 is an optimal policy, we act optimally (take the optimal action) by choosing the action from formula_59 with the highest value at each state, formula_3.