 formula_27, and successively following policy formula_28.Hence, roughly speaking, the value function estimates "how good" it is to be in a given state.where the random variable formula_30 denotes the return, and is defined as the sum of future discounted rewards (gamma is less than 1, as a particular state becomes older, it's effect on the later states becomes less and less.