brackets on the right, the above infinite-horizon decision problem is equivalent to: subject to the constraints Here we are choosing , knowing that our choice will cause the time 1 state to be . That new state will then affect the decision problem from time 1 on. The whole future decision problem appears inside the square brackets on the right. The Bellman equation So far it seems we have only made the problem uglier by separating today's decision from future decisions. But we can simplify by noticing that what is inside the square brackets on the right is the value of the time 1 decision problem, starting from state . Therefore, we can rewrite the problem as a recursive definition of the value function: , subject to the constraints: This is the Bellman equation. It can be simplified even further if we drop time subscripts and plug in the value of the next state: The Bellman equation is classified as a functional equation, because solving it means finding the unknown function , which is the value function. Recall that the value function describes the best possible value of the objective, as a function of the state . By calculating the value function, we will also find the function that describes the optimal action as a function of the state; this is called the policy function. In a stochastic problem In the deterministic setting, other techniques besides dynamic programming can be used to tackle the above optimal control problem. However, the Bellman Equation is often the most convenient method of solving stochastic optimal control problems. For a specific example from economics, consider an infinitely-lived consumer with initial wealth endowment at period . They have an instantaneous utility function where denotes consumption and discounts the next period utility at a rate of . Assume that what is not consumed in period carries over to the next period with interest rate . Then the consumer's utility maximization problem is to choose a consumption plan that solves subject to and The first constraint is the capital accumulation/law of motion specified by the problem, while the second constraint is a transversality condition that the consumer does not carry debt at the end of their life. The Bellman equation is Alternatively, one can treat the sequence problem directly using, for example, the Hamiltonian equations. Now, if the interest rate varies from period to period, the consumer