 In this step, given a stationary, deterministic policy formula_28, the goal is to compute the function values formula_67 (or a good approximation to them) for all state-action pairs formula_51.Assuming (for simplicity) that the MDP is finite, that sufficient memory is available to accommodate the action-values and that the problem is episodic and after each episode a new one starts from some random initial state.Then, the estimate of the value of a given state-action pair formula_51 can be computed by averaging the sampled returns that originated from formula_51 over time.