Why is the target called "target" in Monte Carlo and TD learning if it is not the true target? I was going through Sutton's book and, using sample-based learning for estimating the expectations, we have this formula
$$
\text{new estimate} = \text{old estimate} + \alpha(\text{target} - \text{old estimate})
$$
What I don't quite understand is why it's called the target, because since it's the sample, it’s not the actual target value, so why are we moving towards a wrong value?
It is our &quot;current&quot; target. We assume that the value we get now is at least a closer approximation to the &quot;true&quot; target.
We're not so much moving towards a wrong value as we are moving away from a more wrong value.
Of course, it is all base on random trials, so saying anything definite (such as: &quot;we are guaranteed to improve at each step&quot;) is hard to show without working probabilistically. The expectation of the error of the value function (as compared to the true value function) will decrease, that is all we can say.