Sutton & Barto's notation $V_{t+n}$ in Chapter 7: $n$-step Bootstrapping
So, what is the purpose of the new index for $V$ in Chapter 7, and why is it more important at this particular chapter?


My guess would be that your intuition is correct, and that it's mostly introduced just to clarify exactly which "version" of our value function approximator is going to be used in any particular equation. In previous chapters, which discuss single-step update rules, I guess the authors assumed there was less potential for confusion, and therefore no need to clarify this. Without the clarification, some people might for instance wonder if we should use $V_t$ for our value estimates of an $n$-step return $G_{t+n}$, regardless of how large $n$ is.




  However, I think this introduced a technical error, because suppose we just learned from an 10-step episode ($T=10$), using $n = 3$. Then the latest estimate of $v_\pi$ is $V_{T-1} = V_{10 - 1} = V_{9}$. Then at the next episode, the first time $V_{t + n}$ is used to inform a target update, it will be for $\tau = 0$ (from the pseudo-code), which implies $t - n + 1 = 0$, so $t = n - 1$, that is, $V_{t+n}=V_{n-1+n}=V_{2n-1}=V_5$, which is not the most up-to-date estimate $V_9$ of $v_\pi$.


Once we start considering a situation with more than a single episode, the $V_t$ notation becomes quite confusing. You should read $V_t$ as "the value function approximator that we have available at time $t$ of the current episode". So, if we were to use the symbol $V_0$ within the context of a second episode, that would be identical to what was referred to as $V_T$ in the context of the first episode. The $V_t$ notation can be convenient if we're thinking about our equations with our minds in "math-mode", but becomes highly confusing once we start thinking about practical implementations involving multiple episodes -- this is probably why they did not include it in the pseudocode.

If you really wanted to use the subscript-notation in the pseudocode, you'd have to add an extra term in the subscript that adds up all the durations of all previous episodes. If we then try to work out your example situation, we'd run into another problem though... we'd want to use $V_{t+n+T} = V_{2n-1+T} = V_{15}$ at the first iteration where $\tau = 0$ in the second episode. But, across the two episodes, only $13$ steps have passed, so this does not yet exist! You run into the same issue if you try to work out what happened when $\tau = 0$ in the first episode: applying exactly the same reasoning as in your quote, we would have wanted to use $V_5$ after only $3$ time steps passed in the first episode.

The problem here is that you're trying to use the variable named $t$ in the pseudocode as the subscript for $V$. To get a better idea of what's going on here, let's loop back to the previous page and examine the definition of the $n$-step return:

$$G_{t:t+n} \doteq R_{t+1} + \gamma R_{t+2} + \dots + \gamma^{n-1} R_{t+n} + \gamma^n V_{t+n-1} (S_{t+n}).$$

Ok, we've got that. Now, let's take another look at the update rule in which we use this quantity:

$$V_{t+n}(S_t) \doteq V_{t+n-1}(S_t) + \alpha \left[ G_{t:t+n} - V_{t+n-1} (S_t) \right].$$

Ok. So $V_{t+n-1}$ appears three times in the update rule. Two times explicitly, estimating the value of $S_t$, and once more "hidden" in the definition of $G_{t:t+n}$, where it is used to estimate the value of $S_{t+n}$. Note very carefully what it is that this update rule is doing; it's updating the state estimate of $S_t$. If you now look at the pseudocode again, you'll see a comment on the line where $\tau$ is computed: ($\tau$ is the time whose state's estimate is being updated)!

What this means, is that in the pseudocode, you should be using $\tau$ as the subscript for $V$! If you do that, it'll at least be correct for the first episode. In the pseudocode, the update rule looks like:

$$V(S_{\tau}) \gets V(S_{\tau}) + \dots$$

Plugging in the subscripts from the mathematical definition leads to:

$$V_{\tau + n}(S_{\tau}) \gets V_{\tau + n - 1}(S_{\tau}) + \dots$$

Since the pseudocode defines $\tau = t - n + 1$, we can substitute above:

$$
\begin{aligned}
V_{t - n + 1 + n}(S_{t - n + 1}) &amp;\gets V_{t - n + 1 + n - 1}(S_{t - n + 1}) + \dots\\
V_{t + 1}(S_{t - n + 1}) &amp;\gets V_{t}(S_{t - n + 1}) + \dots
\end{aligned}$$ 

and now it should make sense again from a practical point of view. At every time step $t$, where $t$ measures number of steps of experience that we have simulated, we simply use the latest value function $V_t$ we have available at that time for bootstrapping in the update rule. When $t + 1 &lt; n$, $S_{t - n + 1}$ is undefined. In these cases, the above update rule doesn't work, which makes sense intuitively because we have not yet progressed far enough into the episode to be capable of compute $n$-step returns.