Monte Carlo for non-episodic tasks
Can someone explain what exactly breaks down for non-episodic tasks for Monte Carlo methods in Reinforcement Learning?

It is this first part of the sentence that you quote:

&quot;To ensure that well-defined returns are available...&quot;

In more detail, the return distribution in an episodic MDP can be defined as
$$G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}$$
where $R_x$ is reward distribution at time step $x$, $\gamma$ is discount factor, and $T$ is the last time step of the episode.
From this you can see that you will need to collect all of $r_{t+1}$ to $r_T$ for an episode in order to calculate a single concrete return value $g_t$. You can only collect $r_T$ at the end of an episode.
In non-episodic tasks, there is no terminal time step $T$, and the return definition becomes:
$$G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$
This cannot be resolved directly by collecting data for infinite time steps. Or in other words you can never sample an actual return value as it is defined. To get around this you could do one or more of the following:

You could use an artifical time horizon $h$, and define $T = t + h$. This will result in a biased sample. However, you can make the bias arbitrarily small relative to total return by making $h$ large. If you are using a discount factor, that may allow for smaller value of $h$.

You could use temporal difference (TD) learning to bootstrap return estimates without needing a full sample of return. This will result in a biased sample, but the bias due to bootstrapping should reduce as more samples are collected.

You could use TD with eligibilty traces and a high trace decay parameter $\lambda = 1$, which is a way of getting very similar learning behaviour to Monte Carlo, but still allowing learning to occur on each step in a continuing problem.


These options technically stop the approach being true Monte Carlo. The last two are described in the book under TD methods.