How to formulate discounted return in cartpole?
It's not true that finite horizon MDP can't use a discount factor. Any discount factor $\gamma \leq 1$ is fine for finite horizon.
Whether or not cartpole is a finite or infinite horizon depends on the environment implementation. The default environment in openai gym uses $T=200$ as the horizon. This is customizable, and most papers I see which use cartpole as a test environment use $T=1000$ instead. The task is considered &quot;solved&quot; when the agent can reach $T$ timesteps consistently (e.g., the last 100 episodes have all reached $T$ timesteps).
If you want to treat cartpole as an infinite horizon task, that's fine too. Clearly you need $\gamma &lt; 1$ here, and you can just $T$ to some arbitrary large number.