How to implement RL policies learned on a finite horizon?
A normal way to deal with training on infinite horizon (aka &quot;continuing&quot; or &quot;non-episodic&quot;) problems is to use TD learning or other bootstrapping methods (of which Q-learning in DQN is one example), and to treat the cutoff at $T$ for pseudo-episodes as a training artefact.
If the state at time $T$ was really a terminal state, the TD target would be just $r$ because $q(s^T,\cdot) = 0$ by definition, but that doesn't apply in your case.
So always use the bootstrapped TD target - e.g. $r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$ for single step TD target with a Q estimate having $\theta$ as learned params - and don't treat the horizon data any differently.
If you do this, then your concerns about under-estimates should not be an issue. Your pseudo-episodes do need to be long enough to observe the impact of multiple requests in the long term is the main issue (setting $T$ too low so that the system does not reach any kind of equilibrium would be a problem).
You could also use an average reward setting and differential value functions. It is slightly better from a theoretical standpoint, but Q-learning and DQN is fine if you don't want to be bothered with that. The same basic rule applies - ignore &quot;episode end&quot; for constructing TD targets because it is just a training artefact. Also ensure you set $T$ high enough that the long term impacts of a policy are observable.
If your starting state is special (e.g. cars all in fixed places, and no requests in progress) this could also be an issue, because the real world system will rarely be in that state but you will have many episode starts with it. If you cannot start the system in a reasonable random initial state for your problem, you may also want to discard data from episode starts and allow a run-in time before using data for training from the pseudo-episodes.