How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?
If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an end of episode, which assumes a return of 0 from any terminal state $S_{T}$. Of course that is because it is not the end of the episode.
You have two &quot;natural&quot; options to tweak DQN to match to theory at the end of a pseudo-episode:

Store the state, action, reward, next_state tuple as normal and use the standard one step TD target $G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$

Completely ignore the last step and don't store it in memory. There is no benefit to this as opposed to the above option, but it might be simpler to implement if you are using a pre-built RL library.


Both these involve ignoring any done flag returned by the environment for the purposes of calculating TD targets. You still can use that flag to trigger the end of a loop and a reset to new starting state.
You should also take this approach if you terminate an episodic problem early after hitting a time step limit, in order to reset for training purposes.

As an aside (and mentioned in comment by Swakshar Deb), you can also look into the average reward setting for non-episodic environments. This solves the problem of needing to pick a value for $\gamma$. If you have no reason to pick a specific $\gamma$ in a continuing problem, then it is common to pick a value close to 1 such as 0.99 or 0.999 in DQN - this is basically an approximation to average reward.