Would Deep Q Learning work for a finite horizon problem?
DQN solves optimal control problems for maximising average reward. Although it typically uses discounted reward, the discount factor is not part of the setting - instead it is part of the solution hyperparameters, and usually set quite high e.g. 0.99 - when using function approximators.

The TD target used in DQN is a problem for you:

$$G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}Q(S_{t+1},a')$$

as it relies on a Bellman equation that no longer holds for the given value functions. In your case, there does not seem to be any way to express the TD target for time $t$ by referencing other Q values, as you would need to then subtract future rewards, which is very ungainly. Instead, you could use a simple truncated Monte Carlo return

$$G_{t:t+3} = R_{t+1} + R_{t+2} + R_{t+3}$$

There are a few different ways you could do this, but the simplest and closest to DQN would IMO be to:


Store trajectories in order in the experience replay table
For each item in the training mini-batch:


Pick a start state/action pair to assess $s_t, a_t$ randomly from replay table
Check that $a_{t+1}$ and $a_{t+2}$ are maximising actions in your current target policy for $s_{t+1}$ and $s_{t+2}$, reject the sample if not (note you don't need to check or reject $a_t$, and that is how your code learns about exploratory actions)
Calculate the TD target, $g_t = r_{t+1} + r_{t+2} + r_{t+3}$
Your training data for that example is $s_t, a_t, g_t$



The checking for maximising action parts could be quite slow, so you might prefer to simplify the approach and not use off-policy. Alternatively, if $epsilon$ is low enough you could just store the three step returns directly in the experience replay table (wait until you have the data from $t+3$ before storing data for $t$) and ignore the fact that some returns are from exploratory actions, thus noisy/biased . . . this approach is used in n-step returns in DQN "Rainbow" version and works well enough in practice on the Atari problems despite being on shaky theoretical ground.



Note I am using the convention $s_t, a_t, r_{t+1}, s_{t+1}$ to represent a step in the trajectory, whilst in the question you appear to be using $s_t, a_t, r_t, s_{t+1}$ with a different reward index. You will need to convert back if you want to stick with your convention.