Time horizon T in policy gradients (actor-critic)
In this case, I think it doesn't matter when you reach $s_{spec}$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $t\in \{0, ..., T\}$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).

In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.

The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.

I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.