How to deal with episode termination in Advantage Actor-Critic algorithm?
First a small note: I don't think your expression for $A(s_{T-1}, a_{T - 1})$ looks correct. If we assume that $V(s_T) = 0$ (i.e., assume that we cannot possibly reach the goal in one single step from $s_{T - 1}$), we have:

\begin{align}
A(s_{T-1}, a_{T-1}) &amp;= r(s_{T-1}, a_{T-1}) + \gamma V(s_{T}) (1 - done_{T})  - V(s_{T-1}) \\
&amp;= r(s_{T-1}, a_{T-1}) - V(s_{T-1}).
\end{align}

In this expression, we'd normally have that $r(s_{T-1}, a_{T-1}) = -0.1$, whereas you seem to have mistakenly taken $+0.1$ in your post.



Those details aside; yes, there can be sudden bumps of positive advantage estimates as you described. This is not a problem though, this is exactly what we'd expect to happen given your description of the environment.

You describe an environment in which the agent is likely going to be wandering around randomly (at least when it hasn't learned a good policy yet), and incurring negative rewards over and over again. This naturally leads to negative value estimates for all encountered states. Suddenly, it does something and the episode terminates; it receives a nice reward of $0$ rather than yet another negative reward (this actually "feels" like a bonus, a positive reward, something more than was expected). When your agent has not yet learned a good policy that can reach the better reward of $1$, this is indeed a good result, a good action, and it rightfully should get reinforced.

Because this event of the episode terminating is mostly uncorrelated with the state (I say "mostly", because in theory it probably ends up being a slightly rarer event in states close to the goal than in states far away from the goal), it will eventually (after sufficient training time) end up occurring approximately equally often in all states. From the perspective of an agent that is oblivious to the current time step, this will be perceived as an event that can simply occur by pure chance in a nondeterministic environment.

This is not necessarily a problem. It can slow down learning due to increased variance in your reward observations (which can be addressed by using low learning rates / large batch sizes), but Reinforcement Learning algorithms are almost always naturally built to handle nondeterministic environments, it can work this out, it can average out all the different outcomes observed for the same state+action pairs. This is not a problem that requires dealing with.




  My ideas:
  
  
  Differentiate between actual episode terminations and ones caused by the time limit, e.g. for them we will not replace next step value estimate with $0$.
  Somehow add $t$ to the state such that the value function can learn to anticipate the termination of the episode.
  


The first idea fundamentally changes the quantity that your algorithm is learning, it will essentially make your learning algorithm incorrect. There's always a chance that it might still appear to learn something useful (many Machine Learning/Reinforcement Learning algorithms can still appear to be okay even when there are bugs/technically incorrect parts), but it'll very likely perform worse.

The second idea, while not necessary as explained in my answer above, may still be beneficial to learning speed provided it is done well. It may help because it can add to the power of your algorithm to "explain" its observations, and more importantly "explain" the variance in its observations.

The main problem I see with adding $t$ to your input is that it is not naturally a binary variable. Very often you'll find that we're just using a bunch of binary inputs in (Deep) Reinforcement Learning algorithms. When all inputs are of the same magnitude like that, it tends to be easier to get the learning process to run well, tune hyperparameters like learning rate, etc. If you suddenly plug in an additional input which can take significantly larger values (like straight up adding $t$ as an input), this will be more difficult. Adding $\frac{t}{T}$ as an input may be better, since that will always still be bounded between $0$ and $1$.