How to deal with the terminal state in SARSA in a multi-agent setting?
The SARSA update rule looks like:
$$Q(S, A) \gets Q(S, A) + \alpha \left[ R + \gamma Q(S', A') \right].$$
Very similar, the $Q$-learning update rule looks like:
$$Q(S, A) \gets Q(S, A) + \alpha \left[ R + \gamma \max_{A'} Q(S', A') \right].$$
Both of these update rules are formulated for single-agent Markov Decision Processes. Sometimes you can make them work reasonably ok in multi-agent settings, but it is crucial to remember that these update rules should still always be implemented &quot;from the perspective&quot; of a single learning agent, who is oblivious to the presence of other agents and pretends them to be a part of the environment.
What this means is that the states $S$ and $S'$ that you provide in update rules really must both be states in which the learning agent is allowed to make the next move (with the exception being that $S'$ is permitted to be a terminal game state.
So, suppose that you have three subsequent states $S_1$, $S_2$, and $S_3$, where the learning agent gets to select actions in states $S_1$ and $S_3$, and the opponent gets to select an action in state $S_2$. In the update rule, you should completely ignore $S_2$. This means that you should take $S = S_1$, and $S' = S_3$.
Following the reasoning I described above literally may indeed lead to a tricky situation with rewards from transitioning into terminal states, since technically every episode there will be only one agent that directly causes the transition into a terminal state. This issue (plus also some of my explanation above being repeated) is discussed in the &quot;How to see terminal reward in self-play reinforcement learning?&quot; question on this site.