How can both agents know the terminal reward in self-play reinforcement learning? There seems to be a major difference in how the terminal reward is received/handled in self-play RL vs &quot;normal&quot; RL, which confuses me.
I implemented TicTacToe the normal way, where a single agent plays against an environment that manages the state and also replies with a new move. In this scenario, the agent receives a final reward of $+1$, $0$ and $-1$ for a win, draw, and loss, respectively.
Next, I implemented TicTacToe in a self-play mode, where two agents perform moves one after the other, and the environment only manages the state and gives back the reward. In this scenario, an agent can only receive a final reward of $+1$ or $0$, because, after his own move, he will never be in a terminal state in which he lost (only agent 2 could terminate the game in such a way). That means:

In self-play, episodes end in such a way that only one of the players sees the terminal state and terminal reward.

Because of point one, an agent can not learn if he made a bad move that enabled his opponent to win the episode. Simply because he does not receive a negative reward.


This seems very weird to me. What am I doing wrong?  Or if I'm not wrong, how do I handle this problem?
When one agent makes a move, that move should be perceived as part of the "state transition" executed "by the environment" from the perspective of the other agent.

For example, suppose that, as a "neutral third party" we view the game as follows, as a sequence of states, actions and a terminal reward. I use A to denote actions selected by the first player, and B to denote actions selected by the second player:

S1 -&gt; A1 -&gt; S2 -&gt; B1 -&gt; S3 -&gt; A2 -&gt; S4 -&gt; B2 -&gt; S5 -&gt; A3 -&gt; Terminal Reward

Then, the first player should only get the following observations:

S1 -&gt; A1 -&gt; S3 -&gt; A2 -&gt; S5 -&gt; A3 -&gt; Terminal Reward

note how states S2 and S4 are skipped entirely, they are not really states from the perspective of the first player, they're just halfway through the transition caused by the first player's action and are not interesting for the first player.

Similarly, the second player should only get the following observations:

S2 -&gt; B1 -&gt; S4 -&gt; B2 -&gt; Terminal Reward