How can both agents know the terminal reward in self-play reinforcement learning?
When one agent makes a move, that move should be perceived as part of the "state transition" executed "by the environment" from the perspective of the other agent.

For example, suppose that, as a "neutral third party" we view the game as follows, as a sequence of states, actions and a terminal reward. I use A to denote actions selected by the first player, and B to denote actions selected by the second player:

S1 -&gt; A1 -&gt; S2 -&gt; B1 -&gt; S3 -&gt; A2 -&gt; S4 -&gt; B2 -&gt; S5 -&gt; A3 -&gt; Terminal Reward

Then, the first player should only get the following observations:

S1 -&gt; A1 -&gt; S3 -&gt; A2 -&gt; S5 -&gt; A3 -&gt; Terminal Reward

note how states S2 and S4 are skipped entirely, they are not really states from the perspective of the first player, they're just halfway through the transition caused by the first player's action and are not interesting for the first player.

Similarly, the second player should only get the following observations:

S2 -&gt; B1 -&gt; S4 -&gt; B2 -&gt; Terminal Reward