Why is "next state" kept in RL experience replay?
True, but if you take a look at the equations of Q learning or Advantage functions in Policy Gradients you will see that the expected value of the next state is being used. For this, you need to know in which state you have landed from state $s_t$. For example, you can approximate your future reward from state $s_{t+1}$ by using $Q(s_{t+1})$. To do this you need to give to your network as input the $s_{t+1}$.