How differential semi-gradient Sarsa updates estimated average reward?
In a continuing task, TD error can increase w to infinity unless its expected value is zero. By subtracting the TD average estimation, the expected value of our update value is zero and w cannot go to infinity. TD error is a biased estimation of the average reward ( with a bias that goes to zero as the number of updates goes to infinity assuming that every state can be reached from every state since the average reward is independent of the state action combination with which we start).