How we are calculating average reward ($r(\pi)$) if the policy changes over time?
You are correct: to evaluate a policy, we need to fix it.

We can temporarily fix it, just to evaluate it over a number of test cases. For a fair comparison, we should fix the start states and random seeds used for the transitions.
We can wait until convergence / until we are satisfied. The resulting policy would be what we implement in the &quot;true&quot;, trained agent. This is important when exploration might be harmful in the &quot;real world&quot; domain where the agent will be operating.
We can also measure average reward of the &quot;non-stationary&quot; policy, and assume that, once the agent is doing well, this should be close enough to evaluating the fixed policy. This is not ideal, but on the other hand it is trivial to implement, and is often used to track the learning process. If you have a life-long learning agent, this might be the best you can do.