Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:
$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(s_t = s \vert \pi)}\tag1$$
and the discounted total reward $J(\pi)$ as:
$$J(\pi) = \frac{1}{1-\gamma} E_{\substack{s\sim d^\pi \\ a \sim \pi \\ s' \sim P}}[R(s,a,s')]\tag2$$
I have two questions regarding these equations.
Question 1
Intuitively, I understand that $d^\pi(s)$ returns the discounted probability of landing on state $s$ when executing policy $\pi$.
I understand that the summation part of $(1)$ results in values that are greater than $1$, and are, therefore, not fit for a probability distribution. But I do not understand why the value that results from this is multiplied by $(1-\gamma)$.
I have read in this question that &quot;$(1−\gamma)$ normalizes all weights introduced by γ so that they are summed to $1$&quot;. I have confirmed that this is true, but I don't understand why.
I tested this with a simple example:
Suppose there is are only two states $s_A$ and $s_B$ and the probabilty of landing on $s_A$ is $0.4$ and on $s_B$ is $0.6$, independently of the previous state or action taken (therefore, independently of the policy $\pi$). Also suppose we set the maximum number of time steps $t_{max} = 1000$ (to make the equation easy to compute) and $\gamma = 0.9$.
Then:
$$d^\pi(s_A) = (1-0.9) \sum_{t=0}^{1000} 0.9^t \cdot 0.4 \approx (1-0.9) \cdot 4$$
and
$$d^\pi(s_B) \approx (1-0.9) \cdot 6$$
So indeed if we sum them and multiply by $(1-\gamma)$ we get:
$$(1-0.9)\cdot(4+6) = 1$$
Q: My question is why does multiplying by $(1-\gamma)$ normalize to $1$? And what does $(1-\gamma)$ represent in this context?
Question 2
Similarly, I can't understand the use of $\frac{1}{1-\gamma}$ in $(2)$.
Q: How does multiplying the expected value of the reward function by $\frac{1}{1-\gamma}$ result in the discounted reward, instead of multiplying by $\gamma$? What does $\frac{1}{1-\gamma}$ represent?
Question 1
The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows
$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$
When you multiply by $1-\gamma$ you get
$$ 1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$
Which can be equivalently written as
$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$
Hence we can see that by multiplying the coefficients by $(1-\gamma)$ we get a weighted sum of transition probabilities

Question 2
Multiplying by $(1-\lambda)$ is to cancel out the normalisation term which was included in the definition of the discounted distribution of the states. Partially expanding out your expressions for the discounted reward you get this
$$ J(\pi) = \frac{1}{1-\gamma}\sum_\limits{s}E_
{a\sim\pi,s'\sim P}[R(s,a,s')]\cdot\left((1-\gamma)\sum_\limits{t=0}^{\infty}\gamma^tP(s_t=s|\pi)\right)$$
The discounted return form is usually denoted
$$J(\pi) = E[\sum\gamma^iR_i(s,a,s')]$$
Where you notice the discounting terms aren't normalised which motivates cancelling it out