Maximum Entropy Policy Gradient Derivation
Writing the objective in a different way may be helpful.
If we only focus on the reward part and ignore the entropy part since the focus in the question is about the second summation, the original objective is
$$
J(\theta)=\sum_{t=1}^T\mathbb{E}_{(s_t,a_t)\sim q(s_t,a_t)}[r(s_t,a_t)]
$$
If we look at it from the perspective of the full trajectory $\tau$, just like the ELBO in the paper Eq.19, the objective is
$$
J(\theta)=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^Tr(s_t,a_t)\right]
$$
Then
$$
\begin{aligned}
\nabla_\theta J(\theta)&amp;=\nabla_\theta\sum_\tau q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&amp;=\sum_\tau \nabla_\theta q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&amp;=\sum_\tau q_\theta(\tau)\nabla_\theta \log q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&amp;=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\nabla_\theta \log q_\theta(\tau)\sum_{t=1}^Tr(s_t,a_t)\right]
\end{aligned}
$$
Since $q_\theta(\tau)=q(s_1)\prod_{t=1}^T q(s_{t+1}|s_t|a_t) q_\theta(a_t|s_t)$,
$$
\nabla_\theta \log q_\theta(\tau)=\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t)
$$
Then
$$
\nabla_\theta J(\theta)=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t) \sum_{t=1}^Tr(s_t,a_t)\right]
$$
That is where the second summation comes from.
Since policy at time $t'$ cannot affect reward at time $t$ when $t&lt;t'$ (causality),
$$
\begin{aligned}
\nabla_\theta J(\theta)&amp;=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t) \sum_{t'=t}^Tr(s_t',a_t')\right]\\
&amp;=\sum_{t=1}^T \mathbb{E}_{(s_t,a_t)\sim q(s_t,a_t)}\left[\nabla_\theta \log q_\theta(a_t|s_t) \sum_{t'=t}^Tr(s_t',a_t')\right]
\end{aligned}
$$