Where does entropy enter in Soft Actor-Critic?
In the answer I'll be using notation similar to the one from the SAC paper.
If we look at the standard objective function for policy gradient methods we have
\begin{align}
J_\pi &amp;= V_\pi(s_t)\\
&amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t)]\\
&amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + V(s_{t+1})]]\\
&amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) +  \mathbb E_{a_{t+1} \sim \pi(a|s_{t+1})}[ \mathbb E_{s_{t+2} \sim p(s|s_{t+1}, a_{t+1})} [r(s_{t+1}, a_{t+1}) + V(s_{t+2})]]]]\\
&amp;\cdots\\
&amp;= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r( s_t, a_t)]
\end{align}
If you keep unwinding this $V(s_{t+i})$ you will get expected sum of rewards.
We can define soft state value as
\begin{align}
V(s_t) &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathcal H(\cdot|s_t)]\\
&amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]]\\
&amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]
\end{align}
third equality comes from the fact that $\mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]$ is nonrandom so its the same thing as if we are sampling over $\pi$ only once.

In maximum entropy framework objective function would then be 
\begin{align}
J_\pi &amp;= V_\pi(s_t)\\
&amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]\\
&amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) - \log(\pi(a_t|s_t)) + V(s_{t+1})]]\\
&amp; \cdots\\
&amp;= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) -\log(\pi(a_t|s_t))]\\
&amp;= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) + \mathcal H(\cdot|s_t)]
\end{align}