Why state-action value function as an expected value of the return and state value function, does not need to follow policy?
Let's first write the state-value function as
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E,a_t \sim \pi}[r(s_t,a_t) + \gamma G_{t+1} | S_t = s, A_t = a]\; ;$$
where $r(s_t,a_t)$ is written to show that the reward gained at time $t+1$ is a function of the state and action tuple we have at time $t$ (note here that $G_{t+1}$ would be just the sum of future reward signals). This allows us to show that the expectation is taken under the joint distribution of $s,r\sim E$ where $E$ is the environment and actions are taken from our policy distribution.
As we have conditioned on knowing $a_t$ then the only random variable in the expectation that is dependent on our policy $\pi$ is $G_{t+1}$ because this is the  sum of future reward signals and so will depend on future state-action values. Thus, we can rewrite again as
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma \mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] | S_t = s, A_t = a]\;,$$
where the inner expectation (coupled with the fact its inside an expectation over the state and reward distributions) should look familiar to you as the state value function, i.e.
$$\mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] = v_{\pi}(s')\;.$$
This leads us to get what you have
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma v_{\pi}(s') | S_t = s, A_t = a]\;,$$
where the only difference is that we have made clear what our expectation is taken with respect to.
The expectation is taken with respect to the conditional distribution $S_{t+1},R_{t+1}|A_t,S_t$, and we usually include the $\pi$ subscript to denote that they are also taking the expectation with respect to the policy, but here this does not effect the first term as we have conditioned on knowing $A_t$ and only applies to the future reward signals.