How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely


$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$?

Yes this is 100% correct.
There is no &quot;trick&quot; to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between $v_{\pi}$ and $q_{\pi}$ in that direction.
Note that for a deterministic policy, with $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ then the relationship is
$$v_\pi(s) = q_\pi(s, \pi(s))$$
The related exercise in the book - expressing $q_{\pi}$ in terms of $v_{\pi}$ and the MDP characteristics - is more complex because it involves a time step.