Proof of the connection between V and Q in Reinforcement Learning
It follows very simply from the definitions of V and Q.
These are the relevant definitions:
$$G_t \space \dot{=} \space \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$
$$V^{\pi}(s) \space \dot{=} \space \mathbb{E}_{A \sim \pi}[G_t|S_t=s]$$
$$Q^{\pi}(s, a) \space \dot{=} \space \mathbb{E}_{A \sim \pi}[G_t|S_t=s, A_t=a]$$
You can ignore the definiton of return, $G_t$, I included it for completeness, and to simplify the other definitions.
The only difference between the definitions of V and Q is adding the selection of $a$ in the conditions. If you remove the action selection in the definition of Q, and condition it instead on following the policy, you are back to the definition of V.
You can also expand the expectation in the definition of V using the policy function to show the same thing:
$$V^{\pi}(s) = \mathbb{E}_{A \sim \pi}[G_t|S_t=s]$$
$$V^{\pi}(s) = \sum_{a}\pi(a|s) \mathbb{E}_{A \sim \pi}[G_t|S_t=s, A_t=a]$$
$$V^{\pi}(s) = \sum_{a}\pi(a|s) Q^{\pi}(s,a)$$
Convert this back to an expectation over the policy:
$$V^{\pi}(s) = \mathbb{E}_{A \sim \pi}[Q^{\pi}(S_t,A_t)|S_t=s]$$