Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value?
I think this equation answer your question:
$$ q_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum_{a \in A}\pi^{i+1}(a|s)q_{\pi^{i}}(s,a)$$

value of the Q while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$ is equal to the expected q value while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$. And for the second part of your question the answer is:

$$ V_{\pi^{i}}(s) = q_{\pi^{i}}(s,\pi^{i}(s))$$

state Value function following the policy $\pi^{i}$ is the same as action-value function while taking action from policy $\pi^{i}$ and thereafter following the policy $\pi^{i}$.