What is the Q function and what is the V function in reinforcement learning? It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got something wrong.

Definitions

Q- and V-learning are in the context of Markov Decision Processes. A MDP is a 5-tuple $(S, A, P, R, \gamma)$ with


$S$ is a set of states (typically finite)
$A$ is a set of actions (typically finite)
$P(s, s', a) = P(s_{t+1} = s' | s_t = s, a_t = a)$ is the probability to get from state $s$ to state $s'$ with action $a$.
$R(s, s', a) \in \mathbb{R}$ is the immediate reward after going from state $s$ to state $s'$ with action $a$. (It seems to me that usually only $s'$ matters).
$\gamma \in [0, 1]$ is called discount factor and determines if one focuses on immediate rewards ($\gamma = 0$), the total reward ($\gamma = 1$) or some trade-off.


A policy $\pi$, according to Reinforcement Learning: An Introduction by Sutton and Barto is a function $\pi: S \rightarrow A$ (this could be probabilistic).

According to Mario Martins slides, the $V$ function is
$$V^\pi(s) = E_\pi \{R_t | s_t = s\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s\}$$
and the Q function is
$$Q^\pi(s, a) = E_\pi \{R_t | s_t = s, a_t = a\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s, a_t=a\}$$

My thoughts

The $V$ function states what the expected overall value (not reward!) of a state $s$ under the policy $\pi$ is.

The $Q$ function states what the value of a state $s$ and an action $a$  under the policy $\pi$ is.

This means,
$$Q^\pi(s, \pi(s)) = V^\pi(s)$$

Right? So why do we have the value function at all? (I guess I mixed up something)
$V^\pi(s)$ is the state-value function of MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$.
In the expression
$$V^\pi(s) = E_{\pi} \{G_t \vert s_t = s\} $$
$G_t$ is the total DISCOUNTED reward from time step $t$, as opposed to $R_t$ which is an immediate return. Here you are taking the expectation of ALL actions according to the policy $\pi$.
$Q^\pi(s, a)$ is the action-value function. It is the expected return starting from state $s$, following policy $\pi$, taking action $a$. It's focusing on the particular action at the particular state.
$$Q^\pi(s, a) = E_\pi \{G_t | s_t = s, a_t = a\}$$
The relationship between $Q^\pi$ and $V^\pi$ (the value of being in that state) is
$$V^\pi(s) = \sum_{a ∈ A} \pi (a|s) * Q^\pi(s,a)$$
You sum every action-value multiplied by the probability to take that action (the policy $\pi(a|s)$).
If you think of the grid world example, you multiply the probability of (up/down/right/left) with the one step ahead state value of (up/down/right/left).