Why is there an expectation sign in the Bellman equation? In chapter 3.5 of Sutton's book, the value function is defined as:



Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the agent is following a fixed policy $\pi$, why there should be an expectation when the trajectory of the future possible states is fixed (or maybe I am getting it wrong and it's not). In total, if the expectation here has the meaning of averaging over a series of trajectories, what are those trajectories and what are the weights of them when we want to compute the expected value over them according to this Wikipedia definition of the expected values?
There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-


The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\pi(s_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $t$, $a_t$ is distributed as-
$$a_t \sim \pi(s_t)$$
Even if the policy $\pi$ being followed by an agent is deterministic, there still needs to be an expectation over the behavior of the underlying stochastic MDP environment. That is, any action $a_t$, in general, only provides us with a distribution over the possible next states of the agent. That is, 
$$P(s_{t + 1} = s') = P_{\pi}(s' | s_t) = \sum_{a \in A} T(s,a,s') \times P_{\pi}(a_t = a)$$ 
Here $T(s, a, s')$ is the transition function for the MDP and the above equation captures the stochasticity arising from both 1 and 2.


As you see the expectation does not have to do with averaging over a collection of trajectories. However, that idea is often used in Monte-Carlo estimation of value functions. 

EDIT:
As pointed out in the comments, it is not correct to say that the expectation is not over a collection of trajectories.