How do we get the true value in the prediction objective in reinforcement learning?
The true value $v_{\pi}(s)$ is a conceptual target for the $\overline{VE}$ in the book. You often do not know it in real problems. However, it is still used in two main ways in the book:


Theoretically for analysis of different aprpoximation schemes, which can be shown to converge to minimise the $\overline{VE}$ objective, or a related one. 
In toy problems when exploring the nature of approximation in Reinforcement Learning (RL), it is possible to use tabular methods guranateed to get close to zero error, and then compare them to approximate methods. There are several plots of this type in the book.


The book shows derivation of gradient descent methods that start with minimising $\overline{VE}$ as an objective, and that use samples of $v_{\pi}(s)$ such as Monte Carlo return $G_t$ or the TD target) in place of the unknown $v_{\pi}(s)$ in the loss function. These also rely on the fact that the sample distribution will be weighted by $\mu(s)$ if they are taken naturally from the environment whilst the agent is following the policy $\pi$ so that $\mu(s)$ also does not need to be explicitly known. 

Outside of toy problems deliberately constructed to demonstrate that this theory is correct, you will not know $v_{\pi}(s)$ or be able to calculate $\overline{VE}$. However, you will know from the theory that if you follow the update rules derived in the book for approximate gradient descent methods, that the process should find a local minimum for $\overline{VE}$, for whatever state approximation scheme you have chosen to use.

Usually you cannot even approximate $\overline{VE}$ from raw data, as the variance in returns will add noise to the signal, and there is no way to separate variance in returns from error in approximation in the general case. However there are a couple of scenarios that do lend themselves to measuring this objective, provided you already have your estimate $\hat{v}(s,w)$ and the policy remains fixed throughout:


Simple, fast (perhaps simulated), environments which can be solved to arbitrary accuracy using tabular methods. In this case, you first calculate $v_{\pi}(s)$ using a non-approximate method, then sample many approximations by running the environment using policy $\pi$ and treating that as your data set.
Fully deterministic environments where $\pi$ is also deterministic. These have a variance of $0$ for Monte Carlo returns, so each observed return from any given state is already the true value of $v_{\pi}(s)$. Again you can just run the environment many times to get your data set to calculate  $v_{\pi}(s)$ and $\hat{v}(s,w)$ for the observed states in the correct frequencies, and thus have data to calculate $\overline{VE}$.