Why do we calculate the mean squared error loss to improve the value approximation in Advantage Actor-Critic Algorithm?
I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to train the network is 
$$\mathbb{E}[(r + \gamma \max_{a'} Q(s',a') - Q(s,a))^2]\;.$$
Here, $r + \gamma \max_{a'} Q(s',a')$ is your target, what you want your network to aim towards, and $Q(s,a)$ is what your network predicted. (Note that I have left off some details that can be found in the Nature paper for simplicity). 

As for actor-critic methods, most popular actor-critic methods will use the value function to 'replace' the action-value function by using the following relationship:
$$\mathbb{E}[r + \gamma v_\pi(s')] = Q_\pi(s,a)\;.$$
This relationship can be proved by looking at exercise 3.13 (or somewhere around there) in the Sutton and Barto textbook. This looks like what the author is doing in the textbook you are reading.

Based on what I said at the start regarding how state-action value functions are trained, it is analogous to train a critic network that approximates the value function in the same way. 

e1: spelling.

e2: added link to nature paper.