Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?
When you say simple policy gradient, I assume you mean something like REINFORCE. 

The main difference between actor-critic and REINFORCE-like algorithms is in how they estimate the reward to go: 


in REINFORCE, you wait until a trajectory terminates to make any updates, and your estimator of the reward to go is the actual reward to go that was observed in the trajectory. 
in actor critic algorithms, instead of using an entire trajectory to estimate a reward to go, you use the action value function (note that this way, you can compute updates at every time step). The reason why you use the action-value function is because at each transition, you have already committed to some action. 


The benefit of the actor critic estimator is that it exhibits much less variance. REINFORCE estimators are Monte Carlo estimators, which are known to exhibit extremely high variance. 

Now, the actor critic method in the slides you posted takes variance reduction one step further. Instead of estimating reward to go, it's estimating advantage -- that is, the difference between the current action value and your estimated state value. The state value function term doesn't depend on the action, so it doesn't affect the expected value of the policy gradient. However, it serves as a 'baseline', which helps reduce the variance of the reward to go estimator even further.