Why semi-gradient is used instead of the true gradient in Q-learning?
The problem does not come from the above equation but from the bootstrapping method. Sutton's reinforcement learning book (2nd) gave a pretty good explanation to the problem in Chapter 9.3.
The idea here is you cannot obtain the true value for $V_\pi$, but you want to approximate it. My understand is that when bootstrapping, like with the TD method, $V_\pi$ depends on $w$. This breaks the assumption that $V_\pi$ is independent of $w$ and thus the gradient is not the true gradient and we call it semi-gradient. However, if you are not bootstrapping, as with Monte Carlo methods, $V_\pi$ will be unbiased and the above will be guaranteed to converge to a local minimal.