In vanilla policy gradient is the baseline lagging behind the policy?
So is the baseline expected to be used in the next iteration when our policy has changed?


Yes.


  To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?


The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, $a$, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve. 

You could in theory use a fixed offset instead of $V(s)$, or any arbitrary function that does not depend on $a$. In some settings the average reward $\bar{R}$ seen so far is used. 

Using a rough approximation to $V(s)$ - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for adjustments to that  policy). The more accurate $V(s)$ is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.

For more on this, see Sutton &amp; Barto, chapter 13, section 13.4.