Is the policy gradient expression in Fundamentals of Deep Learning wrong?
There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.
In short, we want to first find the derivative of the RL objective $J(\theta) = v_\pi(s_0)$, where $\pi$ is our policy that depends on some parameters $\theta$. The policy gradient theorem tells us that
$$\nabla_\theta J(\theta) = \mathbb{E}_{a\sim\pi, s\sim\mu}\left[ G_t \nabla_\theta \log\pi(A_t|S_t)\right]\;;$$
where $\mu$ is our state distribution induced by the policy $\pi$ and $G_t$ are the discounted returns.
Now, as we want to maximise our objective $J(\theta)$, we want to perform gradient ascent. That is, our parameters should be updated according to
$$\theta_{t+1} = \theta_t + \alpha \nabla_\theta J(\theta)\;.$$
The reason that you will often see the update written as a minimisation problem is because most software is built to minimise functions, rather than maximise. This is not a problem though as finding the maximum of $f$ is just the same as finding the minimum of $-f$. Using this, we can then say our parameters can be updated using gradient descent on the negative of our objective, i.e. we get our parameter update rule to be
$$\theta_{t+1} = \theta_t - \alpha \nabla_\theta (-J(\theta))\;;$$
which is exactly what you have in the book.