Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?
$\mu(s)$ is not in equation (9.4) because we are assuming that the examples by which we update our parameter $w$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $w$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by $\alpha$.

The minus is there because we are performing gradient descent. For more information on this, see e.g. the wikipedia page