Trouble understanding the partial differentiation used in reinforcement learning
Your understanding of what's going on seems to be correct, just one little clarification: $u$ should be the model parameters of the deterministic policy $\mu(s,u)$ and not a distribution itself, same as $w$ are the model parameters of $Q(s,a,w)$, but that's probably what you meant (or I might be unfamiliar with the formulation).

Concerning your actual question, the update step implied by $\frac{\partial l}{\partial u}$ is supposed to make the deterministic policy $\mu(s,u)$ to get closer to the optimal $a$, which maximizes $Q(s,a,w)$. As $a = \mu(s,u)$, we have a composite function on our hands

$$ Q(s,a,w) = Q(s, \mu(s, u), w)$$

When updating the actor paramters $u$ such that $Q$ is maximized, we need to make a step in the direction of the gradient of $Q$ with respect to $u$ which, since it is a composite function, is computed using the chain rule

$$ \frac{\partial Q}{\partial u} = \frac{\partial Q}{\partial a}\frac{\partial a}{\partial u}$$

The notation is a bit sloppy, replacing $\mu$ and $a$ and so on, but that also seems to be the case in the literature. So what is going on intuitively consits of two parts:


moving $a$ in the direction of $\frac{\partial Q}{\partial a}$ will increase $Q$, e.g. in 1D if $\frac{\partial Q}{\partial a} &gt; 0$ increasing $a$ would increase $Q$ and if $\frac{\partial Q}{\partial a} &lt; 0$ increasing $a$ would decrease $Q$
moving $u$ in the direction of $\frac{\partial a}{\partial u}$ will increase $a$, in 1D the example would be the same as above


If you multiply these together and update $u$ according to the product you end up moving $u$ such that $Q$ increases by either increasing or decreasing $a$, which is exactly what you want to do.