multipying negated gradients by actions for the loss in actor nn of DDPG
This answer is not mine but a reply I received on Quora here:  https://www.quora.com/Why-is-the-loss-for-DDPG-Actor-the-product-of-gradients-of-Q-values-actions/answer/James-MacGlashan?filter=&amp;nsrc=2&amp;snid3=4129828389

Hopefully it helps someone rather than me just deleting the question and I am sorry for my lack of math jax knowledge.  

Let’s forget about DDPG, Q-functions, actors, and actions for a minute. I know, know, blasphemy, but bear with me for a moment and we’ll get back to that.

Let’s just talk about loss functions. We’re all good with loss functions, right? They’re the backbone of our standard supervised deep learning literature. That is, in supervised learning, what we do is we define a loss function that defines how bad a prediction is from what it should be.

As a grounding example, let’s revisit the L2 loss we use for a regression problem. We define this loss as:

L
2
(
p
,
l

)

1
2
(
p
−
l
)
2

Where 
p
 is a prediction from our neural net and 
l
 is the “label:” the value the prediction should have been. If we can tune our neural net parameters so that this loss is always zero, we win! In other words, we want to minimize this loss function.

And how do we minimize it? The standard way these days is to use some flavor of stochastic gradient descent (SGD). And to do that, what we want to do is differentiate our loss in terms of our neural-net parameters 
w
. If we used vanilla SGD, we’d do:

w
t
+

1

w
t
−
α
∇
w
L
2

Okay, but there’s a catch here, our loss function was in terms of 
p
, not 
w
. However, we describe our neural net predictions as a function of some input and the neural-net parameters w: 
p
(
x
,
w
)
 for which we know how to compute 
∇
w
p
. Since we know that, we can use the chain rule of differentiation, which says:

∇
w
L

2

∇
w
p
∇
p
L
2

Where since w is a vector, 
∇
w
p
 will refer to its (transposed) Jacobian matrix.

In our modern era of deep learning libraries, you typically don’t do that yourself. Instead, your library will use the autograd/backpropagation algorithm to compute 
∇
w
L
2
 for you, and it will do it by decomposing it slightly differently than we did above.

Despite that fact that modern autograd libraries will do things slightly differently, it will be important to see that we can express the gradient of our loss in terms of neural net parameters in that way.

Back to Actor Critic

Okay, now that we’ve refreshed our memory about loss functions and what applying the chain rule to a loss function looks like, let’s get back to Actor Critic and DDPG.

Let’s ask our selves: What do we really want actor critic to do?

Suppose someone handed you the actual Q-values for some policy. How can you improve that policy? Simple: by making a new policy that selects actions that maximize the Q-values you were handed. This is the foundation of RL. It’s why in Q-learning you set your policy to greedily select the action with the max Q-value (and maybe add some noise for exploration reasons).

Okay, lets assume I have a Q-function: how do I find a policy that maximizes the Q-function if the actions are continuous?? I can’t do what Q-learning does by simply evaluating each action and choosing the max, because there are an infinite number of actions to evaluate!

Wait a second though. We know how to maximize a function over continuous values. We’ve been doing it for ages in supervised learning! Note that maximizing a function is the same as minimizing the negative value of that function. That is:

arg
max
x
f
(
x

)

arg
min
x
−
f
(
x
)

Okay cool, so if I want to maximize some function, I can just use regular SGD on the negative value of that function.

From that, we now have an insight: what if we think of 
−
Q
(
s
,
a
)
 as a loss function, where the actions 
a
 are “predictions” from some neural net: the actor. In that case, we can use good ol’ SGD to train our actor to choose actions that maximize the Q-values.

To do so of course requires that we have a representation of Q that we can differentiate. But if we simultaneously train a neural net to estimate Q, and we believe its estimates are good**, then we can easily use it as the “loss” function for our actor!

Indeed, if -Q is our loss function, with actions 
a
 acting as our “predictions” from our neural-net actor model 
μ
 lets substitute that back into chain rule expression for loss functions that we wrote earlier and with which we’re familiar:

∇
w
−

Q

∇
w
μ
∇
a
−
Q

And there it is!

(You may note that the DDPG paper reverses the order of the multiplication from the above. The order really just depends on what convention the paper takes the gradients and Jacobian matrices to mean. If you go back to the original deterministic policy gradients paper, you’ll see they write it out as we did above.)

So why doesn’t DDPG just say to use autograd on -Q like we do with loss functions ?

Okay, so now you’re probably asking yourself:

Why didn’t the paper just say “use autograd on -Q”???

After all, in supervised learning papers we never write out the chain rule in that way, we just assume practitioners have an autograd library and only write the forward version of the loss.

The reason is if you’re not careful, running pure autograd will burn you!

In DDPG, recall that we’re training two neural networks: Q and 
μ
. If you simply feed 
μ
 into Q and run autograd, it’s going to produce gradients on the parameters of Q in addition to the parameters on 
μ
! But when optimizing the actor, we don’t want to change the Q function! In fact, if we did, it would be really easy to make any actor maximize Q by making Q always output huge values!

To run autograd through Q would be like letting a supervised learning algorithm change the values of labels, which clearly is not right!

You can’t simply “block gradients” on Q either, because then no gradients will flow back onto the actor.

So instead, you simply have to make sure you apply the chain rule through Q to 
μ
, without adjusting any of the parameters of Q. One way to do that is to compute the gradients of each manually and multiply them, as written in the paper.

There are other ways to do it in autograd libraries too, but now you’re starting to get really specific about which library you’re using. So for an academic paper, it’s better to simply write out precisely what gradient should be computed for the actor, and then let the practitioner decide the best way to compute it in their library of choice.

** This assumption that a neural net’s estimates of the Q-function are good is really important. Because it’s an estimate, it will have errors, and a limitation of the DDPG algorithm is that your actor will exploit whatever errors exist in your neural net’s estimate of Q. Consequently, finding ways to ensure the Q-estimate is good is a very important area of work.