Am I understanding PyTorch's add_ and mul_ correctly?
In this notebook the author writes the following nesterov update:

def nesterov_update(w, dw, v, lr, weight_decay, momentum):
    dw.add_(weight_decay, w).mul_(-lr)
    v.mul_(momentum).add_(dw)
    w.add_(dw.add_(momentum, v))


As I understand it, a.add(b) in PyTorch implements a+b and a.add(b,c) implements a+(b*c), because b is in the slot of the alpha parameter. And lastly, add_ does the in-place version of add.

Q: Am I right so far?

Then, if I were to sketch the above nesterov update in an expanded form that illustrates the logic, I would write:

dw = -lr*(dw + weight_decay*w)
v = v*momentum + dw
w = w + dw + momentum*v


Q: is this correct?

I'm not planning to use the above expanded "code," I'm just writing it this way to try communicate what I'm understanding that it's doing, to check.

It is important to note the PyTorch version (1.1.0) the tutorial is using. According to 1.1.0, function prototype for torch.add is torch.add(input, value=1, other, out=None). So, your interpretation of the following line:

dw.add_(weight_decay, w)


as: dw = dw + weight_decay * w is correct. So, the answer to your first question is, yes, you are right.

However, with the latest versions of PyTorch, you would get an error if torch.add is used in the same fashion.

a = torch.FloatTensor([0, 1.0, 2.0, 3.0])
b = torch.FloatTensor([0, 4.0, 5.0, 6.0])
c = 1.0
z = a.add(b, c)


The above code gives: (In PyTorch 1.5.0)

TypeError: add() takes 1 positional argument but 2 were given


However, if you perform the following, then it works fine.

z = a.add(b, alpha=c)


Note that, the prototype of torch.add is now: torch.add(input, other, *, alpha=1, out=None)



The answer to your second question is, yes, you are right.
