Computing gradients of intermediate nodes in PyTorch
I'm trying to learn how autograd works in PyTorch. In the simple program below, I don't understand why gradients of loss w.r.t W1 and W2 are None.  As far as I understand from the documentation, W1 and W2 are volatile, therefore gradients cannot be computed.  Is that it? I mean, how I cannot take derivative of the loss w.r.t intermediate nodes? Can anyone explain me what I am missing here?

import torch
import torch.autograd as tau

W = tau.Variable(torch.FloatTensor([[0, 1]]), requires_grad=True)
a = tau.Variable(torch.FloatTensor([[2, 2]]), requires_grad=False)
b = tau.Variable(torch.FloatTensor([[3, 3]]), requires_grad=False)

W1 = W  + a * a
W2 = W1 - b * b * b
Z = W2 * W2

print 'W:', W
print 'W1:', W1
print 'W2:', W2
print 'Z:', Z

loss = torch.sum((Z - 3) * (Z - 3))
print 'loss:', loss

# free W gradient buffer in case you are running this cell more than 2 times
if W.grad is not None: W.grad.data.zero_()

loss.backward()
print 'W.grad:', W.grad

# all of them are None
print 'W1.grad:', W1.grad
print 'W2.grad:', W2.grad
print 'a.grad:', a.grad
print 'b.grad:', b.grad
print 'Z.grad:', Z.grad


When required, intermediate gradients are accumulated in a C++ buffer but in order to save memory they are not retained by default (exposed in python object).
Only gradients of leaf Variables set with requires_grad=True will be retained (so Win your example)

One way to retain intermediate gradients is to register a hook. One hook for this job is retain_grad() (see PR)
In your example, if you write W2.retain_grad(), intermediate gradient of W2 will be exposed in W2.grad

W1 and W2 are not volatile (you can check by accessing their volatile attribute (ie: W1.volatile)) and cannot be because they are not leaf variables (such as W, a and b). On the contrary, the computation of their gradients is required, see their requires_grad attribute.
If only one leaf variable is volatile, the whole backward graph is not constructed (You can check by making a volatile and look at the loss gradient function)

a = tau.Variable(torch.FloatTensor([[2, 2]]), volatile=True)
# ...
assert loss.grad_fn is None


To sum up


Volatility implies no gradient computation: Useful in inference mode


Only one leaf variable set volatile disable gradient computation

Requiring gradients implies gradient computation. Intermediate ones are exposed or not


Only one leaf variable requiring grad enable gradient computation


