import torch
   a = torch.rand((3, 3), requires_grad=True)
   b = torch.rand((3, 3), requires_grad=True)
   c = torch.rand((3, 3), requires_grad=True)
   d = a + b
   e = b * c
   d.sum.().backward()
This is what the autograd graph for the code above would look like:
[image]
The first step the autograd engine performs as part of the backward
pass is computing the number of dependencies for each node in the
autograd graph. This helps the autograd engine know when a node in the
graph is ready for execution. The numbers in brackets for "add(1)" and
"mul(0)" denote the number of dependencies. As you can see, this means
during the backward pass the "add" node needs 1 input and the "mul"
node doesn't need any inputs (in other words doesn't need to be
executed). The local autograd engine computes these dependencies by
traversing the graph from the root nodes ("d" in this case).
The fact that certain nodes in the autograd graph might not be
executed in the backward pass poses a challenge for distributed