Pytorch - Gradient distribution between functions
Recall that you passed net.parameters() to the optimizer, so it has access to the "Tensor" objects, as well as their associated data. One of the associated data fields associated to each learnable tensor parameter is a gradient buffer. Hence, backward() not only computes the gradients, but stores them in each parameter tensor, so that the gradient vector per parameter is stored along with that parameter. In other words, for some parameter $\theta_i$, backward() stores $ \partial \mathcal{L}(\Theta)/\partial \theta_i$ along with that parameter. The optimizer.step() call then simply updates each parameter via the gradient stored along with it.