How does Learning Rate affect Gradients in Back Prop
I was wondering if the learning rate is set to update the weights during the backpropagation on the layers of a neural network. How and where do the weights get updated?
I can't see the connection between optimizer, learning rate, and the backpropagation function.

Sure a fair question that comes to mind is how are the backpropagation and the optimizer related in all of this. And how are those parameters updated by the optimizer?
optimizer.zero_grad()
loss.backward()
optimizer.step()

Indeed there doesn't seem to be any link between optimizer and loss let alone the parameters of your model.


Optimizer

Every tensor object has a grad attribute which either contains a tensor representing the gradient corresponding to it or None if either it doesn't require gradient computation or simply doesn't have any gradient.
To optimize parameters in PyTorch you would go about initializing an optimizer by passing a list or iterator over those very parameters you want this optimizer to act upon:
mlp = nn.Sequential(nn.Linear(10, 2), nn.Linear(2, 1))
optimizer = torch.optim.SGD(mlp.parameters(), lr=1.0e-3)

An optimizer is not tasked with computing gradients, this is performed by another system in PyTorch.
When you call optimizer.step(), the optimizer will go over each provided parameter and update them based the optimizer's update rule (i.e. the optimizing method used) and the gradient associated with each parameter (namely the grad attribute)
For SGD it will be something like (leaving no_grad considerations aside):
for param in parameters:
    param -= lr*param.grad



Backpropagation

To actually compute the backward propagation you would use torch.autograd.backward which usually comes in the form of torch.Tensor.backward (as a torch.Tensor method).
This function is a mutable operator which will update the grad attribute of all leaf tensor nodes requiring gradient computation. In other words, it will compute the gradient of the tensor backward that was called upon w.r.t each parameter of the model.
For example with model mlp, we backpropagate on a dummy loss:
&gt;&gt;&gt; for x in mlp.parameters():
...     print(tuple(x.shape), x.grad)
(2, 10) None
(2,)    None
(1, 2)  None
(1,)    None

After inference and backpropagation on a random input:
&gt;&gt;&gt; mlp(torch.rand(1, 10)).mean().backward()

Each grad attribute of the model's tensor parameter has been updated by that call:
&gt;&gt;&gt; for x in mlp.parameters():
...    print(tuple(x.shape), x.grad is not None, tuple(x.grad.shape))
(2, 10) True (2, 10)
(2,)    True (2,)
(1, 2)  True (1, 2)
(1,)    True (1,)

Then you can call optimizer.step() to effectively perform the parameter update based on those gradients. Do note, the optimizer can only affect tensors that have been provided to it on initialization (recall the torch.optim.SGD(mlp.parameters(), lr=1.0e-3) part).
Finally you can zero the gradient of those parameter from the optimizer directly with zero_grad:
&gt;&gt;&gt; optimizer.zero_grad()

This is roughly a shorthand for:
for param in mtl.parameters():
    param.grad.zero_()

But its effectiveness is a lot more apparent when using multiple parameters groups and or multiple optimizers on the same model.
