Why is PyTorch handling similar weight update assignments differently?
The below code does work updating the weights using:
w -= lr * w.grad

But when updating the weights using
w = w - lr * w.grad

it throws me:

element 0 of tensors does not require grad and does not have a grad_fn

Why is that and shouldn't both assignments be equal?
import torch

X = torch.tensor([1, 2, 3, 4], dtype=torch.float32)
y = torch.tensor([2, 4, 6, 8], dtype=torch.float32)
w = torch.tensor(0.0, dtype=torch.float32, requires_grad=True)
epochs = 10
lr = 0.002

for epoch in range(1, epochs + 1):
    y_pred = w * X
    loss = ((y_pred - y)**2).mean()
    loss.backward()
    
    print(w.grad)
    with torch.no_grad():
        ### Option 1 - doesn't work
        w = w - lr * w.grad

        ### Option 2 - does work
        w -= lr * w.grad
        
    w.grad.zero_()


The difference is that -= is an in-place op and the alternative is not. Therefore, when using -= inside the .no_grad() context, the variable will compute the operation, but the gradient won't take that op into account.
When you perform a normal subtraction, you'd expect a SubBackward as grad_fn:
import torch
x = torch.tensor([3.], requires_grad=True)
print(x)
# &gt;&gt;&gt; tensor([3.], requires_grad=True)

x = x - 2
print(x)
# &gt;&gt;&gt; tensor([1.], grad_fn=&lt;SubBackward0&gt;)

and indeed, this is what we get. But, if we try -= inside the .no_grad() context:
with torch.no_grad():
    x -= 2

print(x)
# &gt;&gt;&gt; tensor([1.], requires_grad=True)

we get the expected result (i.e., 2), but no backward function (ofc, we specified that with .no_grad()). Note that it still requires_grad=True. However, if we try to run this in-place op out of the .no_grad() context, this is what happens:
x -= 2
# &gt;&gt;&gt; Traceback (most recent call last):
# &gt;&gt;&gt;   File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
# &gt;&gt;&gt; RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

and if we try to run the normal subtraction inside the .no_grad() context, we will get:
x = x - 2
print(x)
# &gt;&gt;&gt; tensor([-1.])

a tensor without requires_grad; and that is the reason you get the error when using this option.
