Inplace arithmetic operation versus normal arithmetic operation in PyTorch Tensors
I am trying to build Linear regression using Pytorch framework and while implementing Gradient Descent, I observed two different outputs based on how I use arithmetic operation in Python code. Below is the code:

 #X and Y are input and target labels respectively
 X = torch.randn(100,1)*10
 Y = X + 3*torch.randn(100,1) +2
 plt.scatter(X.numpy(),Y.numpy())


 #Initialiation of weight and bias
 w = torch.tensor(1.0,requires_grad=True)
 b = torch.tensor(1.0,requires_grad=True)

 #forward pass
 def forward_feed(x):
   y = w*x +b
   return y

 #Parameters Learning
 epochs = 100
 lr = 0.00008
 loss_list = []
 for epoch in range(epochs):
   print('epoch',epoch)
   Y_pred = forward_feed(X)
   loss = torch.sum((Y - Y_pred)**2)
   loss_list.append(loss)
   loss.backward()
   with torch.no_grad():
     w -= lr*w.grad
     b -= lr*b.grad
     w.grad.zero_()
     b.grad.zero_()


If I use this code, I get the expected results i.e my code is able to estimate weight and bias. However, If I change the gradient descent code line like below:

  w =w- lr*w.grad
  b =b- lr*b.grad


I get the below error:

 AttributeError                            Traceback (most recent call 
 last)
 &lt;ipython-input-199-84b86804d4d5&gt; in &lt;module&gt;()
 ---&gt; 41         w.grad.zero_()
      42         b.grad.zero_()

 AttributeError: 'NoneType' object has no attribute 'zero_'


Can anyone please help me with this? 

I did try checking answers on google and found a related link: https://github.com/pytorch/pytorch/issues/7731. But this is exactly opposite to what I am facing. As per this link, they say that inplace assignment is causing a problem because tensors share the same storage. However, For my code, Inplace operation works not the normal operation.

I think the reason is simple. When you do:

w = w - lr * w.grad
b = b - lr * b.grad


The w and b in the left-hand side are two new tensors and their .grad is None. 

However, when you do inplace operation, you do not create any new tensor, you just update the value of the concerned tensor. So, in this scenario, inplace operation is required.
