PyTorch tensors have same value after being added to a list
While learning gradients and optimizing the process by Pytorch, I wanted to figure out the change of loss function values vs weights values with the graph. While I tried to graph, I used both numpy and torch, because I wanted to compare. During to store list of grad. and loss function values.
It works at numpy:
def gradient(x,y, y_predicted):
  return np.dot(2*x, y_predicted-y).mean() 

dw = gradient(x,y,y_pred)
dw_list.append(dw)

[-120.0, -112.8, -106.032, -99.67008, -93.68988, -88.06848, -82.78437, -77.81731, -73.14827, -68.75938]
It doesn't works at torch:
  for epoch in range(n_iters):
    # prediction = forward pass
    y_pred = forward(x)
    
    # loss
    l = loss(y, y_pred)
    loss_list.append(l)
    print('loss')
    print(l_list)
    # gradients = backward pass
    l.backward()  # calculate w.grad = dl/dw   and cumulative 
    
    # update weights
    with torch.no_grad():
        w -= learning_rate * w.grad
    print(f'w.grad before zero setting = {w.grad}')
    dw_list.append(w.grad) 
    print(dw_list)
    #print(f'w.grad before zero setting = {w.grad}')
    # zero gradients
    w.grad.zero_()

[tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485)]
Why dw_list.append(dw) works at numpy, but dw_list.append(w.grad) does not work at torch?
Why only the new value of grad had been filled up the whole array at each iter. at torch tensor?

w.grad is a tensor; it (the same tensor) is appended to the list at each iteration, so the list contains copies of the same tensor, not copies of its value at each point in time as you'd probably intend.
The standard way of handling this is to use:
dw_list.append(w.grad.detach().cpu().numpy()) 

Please have a look at this discussion for why detach() is necessary:
https://discuss.pytorch.org/t/should-it-really-be-necessary-to-do-var-detach-cpu-numpy/35489/6
By contrast, np.mean() returns a new python float object every time it is called, and so the values are different at the end.  The list append() is not doing anything different in the two cases.
P.S. I think this would also work:
dw_list.append(w.grad.clone())

however it would keep the cloned tensors in the graph (and on the gpu, if the originals were on the gpu).  That may or may not be what you want.
