Why this PyTorch regression program reaches zero loss with periodic oscillations?
There is one x and one t with with dimensions 3x1. I am trying to find the w (3x3) and b (3,1) so they can please this equation:
t = w*x + b

The program below does oscillate. I tried to debug it with no success. Can someone else take a look? What did I miss?

class fit():
    def __init__(self, w, b):
        self.w = w
        self.b = b
    
    def forward(self, x):
        return torch.mm(self.w, x) + self.b
        
w = torch.tensor([[1., 1.1, 1.2],
                  [1., 1.1, 1.2],
                  [1., 1.1, 1.2]], requires_grad=True)

b = torch.tensor([[10.], [11.], [12.]], requires_grad=True)

x = torch.tensor([[1.], [2.], [3.]], requires_grad=False)
t = torch.tensor([[0.], [0.9], [0.1]], requires_grad=False)

model = fit(w, b)

alpha = 0.001

loss = []

arange = np.arange(200)

for i in arange:

    z = model.forward(x)

    l = (z - t)**2

    l = l.sum()
    loss.append(l)

    l.backward()

    model.w.data = model.w.data - alpha * model.w.grad
    model.b.data = model.b.data - alpha * model.b.grad

plt.plot(arange, loss)

If I use the other tools (torch.nn.Linear, torch.optim.sgd, torch.nn.smeloss) from PyTorch everything goes as expected.

You need to reset the gradient to 0 after each backprop. By default, pytorch accumulates gradients when you call loss.backward().
Replacing the last 2 instructions of your loop with the following lines should fix the issue :
with torch.no_grad():
    model.w.data = model.w.data - alpha * model.w.grad
    model.b.data = model.b.data - alpha * model.b.grad
    model.w.grad.zero_()
    model.b.grad.zero_()

