Strange behavior of linear regression in PyTorch
I am facing a peculiar problem and I was wondering if there is an explanation. I am trying to run a linear regression problem and test different optimization methods and two of them have a strange outcome when comparing to each other. I build a data set that satisfies y=2x+5 and I add a random noise to that.

 xtrain=np.range(0,50,1).reshape(50,1)
 ytrain=2*train+5+np.random.normal(0,2,(50,1))
 opt1=torch.optim.SGD(model.parameters(),lr=1e-5,momentum=0.8))
 opt2=torch.optim.Rprop(model.parameters(),lr=1e-5)
 F_loss=F.mse_loss
 from torch.utils.data import TensorDataset,DataLoader
 train_d=TensorDataset(xtrain,ytrain)     
 train=DataLoader(train_d,50,shuffle=True)     
 model1=nn.Linear(1,1)
 loss=F_loss(model1(xtrain),ytrain)

 def fit(nepoch, model1, F_loss, opt):
     for epoch in range(nepoch):
         for i,j in train:
         predict = model1(i)
         loss = F_loss(predict, j)
         loss.backward()
         opt.step()
         opt.zero_grad()


When i compare the results of the following commands:

        fit(500000, model1, F_loss, opt1)
        fit(500000, model1, F_loss, opt2)


In the last epoch for opt1:loss=2.86,weight=2.02,bias=4.46
 In the last epoch for opt2:loss=3.47,weight=2.02,bias=4.68

These results do not make sense to me, shouldn't opt2 have a smaller loss than opt1 since the weight and bias it finds is closer to the real value? opt2's method finds weights and biases to be closer to the real value (they are respectively 2 and 5). Am i doing something wrong? 

This has to do with the fact that you are drawing the training samples themselves from a random distribution.

By doing so, you inherently randomized the ground truth to some extent. Sure, you will get values that are inherently distributed around 2x+5, but you do not guarantee that 2x+5 will also be the best fit to this data distribution.
It could thus happen that you accidentally end up with values that deviate quite significantly from the original function, and, since you use a mean squared error, these values get weighted quite significantly.

In expectation (i.e., for the number of samples going towards infinity), you will likely get closer and closer to the expected parameters.

A way to verify this would be to plot your training samples against the parameter set, as well as the (ideal) underlying function.
Also note that Linear Regression does have a direct solution - something that is very uncommon in Machine Learning - meaning you can directly calculate an optimal solution, e.g., with sklearn's function
