Numerical equivalence of PyTorch backpropagation
After i 'v written the simple neural network with numpy, i wanted to compare it numerically with PyTorch impementation. Running alone, seems my neural network implementation converges, so it seems to have no errors.
Also i v checked forward pass matches to PyTorch, so basic setup is correct.

But something different happens while backward pass, because the weights after one backpropagation are different. 

I dont want to post full code here because its linked over several .py files, and most of the code is irrelevant to the question. I just want to know does PyTorch "basic" gradient descent or something different. 

I m viewing the most simle example about full-connected weights of the last layer, cause if it is different, further will be also different:

self.weight +=  self.learning_rate * hidden_layer.T.dot(output_delta )


where 

output_delta  = self.expected -  self.output


self.expected  are expected value, 
self.output is forward pass result

No activation or further  stuff here.

The torch past is:

optimizer = torch.optim.SGD(nn.parameters() , lr = 1.0)

criterion = torch.nn.MSELoss(reduction='sum')   


output = nn.forward(x_train)


loss = criterion(output, y_train)

loss.backward()

optimizer.step()

optimizer.zero_grad()


So it is possible that with SGD optimizer and MSELoss it uses some different delta or backpropagation function, not the basic one mentioned above? If its so i d like to know how to numerically check my numpy solution with pytorch.


I just want to know does PyTorch &quot;basic&quot; gradient descent or something different.

If you set torch.optim.SGD, this means stochastic gradient descent.
You have different implementations on GD, but the one that is used in PyTorch is applied to mini-batches.
There are GD implementations that will optimize parameters after the full epoch. As you may guess they are very &quot;slow&quot;, this may be great for supercomputers to test. There are GD implementations that work for every sample, as you may guess their imperfectness is &quot;huge&quot; gradient fluctuations.
These are all relative terms, so I am using &quot;&quot; 
Note you are using too big learning rates like lr = 1.0, which means you haven't normalized your data at first, but this is a skill you may scalp over time.

So it is possible that with SGD optimizer and MSELoss it uses some different delta or backpropagation function, not the basic one mentioned above?

It uses what you told.
Here is a the example in PyTorch and in Python to show detection of gradients works as expected (used in back propagation) :
x = torch.tensor([5.], requires_grad=True);
print(x) # tensor([5.], requires_grad=True)

y = 3*x**2
y.backward()
print(x.grad) # tensor([30.]) 

How would you get this value 30 in plain python?
def y(x):
    return 3*x**2

x=5
e=0.01 #etha
g=(y(x+e)-y(x))/e 
print(g) # 30.0299

As we expect we got ~30, it would be even better with smaller etha.
