.grad() returns None in pytorch
I am trying to write a simple script for parameter estimation (where parameters are weights here). I am facing problem when .grad() returns None. I have gone through this  and this link also and understood the concept both theoretically and practically. For me following script should work but unfortunately, it is not working.

My 1st attempt: Following script is my first attempt

alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]


        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        pred = torch.tensor([[x_dt],
                             [y0_dt],
                             [y_dt]],device=device

                                   )
        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)


Above code throws an error message 

element 0 of tensors does not require grad and does not have a grad_fn 

at line loss.backward() 

My Attempt 2: Improvement in 1st attempt is as follows:

gamma = torch.tensor(2.0, device=device, dtype=torch.float, requires_grad=True)
alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]


        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        pred = torch.tensor([[x_dt],
                             [y0_dt],
                             [y_dt]],device=device, 
                                   dtype=torch.float,
                                   requires_grad=True)
        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)
#        with torch.no_grad():
#            gamma -= leraning_rate * gamma.grad



Now the script is working but except pred.gred other two return None.

I want to update all the parameters after computing loss.backward() and update them but it is not happening due to None. Can anyone suggest me how to improve this script? Thanks.

You're breaking the computation graph by declaring a new tensor for pred. Instead you can use torch.stack. Also, x_dt and pred are non-leaf tensors so the gradients aren't retained by default. You can override this behavior by using .retain_grad().

gamma = torch.tensor(2.0, device=device, dtype=torch.float, requires_grad=True)
alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]

        # retain the gradient for non-leaf tensors
        x_dt.retain_grad()

        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        # use stack instead of declaring a new tensor
        pred = torch.stack([x_dt, y0_dt, y_dt], dim=0).unsqueeze(1)

        # pred is also a non-leaf tensor so we need to tell pytorch to retain its grad
        pred.retain_grad()

        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)
        with torch.no_grad():
            gamma -= learning_rate * gamma.grad




Closed form solution

Assuming you want to optimize for the parameters defined at the top of the function gamma, alpha_xy, beta_y, etc... Then what you have here is an example of ordinary least squares. See least squares for a slightly friendlier introduction to the topic. Take a look at the components of pred and you'll notice that x_dt, y0_dt, and y_dt are actually independent of each other with respect to the parameters (in this case it's obvious because they each use totally different parameters). This makes the problem much easier because it means we can actually optimize the terms (x_dt - target[0])**2, (y0_dt - target[1])**2 and (y_dt - target[2])**2 separately!

Without getting into the details the solution (without back-propagation or gradient descent) ends up being

# supposing x_train is [N,3] and y_train is [N,3]
x1 = torch.stack((x_train[:, 0], x_train[:, 0] * x_train[:, 2]), dim=0)
y1 = y_train[:, 0].unsqueeze(1)

# avoid inverses using solve to get p1 = inv(x1 . x1^T) . x1 . y1
p1, _ = torch.solve(x1 @ y1, x1 @ x1.transpose(1, 0))

# gamma and alpha1 are redundant. As long as gamma + alpha1 = p1[0] we get the same optimal value for loss
gamma = p1[0] / 2
alpha_xy = p1[1]
alpha1 = p1[0] / 2

x2 = torch.stack((x_train[:, 0], x_train[:, 1]), dim=0)
y2 = y_train[:, 1].unsqueeze(1)

p2, _ = torch.solve(x2 @ y2, x2 @ x2.transpose(1, 0))

beta_y = p2[0]
alpha2 = p2[1]

x3 = torch.stack((x_train[:, 1], x_train[:, 2], x_train[:, 0] * x_train[:, 2]), dim=0)
y3 = y_train[:, 2].unsqueeze(1)

p3, _ = torch.solve(x3 @ y3, x3 @ x3.transpose(1, 0))

alpha0 = p3[0]
alpha_y = p3[1]
alpha3 = p3[2]

loss_1 = torch.sum((x1.transpose(1, 0) @ p1 - y1)**2 + (x2.transpose(1, 0) @ p2 - y2)**2 + (x3.transpose(1, 0) @ p3 - y3)**2)
mse = loss_1 / x_train.size(0)


To test this code is working I generated some fake data which I knew the underlying model coefficients (there's some noise added so the final result won't exactly match the expected).

def gen_fake_data(samples=50000):
    x_train = torch.randn(samples, 3)
    # define fake data with known minimal solutions
    x1 = torch.stack((x_train[:, 0], x_train[:, 0] * x_train[:, 2]), dim=0)
    x2 = torch.stack((x_train[:, 0], x_train[:, 1]), dim=0)
    x3 = torch.stack((x_train[:, 1], x_train[:, 2], x_train[:, 0] * x_train[:, 2]), dim=0)
    y1 = x1.transpose(1, 0) @ torch.tensor([[1.0], [2.0]])  # gamma + alpha1 = 1.0
    y2 = x2.transpose(1, 0) @ torch.tensor([[3.0], [4.0]])
    y3 = x3.transpose(1, 0) @ torch.tensor([[5.0], [6.0], [7.0]])
    y_train = torch.cat((y1, y2, y3), dim=1) + 0.1 * torch.randn(samples, 3)
    return x_train, y_train

x_train, y_train = gen_fake_data()

# optimization code from above
...

print('loss_1:', loss_1.item())
print('MSE:', mse.item())

print('Expected 0.5, 2.0, 0.5, 3.0, 4.0, 5.0, 6.0, 7.0')
print('Actual', gamma.item(), alpha_xy.item(), alpha1.item(), beta_y.item(), alpha2.item(), alpha0.item(), alpha_y.item(), alpha3.item())


which results in

loss_1: 1491.731201171875
MSE: 0.029834624379873276
Expected 0.5, 2.0, 0.5, 3.0, 4.0, 5.0, 6.0, 7.0
Actual 0.50002 2.0011 0.50002 3.0009 3.9997 5.0000 6.0002 6.9994

