PyTorch gradient doesn't flow through a clone of a tensor
I'm trying to have my model learn a certain function. I have parameters self.a, self.b, self.c that are trainable. I'm trying to force self.b to be in a certain range by using tanh. However, when I run the code it appears as the gradient is flowing through the original parameter (self.b), but not through the clone (self.b_opt)
import torch
import torch.nn as nn
import torch.optim as optim

class model(nn.Module): 
        def __init__(self):
            super(model, self).__init__()
            self.a = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.b = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.c = torch.nn.Parameter(torch.rand(1, requires_grad=True))

            self.b_opt = torch.tanh(self.b.clone())

model_net = model()
#function to learn = 5 * (r &gt; 2) * (3) 
optimizer = optim.Adam(model_net.parameters(), lr = 0.1)

for epoch in range(10):
  for r in range(5):
    optimizer.zero_grad()
    loss = 5 * (r &gt; 2) * (3) - model_net.a * torch.sigmoid((r - model_net.b_opt)) * (model_net.c)

    loss.backward(retain_graph=True)
    optimizer.step()
  #print(model_net.a)
  print(model_net.b)
  print(model_net.b_opt)
  #print(model_net.c)
  print()

&gt;&gt;&gt; Parameter containing:
tensor([0.4298], requires_grad=True)
tensor([0.7229], grad_fn=&lt;TanhBackward&gt;)

Parameter containing:
tensor([-0.0277], requires_grad=True)
tensor([0.7229], grad_fn=&lt;TanhBackward&gt;)

Parameter containing:
tensor([-0.5007], requires_grad=True)
tensor([0.7229], grad_fn=&lt;TanhBackward&gt;)


gradient does actually flows through b_opt since it's the tensor that is involved in your loss function. However, it is not a leaf tensor (it is the result of operations on tensors, specifically a clone and a tanh, you can check with model_net.b_opt.is_leaf), which means it allows gradients to be propagated but does not accumulate them (b_opt.grad does not exist). You cannot update the value of b_opt through backprop.
However the fact that the value of b is updated proves that the gradient does flow ! What you actually need is to recompute b_opt each time b is updated, because here your b_opt is computed once and for all in the init function. That would be something like
...
loss.backward(retain_graph=True)
optimizer.step()
self.b_opt = torch.tanh(self.b.clone())
...

A much simpler way to do all this though would be to get rid of b_opt altogether, and replace your loss with
loss = 5 * (r &gt; 2) * (3) - model_net.a * torch.sigmoid((r - tanh(model_net.b))) * (model_net.c)

