title: Pytorch Autograd gives different gradients when using .clamp instead of torch.relu
body: <p>I'm still working on my understanding of the PyTorch autograd system. One thing I'm struggling at is to understand why <code>.clamp(min=0)</code> and <code>nn.functional.relu()</code> seem to have different backward passes.</p>
  
  <p>It's especially confusing as <code>.clamp</code> is used equivalently to <code>relu</code> in PyTorch tutorials, such as <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-nn" rel="noreferrer">https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-nn</a>.</p>
  
  <p>I found this when analysing the gradients of a simple fully connected net with one hidden layer and a relu activation (linear in the outputlayer).</p>
  
  <p>to my understanding the output of the following code should be just zeros. I hope someone can show me what I am missing.</p>
  
  <pre class="lang-py prettyprint-override"><code>import torch
  dtype = torch.float
  
  x = torch.tensor([[3,2,1],
                    [1,0,2],
                    [4,1,2],
                    [0,0,1]], dtype=dtype)
  
  y = torch.ones(4,4)
  
  w1_a = torch.tensor([[1,2],
                       [0,1],
                       [4,0]], dtype=dtype, requires_grad=True)
  w1_b = w1_a.clone().detach()
  w1_b.requires_grad = True
  
  
  
  w2_a = torch.tensor([[-1, 1],
                       [-2, 3]], dtype=dtype, requires_grad=True)
  w2_b = w2_a.clone().detach()
  w2_b.requires_grad = True
  
  
  y_hat_a = torch.nn.functional.relu(x.mm(w1_a)).mm(w2_a)
  y_a = torch.ones_like(y_hat_a)
  y_hat_b = x.mm(w1_b).clamp(min=0).mm(w2_b)
  y_b = torch.ones_like(y_hat_b)
  
  loss_a = (y_hat_a - y_a).pow(2).sum()
  loss_b = (y_hat_b - y_b).pow(2).sum()
  
  loss_a.backward()
  loss_b.backward()
  
  print(w1_a.grad - w1_b.grad)
  print(w2_a.grad - w2_b.grad)
  
  # OUT:
  # tensor([[  0.,   0.],
  #         [  0.,   0.],
  #         [  0., -38.]])
  # tensor([[0., 0.],
  #         [0., 0.]])
  # 
  </code></pre>