Why by changing tensor using detach method make backpropagation not always unable to work in pytorch?
After constructing a graph, when using the detach method to change some tensor value, it is expected that an error pops up when computing the back propagation. However, this is not always the case. In the following two blocks of code: the first one raises an error, while the second one does not. Why does this happen?
x = torch.tensor(3.0, requires_grad=True)
y = x + 1
z = y**2
c = y.detach()
c.zero_()
z.backward(retain_graph=True)
print(x.grad) # errors pop up

x = torch.tensor(3.0, requires_grad=True)
y1 = x+1
y2 = x**2
z = 3*y1 + 4*y2
c = y2.detach()
c.zero_()
z.backward(retain_graph=True)
print(x.grad) # no errors. The printed value is 27


TLDR; In the former example z = y**2, so dz/dy = 2*y, i.e. it's a function of y and requires its values to be unchanged to properly compute the backpropagation, hence the error message when applying the in-place operation. In the latter z = 3*y1 + 4*y2, so dz/dy2 = 4, i.e. y2 values are not needed to compute the gradient, as such its values can be modified freely.


In the former example you have the following computation graph:
x ---&gt; y = x + 1 ---&gt; z = y**2
        \
         \ ---&gt; c = y.detach().zero_()

Corresponding code:
x = torch.tensor(3.0, requires_grad=True)
y = x + 1
z = y**2
c = y.detach()
c.zero_()
z.backward() # errors pop up

When calling c = y.detach() you effectively detach c from the computation graph, while y remains attached. However, c shares the same data as y. This means when you call the in-place operation c.zero_, you end up affecting y. This is not allowed, because the y is part of a computation graph, and its values will be needed for a potential backpropagation from variable z.




The second scenario corresponds to this layout:
  /--&gt; y1 = x + 1 \
x                  ---&gt; z = 3*y1 + 4*y2
  \--&gt; y2 = x**2  /
         \
          \ ---&gt; c = y2.detach().zero_()

Corresponding code:
x = torch.tensor(3.0, requires_grad=True)
y1 = x + 1
y2 = x**2
z = 3*y1 + 4*y2
c = y2.detach()
c.zero_()
z.backward()
print(x.grad) # no errors. The printed value is 27

Here again, we have the same setup, you detach then in-place modify c and y with zero_.



The only difference is the operation performed on y and y2 (in the 1st and 2nd example respectively).

In the former, you have z = y**2, so the derivative is 2*y, hence the value of y is needed to compute the gradient of that operation.

In the latter example though z(y2) = constant + 4*y2 so the derivative with respect to y2 is just a constant: 4, i.e. it doesn't require the value of y2 to compute its derivative. You can check this by, for instance, defining in 2nd example z with z = 3*y1 + 4*y2**2: it will raise an error.


