Functionality of `detach()` in PyTorch for this specific case
If you do not detach the mean, then you have lateral dependencies between all elements in the batch (dim=0) when estimating the gradient.
I know that detach() is used for detaching a variable from the computational graph. In that context, are the following expressions x = x - torch.mean(x, dim=0).detach() and x = x - torch.mean(x, dim=0) equivalent? I just want to subtract the mean out, don't want to pass gradients through the average calculation.