title: What is the difference between detach, clone and deepcopy in Pytorch tensors in detail?
body: <p>I've been struggling to understand the differences between <code>.clone()</code>, <code>.detach()</code> and <code>copy.deepcopy</code> when using Pytorch. In particular with Pytorch tensors.</p>
  <p>I tried writing all my question about their differences and uses cases and became overwhelmed quickly and realized that perhaps have the 4 main properties of Pytorch tensors would clarify much better which one to use that going through every small question. The 4 main properties I realized one needs keep track are:</p>
  <ol>
  <li>if one has a new pointer/reference to a tensor</li>
  <li>if one has a new tensor object instance (and thus most likely this new instance has it's own meta-data like <code>require_grads</code>, shape, <code>is_leaf</code>, etc.)</li>
  <li>if it has allocated a new memory for the tensor data (i.e. if this new tensor is a view of a different tensor)</li>
  <li>if it's tracking the history of operations or not (or even if it's tracking a completely new history of operations or the same old one in the case of deep copy)</li>
  </ol>
  <p>According to what mined out from the Pytorch forums and the documentation this is my current distinctions for each when used on tensors:</p>
  <h1>Clone</h1>
  <p>For clone:</p>
  <pre><code>x_cloned = x.clone()
  </code></pre>
  <p>I believe this is how it behaves according to the main 4 properties:</p>
  <ol>
  <li>the cloned <code>x_cloned </code> has it's own python reference/pointer to the new object</li>
  <li>it has created it's own new tensor object instance (with it's separate meta-data)</li>
  <li>it has allocated a new memory for <code>x_new</code> with the same data as <code>x</code></li>
  <li>it is keeping track of the <strong>original</strong> history of operations and in addition included this <code>clone</code> operation as <code>.grad_fn=&lt;CloneBackward&gt;</code></li>
  </ol>
  <p>it seems that the main use of this as I understand is to create copies of things so that <code>inplace_</code> operations are safe. In addition coupled with <code>.detach</code> as <code>.detach().clone()</code> (the &quot;better&quot; order to do it btw) it creates a completely new tensor that has been detached with the old history and thus stops gradient flow through that path.</p>
  <h1>Detach</h1>
  <pre><code>x_detached = x.detach()
  </code></pre>
  <ol>
  <li>creates a new python reference (the only one that does not is doing <code>x_new = x</code> of course). One can use <code>id</code> for this one I believe</li>
  <li>it has created it's own new tensor object instance (with it's separate meta-data)</li>
  <li>it has <strong>NOT</strong> allocated a new memory for <code>x_detached</code> with the same data as x</li>
  <li>it cuts the history of the gradients and does not allow it to flow through it. I think it's right to think of it as having no history, as a brand new tensor.</li>
  </ol>
  <p>I believe the only sensible use I know of is of creating new copies with it's own memory when coupled with <code>.clone()</code> as <code>.detach().clone()</code>. Otherwise, I am not sure what the use it. Since it points to the original data, doing in place ops might be potentially dangerous (since it changes the old data but the change to the old data is <strong>NOT</strong> known by autograd in the earlier computation graph).</p>
  <h1>copy.deepcopy</h1>
  <pre><code>x_deepcopy = copy.deepcopy(x)
  </code></pre>
  <ol>
  <li>if one has a new pointer/reference to a tensor</li>
  <li>it creates a new tensor instance with it's own meta-data (all of the meta-data should point to deep copies, so new objects if it's implemented as one would expect I hope).</li>
  <li>it has it's own memory allocated for the tensor data</li>
  <li>If it truly is a deep copy, I would expect a deep copy of the history. So it should do a deep replication of the history. Though this seems really expensive but at least semantically consistent with what deep copy should be.</li>
  </ol>
  <p>I don't really see a use case for this. I assume anyone trying to use this really meant 1) <code>.detach().clone()</code> or just 2) <code>.clone()</code> by itself, depending if one wants to stop gradient flows to the earlier graph with 1 or if they want just to replicate the data with a new memory 2).</p>
  <p>So this is the best way I have to understand the differences as of now rather than ask all the different scenarios that one might use them.</p>
  <p>So is this right? Does anyone see any major flaw that needs to be correct?</p>
  <p>My own worry is about the semantics I gave to deep copy and wonder if it's correct wrt the deep copying the history.</p>
  <p>I think a list of common use cases for each would be wonderful.</p>
  <hr />
  <h1>Resources</h1>
  <p>these are all the resources I've read and participated to arrive at the conclusions in this question:</p>
  <ul>
  <li>Migration guide to 0.4.0 <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" rel="noreferrer">https://pytorch.org/blog/pytorch-0_4_0-migration-guide/</a></li>
  <li>Confusion about using clone: <a href="https://discuss.pytorch.org/t/confusion-about-using-clone/39673/3" rel="noreferrer">https://discuss.pytorch.org/t/confusion-about-using-clone/39673/3</a></li>
  <li>Clone and detach in v0.4.0: <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/2" rel="noreferrer">https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/2</a></li>
  <li>Docs for clone:
  <ul>
  <li><a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.clone" rel="noreferrer">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.clone</a></li>
  </ul>
  </li>
  <li>Docs for detach (search for the word detach in your browser there is no direct link):
  <ul>
  <li><a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noreferrer">https://pytorch.org/docs/stable/tensors.html#torch.Tensor</a></li>
  </ul>
  </li>
  <li>Difference between detach().clone() and clone().detach(): <a href="https://discuss.pytorch.org/t/difference-between-detach-clone-and-clone-detach/34173" rel="noreferrer">https://discuss.pytorch.org/t/difference-between-detach-clone-and-clone-detach/34173</a></li>
  <li>Why am I able to change the value of a tensor without the computation graph knowing about it in Pytorch with detach? <a href="https://stackoverflow.com/questions/62415251/why-am-i-able-to-change-the-value-of-a-tensor-without-the-computation-graph-know">Why am I able to change the value of a tensor without the computation graph knowing about it in Pytorch with detach?</a></li>
  <li>What is the difference between detach, clone and deepcopy in Pytorch tensors in detail? <a href="https://stackoverflow.com/questions/62437509/what-is-the-difference-between-detach-clone-and-deepcopy-in-pytorch-tensors-in">What is the difference between detach, clone and deepcopy in Pytorch tensors in detail?</a></li>
  <li>Copy.deepcopy() vs clone() <a href="https://discuss.pytorch.org/t/copy-deepcopy-vs-clone/55022/10" rel="noreferrer">https://discuss.pytorch.org/t/copy-deepcopy-vs-clone/55022/10</a></li>
  </ul>