title: Why pytorch needs much more memory than it should?
body: <p>I'm just playing around with pytorch and I'm wondering why it consumes so much memory of my GPU?</p>
  
  <p>I'm using Cuda 10.0 with pythorch 1.2.0 and torchvision 0.4.0.</p>
  
  <pre><code>import torch
  gpu = torch.device("cuda")
  x = torch.ones(int(4e8), device=gpu)
  y = torch.ones(int(1e5), device=gpu)
  </code></pre>
  
  <p>Running the above code I get the error:
  RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 2.00 GiB total capacity; 1.49 GiB already allocated; 0 bytes free; 0 bytes cached)</p>
  
  <p>So, does pytorch needs ~500MB of the gpu memory as overhead? Or what is the problem here?</p>