Why is the memory in GPU still in use after clearing the object?
Starting with zero usage:

&gt;&gt;&gt; import gc
&gt;&gt;&gt; import GPUtil
&gt;&gt;&gt; import torch
&gt;&gt;&gt; GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  0% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |


Then I create a big enough tensor and hog the memory:

&gt;&gt;&gt; x = torch.rand(10000,300,200).cuda()
&gt;&gt;&gt; GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |


Then I tried several ways to see if the tensor disappears.

Attempt 1: Detach, send to CPU and overwrite the variable

No, doesn't work.

&gt;&gt;&gt; x = x.detach().cpu()
&gt;&gt;&gt; GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |


Attempt 2: Delete the variable

No, this doesn't work either

&gt;&gt;&gt; del x
&gt;&gt;&gt; GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |


Attempt 3: Use the torch.cuda.empty_cache() function

Seems to work, but it seems that there are some lingering overheads...

&gt;&gt;&gt; torch.cuda.empty_cache()
&gt;&gt;&gt; GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |


Attempt 4: Maybe clear the garbage collector.

No, 5% is still being hogged

&gt;&gt;&gt; gc.collect()
0
&gt;&gt;&gt; GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |


Attempt 5: Try deleting torch altogether (as if that would work when del x didn't work -_- )

No, it doesn't...*

&gt;&gt;&gt; del torch
&gt;&gt;&gt; GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |


And then I tried to check gc.get_objects() and it looks like there's still quite a lot of odd THCTensor stuff inside... 

Any idea why is the memory still in use after clearing the cache?

It looks like PyTorch's caching allocator reserves some fixed amount of memory even if there are no tensors, and this allocation is triggered by the first CUDA memory access
(torch.cuda.empty_cache() deletes unused tensor from the cache, but the cache itself still uses some memory).

Even with a tiny 1-element tensor, after del and torch.cuda.empty_cache(), GPUtil.showUtilization(all=True) reports exactly the same amount of GPU memory used as for a huge tensor (and both torch.cuda.memory_cached() and torch.cuda.memory_allocated() return zero).
