Why does a single Conv2d with 10x10x3 take up 850mb of gpu
In Pytorch I am optimizing a model. If I run the following code, nvidia-smi shows that I am using  850MiB /  7979MiB of memory on my gpu. Why would this be the case?


  with torch.no_grad():
      A = nn.Conv2d(10,10,3).cuda()


I imagine there are some overhead or a default allocation size specified somewhere but I could not find such documentation. I do recall that tensorflow had a setting to limit the amount of memory allocated.

Related Git Issue

The convolution does not occupy that much memory. You can verify this with torch.cuda.memory_allocated, which shows the memory that is occupied by all tensors in bytes:

torch.cuda.memory_allocated() # =&gt; 0

A = nn.Conv2d(10,10,3).cuda()

torch.cuda.memory_allocated() # =&gt; 4608


The convolution only uses 4608 bytes.

nvidia-smi shows higher memory usage for two separate reasons.

Caching Memory Allocator

PyTorch uses a caching memory allocator, meaning that it holds onto more memory than necessary to avoid device synchronisations.

From PyTorch CUDA Semantics - Memory Management:


  PyTorch uses a caching memory allocator to speed up memory allocations. This allows fast memory deallocation without device synchronizations. However, the unused memory managed by the allocator will still show as if used in nvidia-smi. You can use memory_allocated() and max_memory_allocated() to monitor memory occupied by tensors, and use memory_reserved() and max_memory_reserved() to monitor the total amount of memory managed by the caching allocator.


CUDA Context

When CUDA is first initialised, it creates a context that manages the control of the device. Most notably, the context contains the code of all the different CUDA kernels, of which PyTorch has a large number. The size of the context also varies across different GPU architectures. Some details are discussed in Issue #20532 - Couple hundred MB are taken just by initializing cuda .

The memory you are observing is almost exclusively attributed to the CUDA context.
