PyTorch's `torch.cuda.max_memory_allocated()` showing different results from `nvidia-smi`?
I'm currently making my own custom GPU report and am using torch.cuda.max_memory_allocated(device_id) to get the maximum memory that each GPU uses. However, I've noticed that this number is different from when I run nvidia-smi during the process.
According to the documentation for torch.cuda.max_memory_allocated, the output integer is in the form of bytes. And from what I've searched online to convert the number of bytes to the number of gigabytes, you should divide it by 1024 ** 3. I'm currently doing round(max_mem / (1024 ** 3), 2)
Am I doing the calculation wrong, or am I misunderstanding how torch.cuda.max_memory_allocated works entirely? The memory allocated I've observed from one GPU during the entire process was aroudn 32GB, but torch.cuda.max_memory_allocated(0) / (1024 ** 3) returns around 13.5GB.

Posting a link to the same question asked on the discussion forum for PyTorch. TL;DR torch.cuda.max_memory_allocated is not supposed to accurately resemble the output of nvidia-smi because nvidia-smi actually reserves a lot of memory than what is actually being used. Therefore, torch.cuda.max_memory_reserved will resemble the actual output much closely (although still not entirely accurately).
https://discuss.pytorch.org/t/pytorchs-torch-cuda-max-memory-allocated-showing-different-results-from-nvidia-smi/165706/2
