The profiler records all the memory allocation during the profiler interval. Selecting the “Device” will allow you to see each operator’s memory usage on the GPU side or host side. You must enable profile_memory=True to generate the below memory data as shown here. 
With torch.profiler.profile(
Profiler_memory=True # this will take 1 – 2 minutes to complete. 
)

Important Definitions:
•   “Size Increase” displays the sum of all allocation bytes and minus all the memory release bytes.
•   “Allocation Size” shows the sum of all allocation bytes without considering the memory release.
•   “Self” means the allocated memory is not from any child operators, instead by the operator itself.



GPU Metric on Timeline: