torch.profiler.ProfilerActivity.CUDA], 
```
•   With profiler.record_function(“$NAME”): allows putting a decorator (a tag associated to a name) for a block of function
•   Profile_memory=True parameter under profiler.profile allows you to profile CPU and GPU memory footprint
Visualizing PyTorch Model Performance using PyTorch Profiler
Distributed Training
Recent advances in deep learning argue for the value of large datasets and large models, which requires you to scale out model training to more computational resources. Distributed Data Parallel (DDP) and NVIDIA Collective Communications Library (NCCL) are the widely adopted paradigms in PyTorch for accelerating your deep learning training. 
In this release of PyTorch Profiler, DDP with NCCL backend is now supported.



Computation/Communication Overview