
Augmented Counters (Memory bandwidth, Queue length): Augmented trace files which provide insights into memory copy bandwidth and number of outstanding operations on each CUDA stream.
Patterns
Frequent CUDA Kernels: Find the CUDA kernels most frequently launched by any given PyTorch or user defined operator.


Trace Comparison
Trace Diff: A trace comparison tool to identify and visualize the differences between traces.



HTA source code is available to users via Github. Users can request new features or build their own analysis using the core libraries and data structures provided in the codebase in addition to the features mentioned above.
GPU Training Performance Debugging 101
To understand the GPU performance in distributed training jobs, we consider how the model operators interact with the GPU devices and how such interactions are reflected in certain measurable metrics.