expensive under regular CPU-mode profilers. In these case where
  timings are incorrect, the CUDA-mode autograd profiler may be
  helpful.
Note:
To decide which (CPU-only-mode or CUDA-mode) autograd profiler
  output to look at, you should first check if your script is CPU-
  bound ("CPU total time is much greater than CUDA total time"). If it
  is CPU-bound, looking at the results of the CPU-mode autograd
  profiler will help. If on the other hand your script spends most of
  its time executing on the GPU, then it makes sense to start looking
  for responsible CUDA operators in the output of the CUDA-mode
  autograd profiler.Of course the reality is much more complicated and
  your script might not be in one of those two extremes depending on
  the part of the model you're evaluating. If the profiler outputs
  don't help, you could try looking at the result of
  "torch.autograd.profiler.emit_nvtx()" with "nvprof". However, please
  take into account that the NVTX overhead is very high and often