title: How to get detailed Nvidia GPU usage?
body: <p>Nvidia-smi only provides a few metrics to measure GPU utilization. Most importantly, <code>utilization.gpu</code> represents the <a href="https://enterprise-support.nvidia.com/s/article/Useful-nvidia-smi-Queries-2" rel="nofollow noreferrer">percent of time over the past sample period during which <strong>one or more</strong> kernels was executing on the GPU</a>. Thus, it seems that a value of 100% does not at all indicate &quot;full&quot; GPU usage.</p>
  <p>Alternatively, <a href="https://docs.nvidia.com/nsight-compute/ProfilingGuide/#metric-collection" rel="nofollow noreferrer">Nsight Compute provides many detailed metrics</a>, but I found it to run very slowly on even small neural networks - it doesn't seem to be the use case. Another option seems to be <a href="https://docs.nvidia.com/deeplearning/frameworks/dlprof-user-guide/index.html" rel="nofollow noreferrer">DLProf</a>, but this again only provides rather granular metrics such as &quot;GPU utilization&quot; and &quot;Tensor Core Efficiency&quot;, whose definitions I could not find.</p>
  <p>Therefore, is there another tool (or parameter) which provides <strong>detailed</strong> GPU usage metrics?</p>