title: How is GPU and memory utilization defined in nvidia-smi results?
body: <p>I am currently using a tool shipped with nvidia's driver 'nvidia-smi' for performance monitoring on GPU. When we use 'nvidia-smi -a', it will give the information of current GPU information, including GPU core and memory usage, temperature and so on like this:</p>
  <blockquote>
  <p>==============NVSMI LOG==============</p>
  <p>Timestamp                       : Tue</p>
  <p>Feb 22 22:39:09 2011</p>
  <p>Driver Version                  :
  260.19.26</p>
  <p>GPU 0:</p>
  <pre><code>    Product Name            : GeForce 8800 GTX
      PCI Device/Vendor ID    : 19110de
      PCI Location ID         : 0:4:0
      Board Serial            : 211561763875
      Display                 : Connected
      Temperature             : 55 C
      Fan Speed               : 47%
      Utilization
          GPU                 : 1%
          Memory              : 0%
  </code></pre>
  </blockquote>
  <p>I am curious about how are the GPU and memory Utilization defined? For example, GPU core's utilization is 47%. It means there are 47% of SMs active working? Or all the GPU cores are busy in 47% time while idle other 53% time? For memory, the utilization stands for the ratio between current bandwidth and max bandwidth, or the busy time ratio in last time unit?</p>