title: CUDA: why is there a large amount of GPU idle time?
body: <p><strong>Question</strong> </p>
  
  <p>total GPU time + total CPU overhead is smaller than the total execution time. Why?</p>
  
  <p><strong>Detail</strong></p>
  
  <p>I am studying how frequent global memory access and kernel launch may affect the performance and I have designed a code which has multiple small kernels and ~0.1 million kernel calls in total. Each kernel reads data from global memory, processes them and then writes back to the global memory. As expected, the code runs much slower than the original design which has only one large kernel and very few kernel launches.</p>
  
  <p>The problem arose as I used command line profiler to get <strong>"gputime"</strong> (execution time for the GPU kernel or memory copy method) and <strong>"cputime"</strong> (CPU overhead for non-blocking method, the sum of gputime and CPU overhead for blocking method ). To my understanding, the sum of all gputimes and all cputimes should exceed the entire execution time (the last <strong>"gpuendtimestamp"</strong> minus the first <strong>"gpustarttimestamp"</strong>), but it turns out the contrary is true (sum of gputimes=13.835064 s,
  sum of cputimes=4.547344 s, total time=29.582793). Between the end of one kernel and the start of the next, there is often a large amount of waiting time, larger than the CPU overhead of the next kernel. Most of the kernels suffer from this problem are: memcpyDtoH, memcpyDtoD and thrust internel functions such as launch_closure_by_value, fast_scan, etc. What is the probable reason?</p>
  
  <p><strong>System</strong>
  Windows 7, TCC driver, VS 2010, CUDA 4.2 </p>
  
  <p>Thanks for your help!</p>