title: How to measure ONLY the inference time in the GPU, using TensorRT and PyCUDA?
body: <p>I want to measure ONLY the inference time in the Jetson TX2. How can I improve my function to do that? As right now I am measuring:</p>
  <ul>
  <li><p>the transfer of the image from CPU to GPU</p>
  </li>
  <li><p>transfer of results from GPU to CPU</p>
  </li>
  <li><p>the inference</p>
  </li>
  </ul>
  <p>Or is that not possible because of the way GPUs work? I mean, how many times will I have to use <code>stream.synchronize()</code> if I divide/segment the function into 3 parts:</p>
  <ol>
  <li>transfer from CPU to GPU</li>
  <li>Inference</li>
  <li>transfer from GPU to CPU</li>
  </ol>
  <p>Thank you</p>
  <p>CODE IN INFERENCE.PY</p>
  <pre><code>def do_inference(engine, pics_1, h_input, d_input, h_output, d_output, stream, batch_size):
  
      &quot;&quot;&quot;
      This is the function to run the inference
      Args:
        engine : Path to the TensorRT engine. 
        pics_1 : Input images to the model.  
        h_input: Input in the host (CPU). 
        d_input: Input in the device (GPU). 
        h_output: Output in the host (CPU). 
        d_output: Output in the device (GPU). 
        stream: CUDA stream.
        batch_size : Batch size for execution time.
        height: Height of the output image.
        width: Width of the output image.
      
      Output:
        The list of output images.
  
      &quot;&quot;&quot;
        
      # Context for executing inference using ICudaEngine
      with engine.create_execution_context() as context:
          
          # Transfer input data from CPU to GPU.
          cuda.memcpy_htod_async(d_input, h_input, stream)
  
          # Run inference.
          #context.profiler = trt.Profiler() ##shows execution time(ms) of each layer
          context.execute(batch_size=1, bindings=[int(d_input), int(d_output)])
  
          # Transfer predictions back from the GPU to the CPU.
          cuda.memcpy_dtoh_async(h_output, d_output, stream)
          
          # Synchronize the stream.
          stream.synchronize()
          
          # Return the host output.
          out = h_output       
          return out
  </code></pre>
  <p>CODE IN TIMER.PY</p>
  <pre><code>for i in range (count):
      start = time.perf_counter()
      # Classification - calling TX2_classify.py
      out = eng.do_inference(engine, image, h_input, d_input, h_output, d_output, stream, 1) 
      inference_time = time.perf_counter() - start
      print(&quot;TIME&quot;)
      print(inference_time * 1000)
      print(&quot;\n&quot;)
      pred = postprocess_inception(out)
      print(pred)
      print(&quot;\n&quot;)
  </code></pre>