title: How to run TensorRT in multiple threads?
body: <p>I am new to TensorRT and CUDA and I am trying to implement an inference server using TensorRT Python API. I follow the <code>end_to_end_tensorflow_mnist</code> and <code>uff_ssd</code> example and everything works ok. However, when I try to use the engine to make inference in multiple threads, I encounter some problems. So I was wondering what is the right way to run TensorRT in multiple threads. </p>
  
  <p>Here's what i have tried. First I create inference engine in the main thread. In the worker thread, I allocate memory space, CUDA Stream and execution context using the engine created in the main thread and make inference: </p>
  
  <pre><code>import pycuda.autoinit  # Create CUDA context
  import pycuda.driver as cuda 
  
  # Main thread
  with open(“sample.engine”, “rb”) as f, trt.Runtime(TRT_LOGGER) as runtime:
      engine = runtime.deserialize_cuda_engine(f.read())
  
  ...
  
  # Worker thread
  with engine.create_execution_context() as context:
      inputs, outputs, bindings, stream = common.allocate_buffers(engine)
      common.do_inference(context, inputs, outputs, bindings, stream)
  
  </code></pre>
  
  <p>The above code produce the following error:</p>
  
  <pre><code>pycuda._driver.LogicError: explicit_context_dependent failed: invalid device context - no currently active context?
  </code></pre>
  
  <p>This sounds like there is no active CUDA context in the worker thread. So, I tried to create CUDA context manually in the worker thread:</p>
  
  <pre><code># Worker thread
  from pycuda.tools import make_default_context()
  cuda.init()  # Initialize CUDA
  ctx = make_default_context()  # Create CUDA context
  
  with engine.create_execution_context() as context:
      inputs, outputs, bindings, stream = common.allocate_buffers(engine)
      common.do_inference(context, inputs, outputs, bindings, stream)
  
  ctx.pop()  # Clean up
  </code></pre>
  
  <p>This time, it gives me another error:</p>
  
  <pre><code>[TensorRT] ERROR: cuda/cudaConvolutionLayer.cpp (163) - Cudnn Error in execute: 7
  [TensorRT] ERROR: cuda/cudaConvolutionLayer.cpp (163) - Cudnn Error in execute: 7
  </code></pre>
  
  <p>I understand the builder or runtime will be created with the GPU context associated with the creating thread. I guess this error is because the engine is associated with main thread but I use it in the worker thread, so my question is:</p>
  
  <ol>
  <li>Is that means I have to rebuild the engine in worker thread, which significantly increase inference overheat?</li>
  <li>Can I share GPU context between main thread and worker threads so that I don't have to create a new GPU context for each new request? If so, how to do that in pycuda?</li>
  </ol>
  
  <p>Any advice will be appreciated. Thanks!</p>