title: caffe multi-gpu CUDNN_STATUS_EXECUTION_FAILED
body: <p>I`m working in a c++ project (VS2105) in windows environment that launches multiple threads. PC has two NVIDIA GeForce GTX 1080. Each thread uses a different caffe classifier in GPU mode.</p>
  
  <p>If all threads use one GPU, the program is working well. It works well both using GPU 0 or using GPU1. The problem comes when I use a different GPU for each thread. For example, if the program launches two treads and classifier in thread 1 uses GPU 0 (<code>Caffe::SetDevice(0);</code>) and classifier in thread 2 uses GPU 1(<code>Caffe::SetDevice(1);</code>), prediction starts working but suddenly I get the error  <code>cudnn_conv_layer.cu:28] Check failed: status == CUDNN_STATUS_SUCCESS (8 vs. 0)  CUDNN_STATUS_EXECUTION_FAILED</code></p>
  
  <p>I have seen similar errors from people working in multithreaded projects with python interface related to CuDNN thread safe issues working with one GPU, but in single GPU the program works fine. </p>
  
  <p>My models are trained on single GPU. Should I train separate models in GPU 0 and in GPU 1?. Should I configure classifiers in any special way to use it in multiGPU multithread environement?.</p>
  
  <p>Im using windows branch of caffe with cuda 8.0 and using c++ interface of caffe linked as static library to the project.</p>