title: does tensorflow take all resource from GPU, making other CUDA code slow?
body: <p><strong>System information</strong></p>
  
  <p>OS Platform and Distribution: Linux Ubuntu 16.04</p>
  
  <p>TensorFlow version: tensorflow-gpu (1.7.0)</p>
  
  <p>Python version: Python 3.5.2</p>
  
  <p>CUDA/cuDNN version: CUDA 9.0 cuDNN 7</p>
  
  <p><strong>Describe the problem</strong></p>
  
  <p>I have a cuda lib build from C++ for post-processing after predict result by tensorflow model.
  I use following way to make python able to use cuda code from C++</p>
  
  <pre><code>lib = ctypes.cdll.LoadLibrary(my.so)
  result = lib.post_process(tensorflow_result)
  </code></pre>
  
  <p>If I test the cuda code alone without tensorflow, it work fine. (I save the result from tensorflow then use cv2.imread to feed into my cuda code)</p>
  
  <p>But when tensorflow is used in my project, my cuda code become 10 times slower....
  My time log is in cuda .so lib, so it's no way that the gap come from python to .so wrap.
  I have try to set the fraction of GPU memory to be allocated in tensorflow by:</p>
  
  <pre><code># Assume that you have 12GB of GPU memory and want to allocate ~4GB:
  gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
  sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
  </code></pre>
  
  <p>but useless....
  so I wonder does tensorflow take all resource from GPU, making other CUDA code slow ?
  the only solution is make my cuda code as a tensorflow OP by register?</p>
  
  <p>Any suggestion? Thanks~~~</p>
  
  <p>----------------------Update----------------------<br>
  I have tested what @AnandCU say.  </p>
  
  <pre><code>config = tf.ConfigProto()
  config.gpu_options.allow_growth = True
  sess = tf.Session(config=config)
  </code></pre>
  
  <p>but it doesn't make my cuda code speed up like I test it alone without tensorflow.</p>