title: Why can't I run tensorflow session on CPU while one GPU device's memory is all allocated?
body: <p>From the tensorflow website (<a href="https://www.tensorflow.org/guide/using_gpu" rel="nofollow noreferrer">https://www.tensorflow.org/guide/using_gpu</a>) I found the following code to manually specify the use of a CPU instead of a GPU:</p>
  
  <pre><code># Creates a graph.
  with tf.device('/cpu:0'):
      a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
      b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
      c = tf.matmul(a, b)
  # Creates a session with log_device_placement set to True.
  sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
  # Runs the op.
  print(sess.run(c))
  </code></pre>
  
  <p>I tried running this on my machine (with 4 GPUs) and got the following error:</p>
  
  <pre><code>2018-11-05 10:02:30.636733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: 
  name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
  pciBusID: 0000:18:00.0
  totalMemory: 10.92GiB freeMemory: 10.76GiB
  2018-11-05 10:02:30.863280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 1 with properties: 
  name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
  pciBusID: 0000:3b:00.0
  totalMemory: 10.92GiB freeMemory: 10.76GiB
  2018-11-05 10:02:31.117729: E tensorflow/core/common_runtime/direct_session.cc:158] Internal: failed initializing StreamExecutor for CUDA device ordinal 2: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY; total memory reported: 11721506816
  Traceback (most recent call last):
    File "./tf_test.py", line 10, in &lt;module&gt;
      sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
    File ".../anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1566, in __init__
      super(Session, self).__init__(target, graph, config=config)
    File ".../anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 636, in __init__
      self._session = tf_session.TF_NewSession(self._graph._c_graph, opts)
  tensorflow.python.framework.errors_impl.InternalError: Failed to create session.
  </code></pre>
  
  <p>It seems that when I create the session, tensorflow tries to initialize a stream executor on all devices.  Unfortunately, one of the GPUs is being used by my colleague right now.  I would hope that his full use of one GPU would not prevent me from using another device (whether GPU or CPU), but this does not seem to be the case.</p>
  
  <p>Does anyone know a workaround to this?  Perhaps something to add to the config?  Is this something that could be fixed in tensorflow?</p>
  
  <p>FYI... here is the output of "gpustat -upc":</p>
  
  <pre><code>&lt;my_hostname&gt;  Mon Nov  5 10:19:47 2018
  [0] GeForce GTX 1080 Ti | 36'C,   0 % |    10 / 11178 MB |
  [1] GeForce GTX 1080 Ti | 41'C,   0 % |    10 / 11178 MB |
  [2] GeForce GTX 1080 Ti | 38'C,   0 % | 11097 / 11178 MB | &lt;my_colleague&gt;:python2/148901(11087M)
  [3] GeForce GTX 1080 Ti | 37'C,   0 % |    10 / 11178 MB |
  </code></pre>