<h3 dir="auto">Environment info</h3>
<ul dir="auto">
<li>Operating System: Ubuntu 14.04,</li>
<li>GPU server with 8 <strong>Tesla K80</strong> GPUs,</li>
<li>Python 2.7 (Anaconda)</li>
</ul>
<p dir="auto">Installed version of CUDA and cuDNN:</p>
<ul dir="auto">
<li>Cuda 7.5, cuDNN v4</li>
</ul>
<p dir="auto">If installed from binary pip package, provide:</p>
<ol dir="auto">
<li>Pip package: GPU-enabled, Python 2.7<br>
<a href="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl</a></li>
<li>The output from <code class="notranslate">python -c "import tensorflow; print(tensorflow.__version__)"</code>:</li>
</ol>
<p dir="auto">I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally<br>
0.9.0</p>
<h3 dir="auto">Steps to reproduce</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
sess = tf.Session()"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
sess = tf.Session()
</code></pre></div>
<p dir="auto">yields the error:<br>
<strong>W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active;</strong><br>
see the detailed logs below please:</p>
<h3 dir="auto">What have you tried?</h3>
<ol dir="auto">
<li>I checked the output of the <code class="notranslate">nvidia-smi</code>. There are other processes run by other users currently. Below is the output.</li>
<li>I re-installed tensorflow again using pip, it didn't solve the problem.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+------------------------------------------------------+                       
| NVIDIA-SMI 352.39     Driver Version: 352.39         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:05:00.0     Off |                    0 |
| N/A   33C    P0    59W / 149W |    190MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:06:00.0     Off |                    0 |
| N/A   30C    P8    30W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0000:09:00.0     Off |                    0 |
| N/A   25C    P8    27W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0000:0A:00.0     Off |                    0 |
| N/A   28C    P8    31W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla K80           Off  | 0000:84:00.0     Off |                    0 |
| N/A   24C    P8    28W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla K80           Off  | 0000:85:00.0     Off |                    0 |
| N/A   54C    P0   106W / 149W |  11461MiB / 11519MiB |     58%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla K80           Off  | 0000:88:00.0     Off |                    0 |
| N/A   49C    P0   134W / 149W |  11461MiB / 11519MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla K80           Off  | 0000:89:00.0     Off |                    0 |
| N/A   69C    P0   131W / 149W |  11462MiB / 11519MiB |     57%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0    155948    C   ../../../build/tools/caffe                      82MiB |
|    0    196261    C   .build_release/test/test_all.testbin            81MiB |
|    5     19542    C   python                                       11436MiB |
|    6     18802    C   python                                       11436MiB |
|    7     85515    C   python                                       11437MiB "><pre class="notranslate"><code class="notranslate">+------------------------------------------------------+                       
| NVIDIA-SMI 352.39     Driver Version: 352.39         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:05:00.0     Off |                    0 |
| N/A   33C    P0    59W / 149W |    190MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:06:00.0     Off |                    0 |
| N/A   30C    P8    30W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0000:09:00.0     Off |                    0 |
| N/A   25C    P8    27W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0000:0A:00.0     Off |                    0 |
| N/A   28C    P8    31W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla K80           Off  | 0000:84:00.0     Off |                    0 |
| N/A   24C    P8    28W / 149W |     22MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla K80           Off  | 0000:85:00.0     Off |                    0 |
| N/A   54C    P0   106W / 149W |  11461MiB / 11519MiB |     58%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla K80           Off  | 0000:88:00.0     Off |                    0 |
| N/A   49C    P0   134W / 149W |  11461MiB / 11519MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla K80           Off  | 0000:89:00.0     Off |                    0 |
| N/A   69C    P0   131W / 149W |  11462MiB / 11519MiB |     57%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0    155948    C   ../../../build/tools/caffe                      82MiB |
|    0    196261    C   .build_release/test/test_all.testbin            81MiB |
|    5     19542    C   python                                       11436MiB |
|    6     18802    C   python                                       11436MiB |
|    7     85515    C   python                                       11437MiB 
</code></pre></div>
<h3 dir="auto">Logs</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:05:00.0
Total memory: 11.25GiB
Free memory: 11.00GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x2f85bd0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:06:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x339e320
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 2 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:09:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x37b5f60
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 3 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:0a:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3bd1860
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 4 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:84:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3ff0cd0
Segmentation fault (core dumped)"><pre class="notranslate"><code class="notranslate">I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:05:00.0
Total memory: 11.25GiB
Free memory: 11.00GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x2f85bd0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:06:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x339e320
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 2 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:09:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x37b5f60
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 3 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:0a:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3bd1860
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 4 with properties: 
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:84:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3ff0cd0
Segmentation fault (core dumped)
</code></pre></div>
<p dir="auto">When I set the nvidia driver in exclusive mode and one of the GPU is already used by another process, I get a segmentation fault:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python -c &quot;import tensorflow as tf;tf.InteractiveSession()&quot;
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 12
Segmentation fault (core dumped)"><pre class="notranslate"><code class="notranslate">$ python -c "import tensorflow as tf;tf.InteractiveSession()"
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 12
Segmentation fault (core dumped)
</code></pre></div>
<p dir="auto">If I limit the visible GPUs to only GPUs that have nothing running on them, it don't segfault:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$CUDA_VISIBLE_DEVICES=1 python -c &quot;import tensorflow as tf;tf.InteractiveSession()&quot;
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 12
I tensorflow/core/common_runtime/gpu/gpu_init.cc:88] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:09:00.0
Total memory: 12.00GiB
Free memory: 11.87GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:112] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:122] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:643] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:09:00.0)
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:47] Setting region size to 12105628263
I tensorflow/core/common_runtime/local_session.cc:45] Local session inter op parallelism threads: 12"><pre class="notranslate"><code class="notranslate">$CUDA_VISIBLE_DEVICES=1 python -c "import tensorflow as tf;tf.InteractiveSession()"
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 12
I tensorflow/core/common_runtime/gpu/gpu_init.cc:88] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:09:00.0
Total memory: 12.00GiB
Free memory: 11.87GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:112] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:122] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:643] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:09:00.0)
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:47] Setting region size to 12105628263
I tensorflow/core/common_runtime/local_session.cc:45] Local session inter op parallelism threads: 12
</code></pre></div>
<p dir="auto">Here is my output of nvidia-smi:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ nvidia-smi 
Wed Nov 11 16:48:27 2015       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.39     Driver Version: 352.39         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 750     Off  | 0000:05:00.0      On |                  N/A |
| N/A   48C    P8     0W /  38W |     25MiB /  2047MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 0000:06:00.0     Off |                  N/A |
| 42%   82C    P2   127W / 250W |    262MiB / 12287MiB |     39%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 0000:09:00.0     Off |                  N/A |
| 22%   46C    P8    17W / 250W |     23MiB / 12287MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 0000:0A:00.0     Off |                  N/A |
| 22%   37C    P8    15W / 250W |    361MiB / 12287MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1171    G   /usr/bin/X                                      17MiB |
|    1     32740    C   python                                         209MiB |
|    3      9429    C   python                                         336MiB |
+-----------------------------------------------------------------------------+
"><pre class="notranslate"><code class="notranslate">$ nvidia-smi 
Wed Nov 11 16:48:27 2015       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.39     Driver Version: 352.39         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 750     Off  | 0000:05:00.0      On |                  N/A |
| N/A   48C    P8     0W /  38W |     25MiB /  2047MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 0000:06:00.0     Off |                  N/A |
| 42%   82C    P2   127W / 250W |    262MiB / 12287MiB |     39%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 0000:09:00.0     Off |                  N/A |
| 22%   46C    P8    17W / 250W |     23MiB / 12287MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 0000:0A:00.0     Off |                  N/A |
| 22%   37C    P8    15W / 250W |    361MiB / 12287MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1171    G   /usr/bin/X                                      17MiB |
|    1     32740    C   python                                         209MiB |
|    3      9429    C   python                                         336MiB |
+-----------------------------------------------------------------------------+

</code></pre></div>
<p dir="auto">I suppose that in the code that check the available GPUs, it don't handle correctly a case when one of the GPUs can't be used.</p>