<h3 dir="auto">ERROR message</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR: /csproject/dygroup2/czeng/downloads/tensorflow/tensorflow/core/kernels/BUILD:1575:1: undeclared inclusion(s) in rule '//tensorflow/core/kernels:training_ops_gpu':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/kernels/training_ops_gpu.cu.cc':"><pre class="notranslate"><code class="notranslate">ERROR: /csproject/dygroup2/czeng/downloads/tensorflow/tensorflow/core/kernels/BUILD:1575:1: undeclared inclusion(s) in rule '//tensorflow/core/kernels:training_ops_gpu':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/kernels/training_ops_gpu.cu.cc':
</code></pre></div>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System:<br>
CentOS 6.5<br>
Installed version of CUDA and cuDNN:<br>
cuda-7.5<br>
cuDNN v5</p>
<h3 dir="auto">Procedure to reproduce:</h3>
<p dir="auto">I used a custom GCC version (which is in /usr/local/GNU/gcc-4.9.2/) following the guide in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="151104976" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/2109" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/2109/hovercard" href="https://github.com/tensorflow/tensorflow/issues/2109">#2109</a> , and executed</p>
<p dir="auto"><code class="notranslate">bazel build --verbose_failures -c opt --config=cuda --genrule_strategy=standalone -j 10 //tensorflow/cc:tutorials_example_trainer </code></p>
<h3 dir="auto">Output of $ echo | gcc4 -E -xc++ - -v</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Using built-in specs.
COLLECT_GCC=gcc4
Target: x86_64-unknown-linux-gnu
Configured with: ../gcc-4.9.2/configure --prefix=/usr/local/GNU/gcc-4.9.2 --enable-clocale=generic
Thread model: posix
gcc version 4.9.2 (GCC)
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.2/cc1plus -E -quiet -v -iprefix /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/ -D_GNU_SOURCE - -mtune=generic -march=x86-64
ignoring nonexistent directory &quot;/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../x86_64-unknown-linux-gnu/include&quot;
ignoring duplicate directory &quot;/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2&quot;
ignoring duplicate directory &quot;/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/x86_64-unknown-linux-gnu&quot;
ignoring duplicate directory &quot;/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/backward&quot;
ignoring duplicate directory &quot;/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include&quot;
ignoring duplicate directory &quot;/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include-fixed&quot;
ignoring nonexistent directory &quot;/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../x86_64-unknown-linux-gnu/include&quot;
#include &quot;...&quot; search starts here:
#include &lt;...&gt; search starts here:
 /csproject/dygroup2/czeng/venv/cudnnv5/include
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/x86_64-unknown-linux-gnu
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/backward
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include-fixed
 /usr/local/include
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../include
 /usr/include
End of search list.
#1 &quot;&lt;stdin&gt;&quot;
#1 &quot;&lt;built-in&gt;&quot;
#1 &quot;&lt;command-line&gt;&quot;
#1 &quot;&lt;stdin&gt;&quot;
COMPILER_PATH=/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.2/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/
LIBRARY_PATH=/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/:/csproject/dygroup2/czeng/venv/cudnnv5/lib64/../lib64/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/csproject/dygroup2/czeng/venv/cudnnv5/lib64/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'"><pre class="notranslate"><code class="notranslate">Using built-in specs.
COLLECT_GCC=gcc4
Target: x86_64-unknown-linux-gnu
Configured with: ../gcc-4.9.2/configure --prefix=/usr/local/GNU/gcc-4.9.2 --enable-clocale=generic
Thread model: posix
gcc version 4.9.2 (GCC)
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.2/cc1plus -E -quiet -v -iprefix /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/ -D_GNU_SOURCE - -mtune=generic -march=x86-64
ignoring nonexistent directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../x86_64-unknown-linux-gnu/include"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/x86_64-unknown-linux-gnu"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/backward"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include-fixed"
ignoring nonexistent directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../x86_64-unknown-linux-gnu/include"
#include "..." search starts here:
#include &lt;...&gt; search starts here:
 /csproject/dygroup2/czeng/venv/cudnnv5/include
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/x86_64-unknown-linux-gnu
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/backward
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include-fixed
 /usr/local/include
 /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../include
 /usr/include
End of search list.
#1 "&lt;stdin&gt;"
#1 "&lt;built-in&gt;"
#1 "&lt;command-line&gt;"
#1 "&lt;stdin&gt;"
COMPILER_PATH=/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.2/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/
LIBRARY_PATH=/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/:/csproject/dygroup2/czeng/venv/cudnnv5/lib64/../lib64/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/csproject/dygroup2/czeng/venv/cudnnv5/lib64/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
</code></pre></div>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Ubuntu 16.04,  GeForce GTX TITAN X</p>
<p dir="auto">Installed version of CUDA and cuDNN:<br>
(please attach the output of <code class="notranslate">ls -l /path/to/cuda/lib/libcud*</code>):<br>
ls /usr/local/cuda-8.0/lib64/libcud*<br>
/usr/local/cuda-8.0/lib64/libcudadevrt.a<br>
/usr/local/cuda-8.0/lib64/libcudart.so<br>
/usr/local/cuda-8.0/lib64/libcudart.so.8.0<br>
/usr/local/cuda-8.0/lib64/libcudart.so.8.0.44<br>
/usr/local/cuda-8.0/lib64/libcudart_static.a<br>
/usr/local/cuda-8.0/lib64/libcudnn.so<br>
/usr/local/cuda-8.0/lib64/libcudnn.so.5<br>
/usr/local/cuda-8.0/lib64/libcudnn.so.5.1.10<br>
/usr/local/cuda-8.0/lib64/libcudnn.so.5.1.5<br>
/usr/local/cuda-8.0/lib64/libcudnn_static.a</p>
<h3 dir="auto">If installed from binary pip package, provide:</h3>
<ol dir="auto">
<li>Installed with pip tensorflow-gpu</li>
<li>~ python -c "import tensorflow; print(tensorflow.<strong>version</strong>)<br>
1.0.0</li>
</ol>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<p dir="auto">Using the provided tf.contrib.learn model in get_started/get_started documentation</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import numpy as np

features = [tf.contrib.layers.real_valued_column(&quot;x&quot;, dimension=1)]
estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)
x = np.array([1., 2., 3., 4.])
y = np.array([0., -1., -2., -3.])
input_fn = tf.contrib.learn.io.numpy_input_fn({&quot;x&quot;:x}, y, batch_size=4, num_epochs=1000)
estimator.fit(input_fn=input_fn, steps=10)
"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
import numpy as np

features = [tf.contrib.layers.real_valued_column("x", dimension=1)]
estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)
x = np.array([1., 2., 3., 4.])
y = np.array([0., -1., -2., -3.])
input_fn = tf.contrib.learn.io.numpy_input_fn({"x":x}, y, batch_size=4, num_epochs=1000)
estimator.fit(input_fn=input_fn, steps=10)

</code></pre></div>
<p dir="auto">The Core model works fine.</p>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">(If logs are large, please upload as attachment or provide link).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python test-oom.py 
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpew3U6z
WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.
WARNING:tensorflow:From /home/gajop/projekti/ubuntu-ranking-dataset-creator/env/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 11.92GiB
Free memory: 11.52GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 11.92G (12799180800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY"><pre class="notranslate"><code class="notranslate">python test-oom.py 
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpew3U6z
WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.
WARNING:tensorflow:From /home/gajop/projekti/ubuntu-ranking-dataset-creator/env/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 11.92GiB
Free memory: 11.52GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 11.92G (12799180800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
</code></pre></div>