<p dir="auto">Operating System:<br>
Ubuntu 16.04</p>
<p dir="auto">Installed version of CUDA and cuDNN:<br>
server@server:~$ ls -l /usr/local/cuda/lib64/libcud*<br>
-rw-r--r-- 1 root root 560184 9月   7 09:56 /usr/local/cuda/lib64/libcudadevrt.a<br>
lrwxrwxrwx 1 root root     16 9月   7 09:56 /usr/local/cuda/lib64/libcudart.so -&gt; libcudart.so.8.0<br>
lrwxrwxrwx 1 root root     19 9月   7 09:56 /usr/local/cuda/lib64/libcudart.so.8.0 -&gt; libcudart.so.8.0.27<br>
-rwxr-xr-x 1 root root 394472 9月   7 09:56 /usr/local/cuda/lib64/libcudart.so.8.0.27<br>
-rw-r--r-- 1 root root 737516 9月   7 09:56 /usr/local/cuda/lib64/libcudart_static.a</p>
<p dir="auto">The commit hash (<code class="notranslate">git rev-parse HEAD</code>)<br>
server@server:~/tensorflow$ git rev-parse HEAD<br>
<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/2ab7e6326296987ea0ce975afb3434a16d1aa21a/hovercard" href="https://github.com/tensorflow/tensorflow/commit/2ab7e6326296987ea0ce975afb3434a16d1aa21a"><tt>2ab7e63</tt></a></p>
<p dir="auto">The output of <code class="notranslate">bazel version</code><br>
server@server:~$ bazel version<br>
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar<br>
Build time: Thu Jan 01 00:00:00 1970 (0)<br>
Build timestamp: Thu Jan 01 00:00:00 1970 (0)<br>
Build timestamp as int: 0</p>
<p dir="auto">Here is my bazel build result:<br>
INFO: From Compiling tensorflow/core/kernels/tile_ops_gpu.cu.cc:<br>
nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.<br>
nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.<br>
INFO: From Compiling tensorflow/core/kernels/tile_ops_gpu.cu.cc:<br>
nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.<br>
nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.<br>
Target //tensorflow/cc:tutorials_example_trainer up-to-date:<br>
bazel-bin/tensorflow/cc/tutorials_example_trainer<br>
INFO: Elapsed time: 985.075s, Critical Path: 952.03s</p>
<p dir="auto">Here is my bazel-bin result:<br>
000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000007/000009 lambda = 39.000000 x = [3.605551 -0.000000] y = [10.816654 -3.605551]<br>
000006/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000001/000002 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000002/000006 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000006/000008 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000004/000004 lambda = 1.200000 x = [-0.894427 0.447214] y = [-0.894427 0.894427]<br>
000006/000007 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000005/000005 lambda = 249281724416.000000 x = [0.894427 -0.447214] y = [278705438720.000000 -130.000000]<br>
000008/000003 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000007/000009 lambda = 2.400000 x = [0.948683 -0.316228] y = [2.213594 -0.948683]<br>
000008/000004 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000003/000009 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000000/000004 lambda = 2.255962 x = [0.930408 -0.366524] y = [2.058176 -0.930408]<br>
000001/000002 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000004/000004 lambda =      nan x = [     nan 0.707107] y = [     nan      nan]<br>
000008/000000 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000003/000008 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000006/000007 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000007/000009 lambda = 2.172414 x = [0.919145 -0.393919] y = [1.969597 -0.919145]<br>
000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000008/000003 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000002/000006 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000000/000004 lambda = 2.115613 x = [0.911220 -0.411921] y = [1.909816 -0.911220]<br>
000003/000008 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000008/000004 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000003/000009 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000006/000006 lambda = 1786627917493567488.000000 x = [715827904.000000 348406848.000000] y = [2844297216.000000 -715827904.000000]<br>
000008/000000 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000007/000009 lambda = 2.080292 x = [0.906183 -0.422885] y = [1.872779 -0.906183]<br>
000008/000003 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000003/000008 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000008/000004 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000006/000007 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000004/000004 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000005/000005 lambda = 3.000000 x = [1.000000 -0.000000] y = [3.000000 -1.000000]<br>
000003/000009 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000008/000000 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]<br>
000006/000006 lambda = 2.584623 x = [0.969760 -0.244061] y = [2.421158 -0.969760]<br>
000004/000004 lambda =      nan x = [     nan      nan] y = [     nan      nan]<br>
000007/000009 lambda = 2.038786 x = [0.900159 -0.435561] y = [1.829356 -0.900159]</p>
<p dir="auto">I've searched the nan problem, according to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="149814648" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/2037" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/2037/hovercard" href="https://github.com/tensorflow/tensorflow/issues/2037">#2037</a>, i've tried "--num_concurrent_sessions=1" and "--num_concurrent_steps=1", and it seems to be fine<br>
Here is the output:<br>
server@server:~/tensorflow$ bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu --num_concurrent_sessions=1 --num_concurrent_steps=1<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:105] Couldn't open CUDA library libcudnn.so.5. LD_LIBRARY_PATH: /usr/local/cuda-8.0/lib64:<br>
I tensorflow/stream_executor/cuda/cuda_dnn.cc:3304] Unable to load cuDNN DSO<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally<br>
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:<br>
name: GeForce GTX TITAN X<br>
major: 5 minor: 2 memoryClockRate (GHz) 1.076<br>
pciBusID 0000:02:00.0<br>
Total memory: 11.92GiB<br>
Free memory: 11.40GiB<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y<br>
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0)<br>
000000/000000 lambda = 1.841570 x = [0.669396 0.742906] y = [3.493999 -0.669396]<br>
000000/000000 lambda = 2.708984 x = [0.982138 -0.188162] y = [2.570089 -0.982138]<br>
000000/000000 lambda = 2.284280 x = [0.934118 -0.356965] y = [2.088423 -0.934118]<br>
000000/000000 lambda = 2.127152 x = [0.912847 -0.408302] y = [1.921937 -0.912847]<br>
000000/000000 lambda = 2.060266 x = [0.903291 -0.429029] y = [1.851815 -0.903291]<br>
000000/000000 lambda = 2.029356 x = [0.898775 -0.438410] y = [1.819504 -0.898775]<br>
000000/000000 lambda = 2.014491 x = [0.896580 -0.442881] y = [1.803979 -0.896580]<br>
000000/000000 lambda = 2.007199 x = [0.895499 -0.445064] y = [1.796367 -0.895499]<br>
000000/000000 lambda = 2.003588 x = [0.894962 -0.446143] y = [1.792599 -0.894962]<br>
000000/000000 lambda = 2.001791 x = [0.894694 -0.446679] y = [1.790724 -0.894694]<br>
000000/000000 lambda = 2.000895 x = [0.894561 -0.446947] y = [1.789788 -0.894561]<br>
000000/000000 lambda = 2.000447 x = [0.894494 -0.447080] y = [1.789321 -0.894494]<br>
000000/000000 lambda = 2.000224 x = [0.894460 -0.447147] y = [1.789088 -0.894460]<br>
000000/000000 lambda = 2.000112 x = [0.894444 -0.447180] y = [1.788971 -0.894444]<br>
000000/000000 lambda = 2.000056 x = [0.894436 -0.447197] y = [1.788913 -0.894436]<br>
000000/000000 lambda = 2.000028 x = [0.894431 -0.447205] y = [1.788883 -0.894431]<br>
000000/000000 lambda = 2.000014 x = [0.894429 -0.447209] y = [1.788869 -0.894429]</p>
<p dir="auto">When I build the pip-packages. i got another error.<br>
Here is the output:<br>
server@server:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package<br>
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.<br>
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.<br>
INFO: Elapsed time: 0.037s</p>
<p dir="auto">Where am I wrong?</p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System:<br>
OS  10.10.5</p>
<p dir="auto">Installed version of CUDA and cuDNN:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ ls -l /usr/local/cuda/lib/libcud*
-rwxr-xr-x  1 root           wheel      8280 Apr 13 01:02 /usr/local/cuda/lib/libcuda.dylib
lrwxr-xr-x  1 root           wheel        45 Apr 13 01:03 /usr/local/cuda/lib/libcudadevrt.a -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudadevrt.a
lrwxr-xr-x  1 root           wheel        50 Apr 13 01:03 /usr/local/cuda/lib/libcudart.7.5.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudart.7.5.dylib
lrwxr-xr-x  1 root           wheel        46 Apr 13 01:03 /usr/local/cuda/lib/libcudart.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudart.dylib
lrwxr-xr-x  1 root           wheel        49 Apr 13 01:03 /usr/local/cuda/lib/libcudart_static.a -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudart_static.a
-rwxr-xr-x@ 1 production204  staff  60108616 Feb  8  2016 /usr/local/cuda/lib/libcudnn.4.dylib
lrwxr-xr-x  1 root           admin        47 Aug 29 18:08 /usr/local/cuda/lib/libcudnn.5.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.5.dylib
lrwxr-xr-x  1 root           admin        45 Aug 29 18:08 /usr/local/cuda/lib/libcudnn.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.dylib
-rw-r--r--@ 1 production204  staff  59311504 Feb  8  2016 /usr/local/cuda/lib/libcudnn_static.a"><pre class="notranslate"><code class="notranslate">$ ls -l /usr/local/cuda/lib/libcud*
-rwxr-xr-x  1 root           wheel      8280 Apr 13 01:02 /usr/local/cuda/lib/libcuda.dylib
lrwxr-xr-x  1 root           wheel        45 Apr 13 01:03 /usr/local/cuda/lib/libcudadevrt.a -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudadevrt.a
lrwxr-xr-x  1 root           wheel        50 Apr 13 01:03 /usr/local/cuda/lib/libcudart.7.5.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudart.7.5.dylib
lrwxr-xr-x  1 root           wheel        46 Apr 13 01:03 /usr/local/cuda/lib/libcudart.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudart.dylib
lrwxr-xr-x  1 root           wheel        49 Apr 13 01:03 /usr/local/cuda/lib/libcudart_static.a -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudart_static.a
-rwxr-xr-x@ 1 production204  staff  60108616 Feb  8  2016 /usr/local/cuda/lib/libcudnn.4.dylib
lrwxr-xr-x  1 root           admin        47 Aug 29 18:08 /usr/local/cuda/lib/libcudnn.5.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.5.dylib
lrwxr-xr-x  1 root           admin        45 Aug 29 18:08 /usr/local/cuda/lib/libcudnn.dylib -&gt; /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.dylib
-rw-r--r--@ 1 production204  staff  59311504 Feb  8  2016 /usr/local/cuda/lib/libcudnn_static.a
</code></pre></div>
<ol dir="auto">
<li>The output from <code class="notranslate">python -c "import tensorflow; print(tensorflow.__version__)"</code>.<br>
(can't get that far, but i'm using 0.10)</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import tensorflow
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
Segmentation fault: 11
"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; import tensorflow
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
Segmentation fault: 11

</code></pre></div>
<p dir="auto">If installed from source, provide</p>
<ol dir="auto">
<li>The commit hash (<code class="notranslate">git rev-parse HEAD</code>)</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="4c49dbebef05442c7e72d6129a30574fcd13f0e1"><pre class="notranslate"><code class="notranslate">4c49dbebef05442c7e72d6129a30574fcd13f0e1
</code></pre></div>
<ol dir="auto">
<li>The output of <code class="notranslate">bazel version</code></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ bazel version
Build label: 0.3.1-homebrew
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Aug 4 09:59:58 2016 (1470304798)
Build timestamp: 1470304798
Build timestamp as int: 1470304798"><pre class="notranslate"><code class="notranslate">$ bazel version
Build label: 0.3.1-homebrew
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Aug 4 09:59:58 2016 (1470304798)
Build timestamp: 1470304798
Build timestamp as int: 1470304798
</code></pre></div>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.
INFO: Elapsed time: 0.076s
"><pre class="notranslate"><code class="notranslate">$ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.
INFO: Elapsed time: 0.076s

</code></pre></div>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<ul dir="auto">
<li>Downgrading to cuDNN4, switching between 4 and 5</li>
<li>Re-installing bazel</li>
<li>Modifying CROSSTOOL file according to various threads</li>
<li>Manually linking CUDA libraries during <code class="notranslate">./configure</code> to not use symlinked libraries</li>
<li>Various other hacks over the last week <g-emoji class="g-emoji" alias="sob" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f62d.png">😭</g-emoji></li>
</ul>