<p dir="auto">Hi,</p>
<p dir="auto">I am trying to build latest TF with CUDNN 6.0 + CUDA 8.0 on Ubuntu 14 but it failed with the following error message. I found some similar issue (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="121548025" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/469" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/469/hovercard" href="https://github.com/tensorflow/tensorflow/issues/469">#469</a>) reported in the past, not sure if the latest tip has fixed it?</p>
<p dir="auto">Any suggestion would be appreciated.</p>
<p dir="auto">Thanks</p>
<p dir="auto"><strong>Build Command</strong></p>
<ul dir="auto">
<li>
<p dir="auto">build command for CUDA that failed<br>
<code class="notranslate">bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package </code></p>
</li>
<li>
<p dir="auto">build for CPU works well<br>
<code class="notranslate">bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package</code></p>
</li>
</ul>
<p dir="auto"><strong>System Info</strong></p>
<ul dir="auto">
<li>bazel version : Build label: 0.5.4</li>
<li>CUDA: 8.0</li>
<li>CUDNN 6.0</li>
<li>TF origin/master latest sync as 10/26/17 (cb7cb40 Merge pull request <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="268431149" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/13972" data-hovercard-type="pull_request" data-hovercard-url="/tensorflow/tensorflow/pull/13972/hovercard" href="https://github.com/tensorflow/tensorflow/pull/13972">#13972</a> from taehoonlee/fix_typos)</li>
</ul>
<p dir="auto"><strong>Error message:</strong><br>
<code class="notranslate">ERROR: $PROJECT_ROOT/tensorflow/tensorflow/stream_executor/BUILD:52:1: undeclared inclusion(s) in rule '//tensorflow/stream_executor:cuda_platform': this rule is missing dependency declarations for the following files included by 'tensorflow/stream_executor/cuda/cuda_blas.cc': '/usr/local/cuda/include/cublas_api.h' '/usr/local/cuda/include/driver_types.h' '/usr/local/cuda/include/host_defines.h' '/usr/local/cuda/include/cuComplex.h' '/usr/local/cuda/include/vector_types.h' '/usr/local/cuda/include/builtin_types.h' '/usr/local/cuda/include/device_types.h' '/usr/local/cuda/include/surface_types.h' '/usr/local/cuda/include/texture_types.h' '/usr/local/cuda/include/cuda_fp16.h' '/usr/local/cuda/include/library_types.h' tensorflow/stream_executor/cuda/cuda_blas.cc: In function 'cudaDataType_t perftools::gputools::cuda::{anonymous}::CUDAComputationType(perftools::gputools::blas::ComputationType)': tensorflow/stream_executor/cuda/cuda_blas.cc:527:1: warning: control reaches end of non-void function [-Wreturn-type] } ^ </code></p>
<p dir="auto">Hey guys,</p>
<p dir="auto">I can compile the cpu version of tensorflow 1.2.0-rc2 without any problem, however, i am blocked when i try to compile the gpu version. I spend almost the whole day to install tensorflow 1.2.0-rc2 gpu version in our cluster, however there is no lucky.</p>
<h3 dir="auto">Here is the error from the terminal:</h3>
<p dir="auto">ERROR: /home/hpc-xin/.cache/bazel/_bazel_hpc-xin/c5e302e32d7acb9c3e4fce8142d1cd66/external/nccl_archive/BUILD:33:1: undeclared inclusion(s) in rule '@nccl_archive//:nccl':<br>
this rule is missing dependency declarations for the following files included by 'external/nccl_archive/src/libwrap.cu.cc':<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/include-fixed/limits.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/include-fixed/syslimits.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/include/stddef.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/new'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/x86_64-unknown-linux-gnu/bits/c++config.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/x86_64-unknown-linux-gnu/bits/os_defines.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/x86_64-unknown-linux-gnu/bits/cpu_defines.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/exception'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/bits/atomic_lockfree_defines.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/bits/exception_ptr.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/bits/exception_defines.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/bits/nested_exception.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/include/stdarg.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/cmath'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/bits/cpp_type_traits.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/ext/type_traits.h'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/cstdlib'<br>
'/gpfs/gpfs1/apps2/gcc/5.4.0/include/c++/5.4.0/cstdio'.<br>
Target //tensorflow/tools/pip_package:build_pip_package failed to build<br>
Use --verbose_failures to see the command lines of failed build steps.<br>
INFO: Elapsed time: 22.226s, Critical Path: 5.20s</p>
<h3 dir="auto">OS Information</h3>
<p dir="auto">Red Hat Enterprise Linux Workstation release 6.7 (Santiago)</p>
<h3 dir="auto">Modules I loaded</h3>
<p dir="auto">gcc/5.4.0, cuda/8.0.61, cudnn/6.0, java/1.8.0_31, sqlite/3.18.0, tcl/8.6.6.8606, python/3.6.1</p>
<h3 dir="auto">Here is how i make the configuration</h3>
<p dir="auto">gcc (v5.4.0): /apps2/gcc/5.4.0 (customized path）<br>
enabled cuda (v8.0): /apps2/cuda/8.0.61 (customized path）<br>
enabled cudnn: /apps2/cudnn/6.0 (customized path）<br>
python (v3.6.1, which is compiled with gcc 5.4.0): /apps2/python/3.6.1 (customized path)<br>
Other options are disabled (such as hadoop, google cloud).</p>
<h3 dir="auto">Here are some files i modified manually before compiling</h3>
<p dir="auto">(1) /tensorflow-1.2.0-rc2/third_party/gpus/crosstool/CROSSTOOL_clang.tpl<br>
/tensorflow-1.2.0-rc2/third_party/gpus/crosstool/CROSSTOOL_nvcc.tpl<br>
Add the following contents in all of "toolchain":<br>
cxx_builtin_include_directory: "/apps2/gcc/5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/include"<br>
cxx_builtin_include_directory: "/apps2/gcc/5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/include-fixed"<br>
cxx_builtin_include_directory: "/apps2/gcc/5.4.0/include/c++/5.4.0"<br>
cxx_builtin_include_directory: "/apps2/gcc/5.4.0/include"<br>
cxx_builtin_include_directory: "/apps2/cuda/8.0.61/include"<br>
cxx_builtin_include_directory: "/apps2/cudnn/6.0/include"</p>
<p dir="auto">(2) /home/hpc-xin/.cache/bazel/_bazel_hpc-xin/c5e302e32d7acb9c3e4fce8142d1cd66/external/protobuf/protobuf.bzl<br>
Add the following content in "ctx.action"<br>
env=ctx.configuration.default_shell_env</p>
<p dir="auto">(3) /home/hpc-xin/.cache/bazel/_bazel_hpc-xin/c5e302e32d7acb9c3e4fce8142d1cd66/external/nccl_archive/Makefile<br>
Some modification as below:<br>
CUDA_HOME ?= /usr/local/cuda --&gt; CUDA_HOME ?= /apps2/cuda/8.0.61</p>
<p dir="auto">I noticed that some of people said this issue could be bypassed by delete the nccl dependence in "/tensorflow-1.2.0-rc2/tensorflow/contrib/BUILD". However, i do not think this is the correct way to solve this issue. I wanna keep this dependence.</p>
<p dir="auto">Thanks so much for your help. Any comments are appreciated.<br>
Best Regards<br>
Xin</p>