<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Well, I added patches</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Windows 10, 64 bit</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Source</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.8.0</li>
<li><strong>Python version</strong>:  3.5, 3.6</li>
<li><strong>Bazel version (if compiling from source)</strong>: 1.12.0, 1.14.0</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:  MSVC 2015 Update v3</li>
<li><strong>CUDA/cuDNN version</strong>: 9.0/7.1</li>
<li><strong>GPU model and memory</strong>: GeForce GT 430</li>
<li><strong>Exact command to reproduce</strong>: ;-)</li>
</ul>
<h3 dir="auto">Description</h3>
<p dir="auto">I wanted to compile tensorflow with GPU support on Windows and I went through the ordeal (I was finally able to build it, but I'll come to that later). I always followed the stuff done in: <a href="https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/tools/ci_build/">https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/tools/ci_build/</a></p>
<p dir="auto">This is how it went:</p>
<p dir="auto">First, I attempted the CPU build.<br>
Initial attempt was with CMake. It took ~3hours, all went fine upto the final linking phase, there link.exe died with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(Lib target) -&gt;
  tf_core_kernels.dir\Release\tf_core_kernels.lib : fatal error LNK1248: image size (100039B2C) exceeds maximum allowable size (FFFFFFFF) "><pre class="notranslate"><code class="notranslate">(Lib target) -&gt;
  tf_core_kernels.dir\Release\tf_core_kernels.lib : fatal error LNK1248: image size (100039B2C) exceeds maximum allowable size (FFFFFFFF) 
</code></pre></div>
<p dir="auto">I was lost here, I tried tinkering with some CMake build options, but all resulted into the same error. So, I gave up CMake.</p>
<p dir="auto">Then, I attempted to build with bazel 1.12.0.<br>
I was using the mingw64 shell, so first, I had to patch <code class="notranslate">tensorflow/tools/pip_package/build_pip_package.sh</code> . I also sent a PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="318680793" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/18953" data-hovercard-type="pull_request" data-hovercard-url="/tensorflow/tensorflow/pull/18953/hovercard" href="https://github.com/tensorflow/tensorflow/pull/18953">#18953</a></p>
<p dir="auto">Surprisingly, the build went fine after that, and then I tried to run the tests. All were successful, except <code class="notranslate">tensorflow/python/kernel_tests/boosted_trees:training_ops_test</code>,  probably because it was not being run serially. Even for running the tests, I had to patch <code class="notranslate">tensorflow/contrib/tensorboard/plugins/trace/trace.py</code> and I also sent a PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="318680816" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/18954" data-hovercard-type="pull_request" data-hovercard-url="/tensorflow/tensorflow/pull/18954/hovercard" href="https://github.com/tensorflow/tensorflow/pull/18954">#18954</a></p>
<p dir="auto">Now comes the real deal. The GPU build.</p>
<p dir="auto">Again, I attempted the build with CMake. It took ~3hours and then link.exe failed with the same error as before. I spent quite some time reading about it and couldn't figure it out as most solutions said 'break your huge lib into smaller libs'. I am still baffled as to why the CI job at <a href="http://ci.tensorflow.org/job/tf-master-win-gpu-cmake/" rel="nofollow">http://ci.tensorflow.org/job/tf-master-win-gpu-cmake/</a> doesn't die with the same problem. I am using the same compiler, same linker,<br>
and my machine is powerful enough. So, I just gave up again on CMake.</p>
<p dir="auto">Going back to bazel now.  By this time, bazel 1.14.0 was was released. So I decided to use that instead. I had to apply the two patches as before, but life isn't so easy.</p>
<p dir="auto">Next big hurdle: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="297766287" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/17067" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/17067/hovercard" href="https://github.com/tensorflow/tensorflow/issues/17067">#17067</a> .</p>
<ul dir="auto">
<li>So I got <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dtrebbien/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dtrebbien">@dtrebbien</a> 's patch from <a href="https://github.com/dtrebbien/protobuf/commit/50f552646ba1de79e07562b41f3999fe036b4fd0">https://github.com/dtrebbien/protobuf/commit/50f552646ba1de79e07562b41f3999fe036b4fd0</a> and made changes to <code class="notranslate">tensorflow/workspace.bzl</code> to apply it to all protobuf checkouts.</li>
</ul>
<p dir="auto">Next big hurdle: NCCL doesn't seem to be officially supported on Windows.</p>
<ul dir="auto">
<li>But bazel is still trying to build nccl related stuff. I had to patch stuff again, to disable all references to nccl in the BUILD files all over the place. <g-emoji class="g-emoji" alias="crying_cat_face" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f63f.png">😿</g-emoji></li>
</ul>
<p dir="auto">Next big hurdle:</p>
<ul dir="auto">
<li>Shared libraries under <code class="notranslate">tensorflow/contrib/rnn/python/ops/</code> failed at link phase with the error:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Creating library bazel-out/host/bin/tensorflow/contrib/rnn/python/ops/_gru_ops.ifso and object bazel-out/host/bin/tensorflow/contrib/rnn/python/ops/_gru_ops.exp
blas_gemm.o : error LNK2019: unresolved external symbol &quot;public: class perftools::gputools::Stream &amp; __cdecl perftools::gputools::Stream::ThenBlasGemm(enum perftools::gputools::blas::Transpose,enum perftools::gputools::blas::Transpose,unsigned __int64,unsigned __int64,unsigned __int64,float,class perftools::gputools::DeviceMemory&lt;float&gt; const &amp;,int,class perftools::gputools::DeviceMemory&lt;float&gt; const &amp;,int,float,class perftools::gputools::DeviceMemory&lt;float&gt; *,int)&quot; (?ThenBlasGemm@Stream@gputools@perftools@@QEAAAEAV123@W4Transpose@blas@23@0_K11MAEBV?$DeviceMemory@M@23@H2HMPEAV623@H@Z) referenced in function &quot;public: void __cdecl tensorflow::functor::TensorCuBlasGemm&lt;float&gt;::operator()(class tensorflow::OpKernelContext *,bool,bool,unsigned __int64,unsigned __int64,unsigned __int64,float,float const *,int,float const *,int,float,float *,int)&quot; (??R?$TensorCuBlasGemm@M@functor@tensorflow@@QEAAXPEAVOpKernelContext@2@_N1_K22MPEBMH3HMPEAMH@Z)
blas_gemm.o : error LNK2019: unresolved external symbol &quot;public: class perftools::gputools::Stream &amp; __cdecl perftools::gputools::Stream::ThenBlasGemm(enum perftools::gputools::blas::Transpose,enum perftools::gputools::blas::Transpose,unsigned __int64,unsigned __int64,unsigned __int64,double,class perftools::gputools::DeviceMemory&lt;double&gt; const &amp;,int,class perftools::gputools::DeviceMemory&lt;double&gt; const &amp;,int,double,class perftools::gputools::DeviceMemory&lt;double&gt; *,int)&quot; (?ThenBlasGemm@Stream@gputools@perftools@@QEAAAEAV123@W4Transpose@blas@23@0_K11NAEBV?$DeviceMemory@N@23@H2HNPEAV623@H@Z) referenced in function &quot;public: void __cdecl tensorflow::functor::TensorCuBlasGemm&lt;double&gt;::operator()(class tensorflow::OpKernelContext *,bool,bool,unsigned __int64,unsigned __int64,unsigned __int64,double,double const *,int,double const *,int,double,double *,int)&quot; (??R?$TensorCuBlasGemm@N@functor@tensorflow@@QEAAXPEAVOpKernelContext@2@_N1_K22NPEBNH3HNPEANH@Z)"><pre class="notranslate"><code class="notranslate">Creating library bazel-out/host/bin/tensorflow/contrib/rnn/python/ops/_gru_ops.ifso and object bazel-out/host/bin/tensorflow/contrib/rnn/python/ops/_gru_ops.exp
blas_gemm.o : error LNK2019: unresolved external symbol "public: class perftools::gputools::Stream &amp; __cdecl perftools::gputools::Stream::ThenBlasGemm(enum perftools::gputools::blas::Transpose,enum perftools::gputools::blas::Transpose,unsigned __int64,unsigned __int64,unsigned __int64,float,class perftools::gputools::DeviceMemory&lt;float&gt; const &amp;,int,class perftools::gputools::DeviceMemory&lt;float&gt; const &amp;,int,float,class perftools::gputools::DeviceMemory&lt;float&gt; *,int)" (?ThenBlasGemm@Stream@gputools@perftools@@QEAAAEAV123@W4Transpose@blas@23@0_K11MAEBV?$DeviceMemory@M@23@H2HMPEAV623@H@Z) referenced in function "public: void __cdecl tensorflow::functor::TensorCuBlasGemm&lt;float&gt;::operator()(class tensorflow::OpKernelContext *,bool,bool,unsigned __int64,unsigned __int64,unsigned __int64,float,float const *,int,float const *,int,float,float *,int)" (??R?$TensorCuBlasGemm@M@functor@tensorflow@@QEAAXPEAVOpKernelContext@2@_N1_K22MPEBMH3HMPEAMH@Z)
blas_gemm.o : error LNK2019: unresolved external symbol "public: class perftools::gputools::Stream &amp; __cdecl perftools::gputools::Stream::ThenBlasGemm(enum perftools::gputools::blas::Transpose,enum perftools::gputools::blas::Transpose,unsigned __int64,unsigned __int64,unsigned __int64,double,class perftools::gputools::DeviceMemory&lt;double&gt; const &amp;,int,class perftools::gputools::DeviceMemory&lt;double&gt; const &amp;,int,double,class perftools::gputools::DeviceMemory&lt;double&gt; *,int)" (?ThenBlasGemm@Stream@gputools@perftools@@QEAAAEAV123@W4Transpose@blas@23@0_K11NAEBV?$DeviceMemory@N@23@H2HNPEAV623@H@Z) referenced in function "public: void __cdecl tensorflow::functor::TensorCuBlasGemm&lt;double&gt;::operator()(class tensorflow::OpKernelContext *,bool,bool,unsigned __int64,unsigned __int64,unsigned __int64,double,double const *,int,double const *,int,double,double *,int)" (??R?$TensorCuBlasGemm@N@functor@tensorflow@@QEAAXPEAVOpKernelContext@2@_N1_K22NPEBNH3HNPEANH@Z)
</code></pre></div>
<ul dir="auto">
<li>Then I stumbled upon <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="278302064" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/15013" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/15013/hovercard" href="https://github.com/tensorflow/tensorflow/issues/15013">#15013</a> and saw that another person had faced the same issue quite some time ago and the OP's solution was <code class="notranslate">add all the .lib from tensorflow and cuda</code> . I can't do that. So, I went looking for where this symbol comes from, and where should it be. Took me a while to realize what's happening and that's when I came across the workaround<br>
of using intermediate interface shared object file at <a href="https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/tensorflow.bzl#L1228-L1237">https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/tensorflow.bzl#L1228-L1237</a>. Apparently, this didn't have enough symbols for the kernels in <code class="notranslate">tensorflow/contrib/rnn/python/ops/</code>. So I started looking for where I could find them and ended up adding:<br>
<code class="notranslate">clean_dep("//tensorflow/stream_executor:stream_executor_impl"),</code> to that list and the link error went away.</li>
</ul>
<p dir="auto">Next hurdle: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="333122041" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/20088" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/20088/hovercard" href="https://github.com/tensorflow/tensorflow/issues/20088">#20088</a></p>
<ul dir="auto">
<li>The compiler doesn't like<br>
<a href="https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/contrib/fused_conv/kernels/fused_conv2d_bias_activation_op.cc#L46">https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/contrib/fused_conv/kernels/fused_conv2d_bias_activation_op.cc#L46</a> when it was already being done at <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/conv_ops_gpu.h#L189">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/conv_ops_gpu.h#L189</a> . So I commented out the typdef in <code class="notranslate">fused_conv2d_bias_activation_op.cc</code> and tried building it. The error went away, but link.exe cried again at the end, because it didn't know where to get the symbol for <code class="notranslate">GetCudnnWorkspaceLimit</code> at <a href="https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/contrib/fused_conv/kernels/fused_conv2d_bias_activation_op.cc#L519">https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/contrib/fused_conv/kernels/fused_conv2d_bias_activation_op.cc#L519</a> . After some searching, I found out that this symbol comes from <a href="https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/core/kernels/conv_ops.cc#L456-L471">https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/core/kernels/conv_ops.cc#L456-L471</a> . So I tried making it part of the intermediary additional_deps_impl.ifso import lib, but no matter what, it always ended up bloating it with symbols &gt; 64k (it went upto ~72k, when I tried adding some deps which might have that symbol). So I gave up on this contrib kernel at this point and when I looked at the CMake build (I could be wrong here) I didn't find it building this kernel either. So I just ended up commenting out all references to <code class="notranslate">fused_conv</code> in the BUILD files.</li>
</ul>
<p dir="auto">Next hurdle: Bazel had made some breaking changes in 1.13.0</p>
<ul dir="auto">
<li>I came across at <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="294430208" data-permission-text="Title is private" data-url="https://github.com/bazelbuild/bazel/issues/4583" data-hovercard-type="issue" data-hovercard-url="/bazelbuild/bazel/issues/4583/hovercard" href="https://github.com/bazelbuild/bazel/issues/4583">bazelbuild/bazel#4583</a> and then applied two other patches locally to get around that problem.</li>
</ul>
<p dir="auto">Next hurdle: Tests won't run.</p>
<ul dir="auto">
<li>Most of them want to import nccl. <g-emoji class="g-emoji" alias="sob" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f62d.png">😭</g-emoji> So I ended up faking it by creating a blank module and then the tests ran fine. I did have to tell bazel to ignore the <code class="notranslate">contrib/lite</code> subset <code class="notranslate">-//${PY_TEST_DIR}/tensorflow/contrib/lite/...</code>.</li>
</ul>
<p dir="auto">For the lost souls who attempt at building tensorflow with GPU support, I hope this issue might be helpful. All the patches and the entire build can be found here: <a href="https://github.com/AnacondaRecipes/aggregate/blob/459dee0989e0c7cc4ab66c49d3d7605cddbb1bc3/tensorflow-gpu-base-feedstock/recipe/meta.yaml">https://github.com/AnacondaRecipes/aggregate/blob/459dee0989e0c7cc4ab66c49d3d7605cddbb1bc3/tensorflow-gpu-base-feedstock/recipe/meta.yaml</a></p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: no</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Windows 10 1803</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: source</li>
<li><strong>TensorFlow version (use command below)</strong>: v1.11.0-rc1</li>
<li><strong>Python version</strong>: 3.6.5</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.15.2</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: vs2017 15.8 / cl.exe 19.15.26726</li>
<li><strong>CUDA/cuDNN version</strong>: 9.2.148.1/7.2.1</li>
<li><strong>GPU model and memory</strong>: 1080ti 11GB</li>
<li><strong>Exact command to reproduce</strong>:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
bazel-bin\tensorflow\tools\pip_package\build_pip_package C:/tmp/tensorflow_pkg"><pre class="notranslate"><code class="notranslate">bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
bazel-bin\tensorflow\tools\pip_package\build_pip_package C:/tmp/tensorflow_pkg
</code></pre></div>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">If build artifact large than 4GB, it can't generate the python package.</p>
<p dir="auto">A easy way to reproduce the issue</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,7.0]: 3.0,3.2,3.5,5.0,5.2,5.3"><pre class="notranslate"><code class="notranslate">Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,7.0]: 3.0,3.2,3.5,5.0,5.2,5.3
</code></pre></div>
<p dir="auto">related link:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="336090628" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/20332" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/20332/hovercard?comment_id=415974623&amp;comment_type=issue_comment" href="https://github.com/tensorflow/tensorflow/issues/20332#issuecomment-415974623">#20332 (comment)</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="361772394" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/22382" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/22382/hovercard" href="https://github.com/tensorflow/tensorflow/issues/22382">#22382</a><br>
<a href="https://github.com/bazelbuild/bazel/blob/0.17.1/third_party/ijar/zip.cc#L74">https://github.com/bazelbuild/bazel/blob/0.17.1/third_party/ijar/zip.cc#L74</a></p>
<h3 dir="auto">Source code / logs</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Uncompressed input jar has size ???, which exceeds the maximum supported output size 4294967295.
Assuming that ijar will be smaller and hoping for the best.

Unzipping simple_console_for_windows.zip to create runfiles tree...
[./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip or
        ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip.zip, and cannot find ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip.ZIP, period.
"><pre class="notranslate"><code class="notranslate">Uncompressed input jar has size ???, which exceeds the maximum supported output size 4294967295.
Assuming that ijar will be smaller and hoping for the best.

Unzipping simple_console_for_windows.zip to create runfiles tree...
[./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip or
        ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip.zip, and cannot find ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip.ZIP, period.

</code></pre></div>