<p dir="auto">Have I written custom code: Yes<br>
OS Platform and Distribution: Linux Ubuntu 17.04<br>
TensorFlow installed from: source<br>
TensorFlow version: 1.6.0-rc1<br>
Python version: 3.6<br>
Bazel version: 0.10<br>
GCC/Compiler version: 6.0<br>
CUDA/cuDNN version: CUDA 9.1 cuDNN 7.0.5<br>
GPU model and memory: NVIDIA Titan Z 12GB<br>
<code class="notranslate">Exact</code> command to reproduce: multi_gpu_model(model, gpus=2)</p>
<p dir="auto">Just upgraded from rc0 to rc1 of release 1.6.0 and I'm now getting the following crash when running the multi_gpu_model function (was working fine with rc0):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="parallel_model = multi_gpu_model(model, gpus=2)"><pre class="notranslate"><code class="notranslate">parallel_model = multi_gpu_model(model, gpus=2)
</code></pre></div>
<p dir="auto">File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/_impl/keras/utils/training_utils.py", line 207, in multi_gpu_model</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="return Model(model.inputs, merged)"><pre class="notranslate"><code class="notranslate">return Model(model.inputs, merged)
</code></pre></div>
<p dir="auto">File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/_impl/keras/engine/topology.py", line 694, in <strong>init</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="self._init_graph_network(*args, **kwargs)"><pre class="notranslate"><code class="notranslate">self._init_graph_network(*args, **kwargs)
</code></pre></div>
<p dir="auto">File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/_impl/keras/engine/topology.py", line 733, in _init_graph_network</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if layer.is_placeholder:"><pre class="notranslate"><code class="notranslate">if layer.is_placeholder:
</code></pre></div>
<p dir="auto">AttributeError: 'Lambda' object has no attribute 'is_placeholder'</p>
<p dir="auto">I rolled back to the 1.5 branch and I'm not having any issues running multi_gpu_model there.</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:<br>
Linux Ubuntu 16.04</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:<br>
Built from source</li>
<li><strong>TensorFlow version (use command below)</strong>:<br>
v1.6.0-rc0-19-gecec1d8 1.6.0-rc1<br>
master</li>
<li><strong>Python version</strong>: 3.5.2</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.10.0</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: 5.4.0</li>
<li><strong>CUDA/cuDNN version</strong>: 9.1.85/7.0.5.15</li>
<li><strong>GPU model and memory</strong>:<br>
GeForce GTX 1080 Ti<br>
totalMemory: 10.91GiB freeMemory: 8.36GiB</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">If tf1.6 is compiled with <code class="notranslate">--march=native</code> then running inference on a large model ends up with crash with cryptic stack trace, 100% reproducible, no matter with or without CUDA (whether CUDA_VISIBLE_DEVICES="" or absent).</p>
<p dir="auto">If compiled <strong>without</strong> avx512 support (i.e. <code class="notranslate">-O3 -msse4.2 -mavx2 -mfma</code> only), then everything works fine.<br>
TF emits a warning though:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2018-02-15 14:03:54.237530: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F"><pre class="notranslate"><code class="notranslate">2018-02-15 14:03:54.237530: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F
</code></pre></div>
<p dir="auto">I checked another issue with 64 byte alignment (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="284180657" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/15588" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/15588/hovercard" href="https://github.com/tensorflow/tensorflow/issues/15588">#15588</a>), but it does not help.</p>
<p dir="auto">Here is a crash trace, if it may help.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(gdb) bt
#0  0x00007fdea3c5c9d5 in Eigen::internal::gemm_pack_lhs&lt;float, long, Eigen::internal::TensorContractionSubMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, 48, 16, 0, false, false&gt;::operator()(float*, Eigen::internal::TensorContractionSubMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt; const&amp;, long, long, long, long) () from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#1  0x00007fdea3cd0fc4 in Eigen::TensorEvaluator&lt;Eigen::TensorContractionOp&lt;Eigen::array&lt;Eigen::IndexPair&lt;long&gt;, 1ul&gt; const, Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const&gt; const, Eigen::ThreadPoolDevice&gt;::Context&lt;Eigen::internal::gemm_pack_lhs&lt;float, long, Eigen::internal::TensorContractionSubMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, 48, 16, 0, false, false&gt;, Eigen::internal::gemm_pack_rhs&lt;float, long, Eigen::internal::TensorContractionSubMapper&lt;float, long, 0, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, 4, 0, false, false&gt;, Eigen::internal::gebp_kernel&lt;float, float, long, Eigen::internal::blas_data_mapper&lt;float, long, 0, 0&gt;, 48, 4, false, false&gt;, Eigen::internal::TensorContractionInputMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, Eigen::internal::TensorContractionInputMapper&lt;float, long, 0, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, Eigen::internal::blas_data_mapper&lt;float, long, 0, 0&gt; &gt;::enqueue_packing_helper(long, long, long, bool) ()
   from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#2  0x00007fdea06232d1 in Eigen::NonBlockingThreadPoolTempl&lt;tensorflow::thread::EigenEnvironment&gt;::WorkerLoop(int) ()
   from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/../libtensorflow_framework.so
#3  0x00007fdea06210e7 in std::_Function_handler&lt;void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function&lt;void ()&gt;)::{lambda()#1}&gt;::_M_invoke(std::_Any_data const&amp;) ()
   from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/../libtensorflow_framework.so
#4  0x00007fde96afbc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fded77d76ba in start_thread (arg=0x7fdeb87d8700) at pthread_create.c:333
#6  0x00007fded750d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)"><pre class="notranslate"><code class="notranslate">(gdb) bt
#0  0x00007fdea3c5c9d5 in Eigen::internal::gemm_pack_lhs&lt;float, long, Eigen::internal::TensorContractionSubMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, 48, 16, 0, false, false&gt;::operator()(float*, Eigen::internal::TensorContractionSubMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt; const&amp;, long, long, long, long) () from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#1  0x00007fdea3cd0fc4 in Eigen::TensorEvaluator&lt;Eigen::TensorContractionOp&lt;Eigen::array&lt;Eigen::IndexPair&lt;long&gt;, 1ul&gt; const, Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const&gt; const, Eigen::ThreadPoolDevice&gt;::Context&lt;Eigen::internal::gemm_pack_lhs&lt;float, long, Eigen::internal::TensorContractionSubMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, 48, 16, 0, false, false&gt;, Eigen::internal::gemm_pack_rhs&lt;float, long, Eigen::internal::TensorContractionSubMapper&lt;float, long, 0, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, 4, 0, false, false&gt;, Eigen::internal::gebp_kernel&lt;float, float, long, Eigen::internal::blas_data_mapper&lt;float, long, 0, 0&gt;, 48, 4, false, false&gt;, Eigen::internal::TensorContractionInputMapper&lt;float, long, 1, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, Eigen::internal::TensorContractionInputMapper&lt;float, long, 0, Eigen::TensorEvaluator&lt;Eigen::TensorMap&lt;Eigen::Tensor&lt;float const, 2, 1, long&gt;, 16, Eigen::MakePointer&gt; const, Eigen::ThreadPoolDevice&gt;, Eigen::array&lt;long, 1ul&gt;, Eigen::array&lt;long, 1ul&gt;, 16, true, false, 0, Eigen::MakePointer&gt;, Eigen::internal::blas_data_mapper&lt;float, long, 0, 0&gt; &gt;::enqueue_packing_helper(long, long, long, bool) ()
   from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#2  0x00007fdea06232d1 in Eigen::NonBlockingThreadPoolTempl&lt;tensorflow::thread::EigenEnvironment&gt;::WorkerLoop(int) ()
   from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/../libtensorflow_framework.so
#3  0x00007fdea06210e7 in std::_Function_handler&lt;void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function&lt;void ()&gt;)::{lambda()#1}&gt;::_M_invoke(std::_Any_data const&amp;) ()
   from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/../libtensorflow_framework.so
#4  0x00007fde96afbc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fded77d76ba in start_thread (arg=0x7fdeb87d8700) at pthread_create.c:333
#6  0x00007fded750d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)
</code></pre></div>