<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:<br>
Ubuntu 16.04 (ppc64le)</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:<br>
Installed from source (v1.0.1)</li>
<li><strong>TensorFlow version (use command below)</strong>:<br>
('v1.0.1-0-ge895d5c-dirty', '1.0.1')</li>
<li><strong>Bazel version (if compiling from source)</strong>:<br>
0.4.4-2017-05-26 (@80a07b5)</li>
<li><strong>CUDA/cuDNN version</strong>:<br>
CUDA = 8.0 and cuDNN = 5.1</li>
<li><strong>GPU model and memory</strong>:<br>
GPU 0: Tesla P100-SXM2-16GB<br>
GPU 1: Tesla P100-SXM2-16GB</li>
<li><strong>Exact command to reproduce</strong>:<br>
bazel test --config=opt --config=cuda //tensorflow/core/kernels:sparse_matmul_op_test_gpu</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">This is regarding failure of test case <code class="notranslate">SparseMatmulOpTest.BroadcastPacketTest</code> in <code class="notranslate">tensorflow/core/kernels/sparse_matmul_op_test.cc</code> file.While executing this test case on ppc64le, it was observed that following line returns unexpected results:<br>
<a href="https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op_test.cc#L255">https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op_test.cc#L255</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="internal::pstoreu(data2, internal::pbroadcast_first&lt;Packet&gt;(
                              internal::ploadu&lt;Packet&gt;(data1)));
"><pre class="notranslate"><code class="notranslate">internal::pstoreu(data2, internal::pbroadcast_first&lt;Packet&gt;(
                              internal::ploadu&lt;Packet&gt;(data1)));

</code></pre></div>
<p dir="auto">Here we are getting expected result on <code class="notranslate">x86</code> for data2 array = <code class="notranslate">[0.170094 0.170094 0.170094 0.170094]</code>, however on <code class="notranslate">ppc64le</code> getting incorrect result i.e. <code class="notranslate">[  0.170094    0.14922 -0.0823886   0.026985]</code></p>
<p dir="auto">I have done some investigation around this - using <code class="notranslate">print/cout</code> statement I tried to understand the code flow on <code class="notranslate">ppc64le </code>as well as on <code class="notranslate">X86 </code>platform.<br>
Here I found that for <code class="notranslate">internal::pbroadcast_first&lt;Packet&gt;</code> line ,on both the platform executed different functions, see below-<br>
On x86 executed <code class="notranslate">EIGEN_STRONG_INLINE Packet4f pbroadcast_first&lt;Packet4f&gt;(const Packet4f&amp; a) </code> function(<a href="https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op.h#L197">https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op.h#L197</a>)<br>
And on ppc64le executed some different function i.e. <code class="notranslate">EIGEN_DEVICE_FUNC inline Packet pbroadcast_first(const Packet&amp; a)</code> (<a href="https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op.h#L92">https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op.h#L92</a>)</p>
<p dir="auto">That is why we are getting incorrect result on ppc64le for data2 array and test fails.<br>
I have done some debugging on this but couldn't find the reason - why control is going to different functions on both the platform for <code class="notranslate">internal::pbroadcast_first&lt;Packet&gt;</code> ?</p>
<p dir="auto">If could get any suggestions/pointers to why this is happening that would be great!<br>
(I am new to tensorflow code but interested/would-like-to debug &amp; help)</p>
<p dir="auto">Thanks!</p>
<h3 dir="auto">Source code / logs</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$  bazel test --config=opt --config=cuda //tensorflow/core/kernels:sparse_matmul_op_test_gpu

exec ${PAGER:-/usr/bin/less} &quot;$0&quot; || exit 1
-----------------------------------------------------------------------------
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
Running main() from test_main.cc
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from SparseMatmulOpTest
[ RUN      ] SparseMatmulOpTest.BroadcastPacketTest
[0.170094 0.170094 0.170094 0.170094] != [  0.170094    0.14922 -0.0823886   0.026985], differences: [         0 -0.0208738  -0.252482  -0.143109]
tensorflow/core/kernels/sparse_matmul_op_test.cc:257: Failure
Value of: areApprox(ref, data2, PacketSize)
  Actual: false
Expected: true
[  FAILED  ] SparseMatmulOpTest.BroadcastPacketTest (0 ms)
[ RUN      ] SparseMatmulOpTest.InterleavePacketTest
[       OK ] SparseMatmulOpTest.InterleavePacketTest (0 ms)
[ RUN      ] SparseMatmulOpTest.Bfloat16ExpandTest
[       OK ] SparseMatmulOpTest.Bfloat16ExpandTest (0 ms)
[ RUN      ] SparseMatmulOpTest.Bfloat16LoadTest
[       OK ] SparseMatmulOpTest.Bfloat16LoadTest (0 ms)
[----------] 4 tests from SparseMatmulOpTest (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] SparseMatmulOpTest.BroadcastPacketTest

 1 FAILED TEST
"><pre class="notranslate"><code class="notranslate">$  bazel test --config=opt --config=cuda //tensorflow/core/kernels:sparse_matmul_op_test_gpu

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
Running main() from test_main.cc
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from SparseMatmulOpTest
[ RUN      ] SparseMatmulOpTest.BroadcastPacketTest
[0.170094 0.170094 0.170094 0.170094] != [  0.170094    0.14922 -0.0823886   0.026985], differences: [         0 -0.0208738  -0.252482  -0.143109]
tensorflow/core/kernels/sparse_matmul_op_test.cc:257: Failure
Value of: areApprox(ref, data2, PacketSize)
  Actual: false
Expected: true
[  FAILED  ] SparseMatmulOpTest.BroadcastPacketTest (0 ms)
[ RUN      ] SparseMatmulOpTest.InterleavePacketTest
[       OK ] SparseMatmulOpTest.InterleavePacketTest (0 ms)
[ RUN      ] SparseMatmulOpTest.Bfloat16ExpandTest
[       OK ] SparseMatmulOpTest.Bfloat16ExpandTest (0 ms)
[ RUN      ] SparseMatmulOpTest.Bfloat16LoadTest
[       OK ] SparseMatmulOpTest.Bfloat16LoadTest (0 ms)
[----------] 4 tests from SparseMatmulOpTest (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] SparseMatmulOpTest.BroadcastPacketTest

 1 FAILED TEST

</code></pre></div>
<p dir="auto">###Environment info:<br>
Operating System: Ubuntu 16.04 (ppc64le)</p>
<p dir="auto">###Installed version of CUDA and cuDNN: cuda &amp; OpenCL disabled and all other features enabled</p>
<p dir="auto">###Build and test INFO<br>
Built TF (version 1.0.1) successfully on Ubuntu 16.04 and RHEL 7.3.<br>
Now I am trying to run the bazel tests -  12 tests are failing out of 1044. Please find the attachment for log details.</p>
<p dir="auto">Thanks!</p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/files/892266/C-C.Test.Failure.analysis.for.TF.xlsx">C-C++Test Failure  analysis for TF.xlsx</a></p>