<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): not sure</li>
<li>TensorFlow version (use command below): v1.12.0-0-ga6d8ffa</li>
<li>Python version: 3.6.6</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version: CUDA 10.0</li>
<li>GPU model and memory: GeForce GTX 1080 Ti, 11178MiB</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
When computing a batched matmul with 32-bit precision and batch size &gt;= 2 ** 16, the first 2 ** 16 - 1 batch elements of the result are correct, and the remaining elements are arbitrary - often zero, but not always. When the <code class="notranslate">sess.run</code> call is made multiple times, the result is usually the same or similar. The bug only seems to occur when using a variable or placeholder.</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
Every batch element of the result should be correct.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
import tensorflow as tf

s = (100000, 1, 1)
p = tf.placeholder(shape=s, dtype=tf.float32)
x = tf.matmul(tf.ones(s), p)

with tf.Session() as sess:
    r = sess.run(x, feed_dict={p: np.ones(s, dtype=np.float32)})
    print(r[2**16 - 5:2**16 + 5, 0, 0])"><pre class="notranslate"><code class="notranslate">import numpy as np
import tensorflow as tf

s = (100000, 1, 1)
p = tf.placeholder(shape=s, dtype=tf.float32)
x = tf.matmul(tf.ones(s), p)

with tf.Session() as sess:
    r = sess.run(x, feed_dict={p: np.ones(s, dtype=np.float32)})
    print(r[2**16 - 5:2**16 + 5, 0, 0])
</code></pre></div>
<p dir="auto">Typical output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[1. 1. 1. 1. 0. 0. 0. 0. 0. 0.]"><pre class="notranslate"><code class="notranslate">[1. 1. 1. 1. 0. 0. 0. 0. 0. 0.]
</code></pre></div>
<p dir="auto"><strong>Other info / logs</strong></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): v1.14.0-rc1-22-gaf24dc91b5 1.14.0</li>
<li>Python version: 3.6.7</li>
<li>GCC/Compiler version (if compiling from source): 8.2.0</li>
<li>CUDA/cuDNN version: cuda-10.0</li>
<li>GPU model and memory:</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
tf.matmul on tensorflow-gpu gave wrong results. Here is a simplified version of the code.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

r = [[ 1.0, 0.0],    [0.0, 1.0]]

x, y = np.meshgrid(list(range(400)), list(range(400)))
coords = np.stack([x,y],-1).reshape((400,400,2,1))
coords = tf.convert_to_tensor(coords,dtype=tf.float32)

r1 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords,feed_dict={r1:r})

plt.matshow(ret[:,:,0,0])
plt.show()"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

r = [[ 1.0, 0.0],    [0.0, 1.0]]

x, y = np.meshgrid(list(range(400)), list(range(400)))
coords = np.stack([x,y],-1).reshape((400,400,2,1))
coords = tf.convert_to_tensor(coords,dtype=tf.float32)

r1 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords,feed_dict={r1:r})

plt.matshow(ret[:,:,0,0])
plt.show()
</code></pre></div>
<p dir="auto">When I ran it on my tensorflow-gpu, here is the result:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://gist.githubusercontent.com/sWizad/3a25d6559ea3308e5cc2731519635c32/raw/2479341ed43fb8e69c94eda3a1362d5cbde7d2d7/Figure_1.png"><img src="https://gist.githubusercontent.com/sWizad/3a25d6559ea3308e5cc2731519635c32/raw/2479341ed43fb8e69c94eda3a1362d5cbde7d2d7/Figure_1.png" alt="bug" style="max-width: 100%;"></a><br>
Looks like it stops computing halfway through and gave the rest 0 as a result.</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
Here is the result with CPU:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/45821224/62137176-08631480-b2d5-11e9-8b8e-6348e8b206e7.png"><img src="https://user-images.githubusercontent.com/45821224/62137176-08631480-b2d5-11e9-8b8e-6348e8b206e7.png" alt="CPU" style="max-width: 100%;"></a></p>
<hr>
<p dir="auto">Below is my old post. Initially, I thought the problem was related to TFRecord, but seems like this problem occurs without even using tfrecord too. For completeness, I keep the old example code with tfrecord.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

def parser(serialized_example):
      fs = tf.io.parse_single_example(
          serialized_example,
          features={ &quot;r&quot;: tf.FixedLenFeature([4], tf.float32) })
      fs[&quot;r&quot;] = tf.reshape(fs[&quot;r&quot;], [2, 2])
      return fs

r = [[ 1.0, 0.0],[0.0, 1.0]]

with tf.io.TFRecordWriter(&quot;cc.test&quot;) as tfrecord_writer:
    feature = {&quot;r&quot;: tf.train.Feature(float_list=tf.train.FloatList(value=np.array(r).flatten() ))}
    example = tf.train.Example(features=tf.train.Features(feature=feature))
    tfrecord_writer.write(example.SerializeToString())
dataset = tf.data.TFRecordDataset([&quot;cc.test&quot;])
dataset = dataset.map(parser).repeat().make_one_shot_iterator()
features = dataset.get_next()

x, y = tf.meshgrid(list(range(400)), list(range(400)))
coords = tf.stack([x, y], -1)     #(h,w,2)
coords = tf.expand_dims(tf.cast(coords,tf.float32),-1) #(h,w,2,1)

r1 = features[&quot;r&quot;]
r2 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords[:,:,0,0])
plt.matshow(ret)
plt.show()"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

def parser(serialized_example):
      fs = tf.io.parse_single_example(
          serialized_example,
          features={ "r": tf.FixedLenFeature([4], tf.float32) })
      fs["r"] = tf.reshape(fs["r"], [2, 2])
      return fs

r = [[ 1.0, 0.0],[0.0, 1.0]]

with tf.io.TFRecordWriter("cc.test") as tfrecord_writer:
    feature = {"r": tf.train.Feature(float_list=tf.train.FloatList(value=np.array(r).flatten() ))}
    example = tf.train.Example(features=tf.train.Features(feature=feature))
    tfrecord_writer.write(example.SerializeToString())
dataset = tf.data.TFRecordDataset(["cc.test"])
dataset = dataset.map(parser).repeat().make_one_shot_iterator()
features = dataset.get_next()

x, y = tf.meshgrid(list(range(400)), list(range(400)))
coords = tf.stack([x, y], -1)     #(h,w,2)
coords = tf.expand_dims(tf.cast(coords,tf.float32),-1) #(h,w,2,1)

r1 = features["r"]
r2 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords[:,:,0,0])
plt.matshow(ret)
plt.show()
</code></pre></div>
<p dir="auto">The code will create "cc.test" file to save the variable <code class="notranslate">r</code> and load it as <code class="notranslate">r1</code>. Then matmul <code class="notranslate">r1</code> with some big varibles.</p>