<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: YES</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: CentOS 7.2 (and also Windows 10)</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>: N/A</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: binary (via <code class="notranslate">pip install tensorflow-gpu</code>)</li>
<li><strong>TensorFlow version (use command below)</strong>: v1.11.0-0-gc19e29306c 1.11.0</li>
<li><strong>Python version</strong>: 3.6.6</li>
<li><strong>Bazel version (if compiling from source)</strong>: N/A</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: N/A</li>
<li><strong>CUDA/cuDNN version</strong>: 9.0/7.1.4</li>
<li><strong>GPU model and memory</strong>: NVIDIA GeForce GTX 1070 (8 GiB)</li>
<li><strong>Exact command to reproduce</strong>: <code class="notranslate">python nmsstest.py</code>; see below for the content of <code class="notranslate">nmstest.py</code></li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">The code in the following section, which calls <code class="notranslate">tf.image.non_max_suppression()</code> in <code class="notranslate">tf.while_loop()</code> many times, crashes abnormally.<br>
Crash reason (and loop count) varies from time to time, for example:</p>
<ul dir="auto">
<li><code class="notranslate">F tensorflow/core/common_runtime/bfc_allocator.cc:384] Check failed: h != kInvalidChunkHandle</code> at loop <code class="notranslate">i == 140</code></li>
<li><code class="notranslate">F tensorflow/core/common_runtime/bfc_allocator.cc:462] Check failed: c-&gt;in_use() &amp;&amp; (c-&gt;bin_num == kInvalidBinNum)</code> at loop <code class="notranslate">i == 76</code></li>
<li><code class="notranslate">Bus error</code> at loop <code class="notranslate">i == 34</code></li>
<li>heap corruption reported by libc (see the following section) at loop <code class="notranslate">i == 35</code></li>
<li>sometimes it crashes sliently without any logs (on Windows)</li>
</ul>
<p dir="auto">I notice that:</p>
<ul dir="auto">
<li>it's since TensorFlow 1.11.0rc0; TF 1.10.1 was okay</li>
<li>it also reproduces on Windows</li>
<li>it also reproduces on CPU version (<code class="notranslate">pip install tensorflow</code>)</li>
<li>it does <strong>not</strong> reproduce if <code class="notranslate">num_threads=1</code>; calling <code class="notranslate">tf.image.non_max_suppression()</code> in parallel seems the trigger</li>
<li>even when I gave a fixed seed to <code class="notranslate">tf.random_uniform()</code>, crash cause varied</li>
</ul>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">The code to reproduce the problem is as follows:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf

if __name__ == '__main__':
    # crashes iff num_threads &gt; 1 on TensorFlow &gt;= 1.11.rc0
    num_threads = 10

    top_k = 1
    batch_size = 32
    num_boxes = 10000
    boxes_op = tf.random_uniform((batch_size,num_boxes,4), 0, 1)
    scores_op = tf.random_uniform((batch_size,num_boxes), 0, 1)
    indices_op = tf.while_loop(
        (lambda b, ta: True),
        (lambda b, ta: (b+1, ta.write(b, tf.image.non_max_suppression(boxes_op[b], scores_op[b], top_k, iou_threshold=0.3)))),
        (tf.constant(0),
         tf.TensorArray(tf.int32, size=batch_size, infer_shape=False, element_shape=(top_k,))),
        back_prop=False,
        parallel_iterations=num_threads,
        maximum_iterations=batch_size)[1].stack()

    with tf.Session() as session:
        for i in range(1000):
            indices = session.run(indices_op)
            print(f'#{i}: {indices.shape}')"><pre class="notranslate"><code class="notranslate">import tensorflow as tf

if __name__ == '__main__':
    # crashes iff num_threads &gt; 1 on TensorFlow &gt;= 1.11.rc0
    num_threads = 10

    top_k = 1
    batch_size = 32
    num_boxes = 10000
    boxes_op = tf.random_uniform((batch_size,num_boxes,4), 0, 1)
    scores_op = tf.random_uniform((batch_size,num_boxes), 0, 1)
    indices_op = tf.while_loop(
        (lambda b, ta: True),
        (lambda b, ta: (b+1, ta.write(b, tf.image.non_max_suppression(boxes_op[b], scores_op[b], top_k, iou_threshold=0.3)))),
        (tf.constant(0),
         tf.TensorArray(tf.int32, size=batch_size, infer_shape=False, element_shape=(top_k,))),
        back_prop=False,
        parallel_iterations=num_threads,
        maximum_iterations=batch_size)[1].stack()

    with tf.Session() as session:
        for i in range(1000):
            indices = session.run(indices_op)
            print(f'#{i}: {indices.shape}')
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python nmstest.py
2018-09-28 13:18:42.760545: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-09-28 13:18:43.123328: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-09-28 13:18:43.124208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: 
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 7.93GiB freeMemory: 7.83GiB
2018-09-28 13:18:43.124232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2018-09-28 13:18:43.351813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-28 13:18:43.351847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0 
2018-09-28 13:18:43.351859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N 
2018-09-28 13:18:43.352078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7558 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
#0: (32, 1)
#1: (32, 1)
#2: (32, 1)
#3: (32, 1)
#4: (32, 1)
#5: (32, 1)
#6: (32, 1)
#7: (32, 1)
#8: (32, 1)
#9: (32, 1)
#10: (32, 1)
#11: (32, 1)
#12: (32, 1)
#13: (32, 1)
#14: (32, 1)
#15: (32, 1)
#16: (32, 1)
#17: (32, 1)
#18: (32, 1)
#19: (32, 1)
#20: (32, 1)
#21: (32, 1)
#22: (32, 1)
#23: (32, 1)
#24: (32, 1)
#25: (32, 1)
#26: (32, 1)
#27: (32, 1)
#28: (32, 1)
#29: (32, 1)
#30: (32, 1)
#31: (32, 1)
#32: (32, 1)
#33: (32, 1)
#34: (32, 1)
#35: (32, 1)
*** Error in `python': malloc(): smallbin double linked list corrupted: 0x00007fb6c001c960 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5e4)[0x7fb9816b75e4]
/lib64/libc.so.6(+0x82d00)[0x7fb9816bad00]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7fb9816bd84c]
/home/hyabe/anaconda3/envs/work/bin/../lib/libstdc++.so.6(_Znwm+0x16)[0x7fb936087084]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow15OpKernelContext15allocate_outputEiRKNS_11TensorShapeEPPNS_6TensorENS_19AllocatorAttributesE+0x48)[0x7fb93f0c0608]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow15OpKernelContext15allocate_outputEiRKNS_11TensorShapeEPPNS_6TensorE+0xc5)[0x7fb93f0c0785]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow7MergeOp7ComputeEPNS_15OpKernelContextE+0xa4)[0x7fb9423854b4]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice13ComputeHelperEPNS_8OpKernelEPNS_15OpKernelContextE+0x37d)[0x7fb93f23ac9d]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice7ComputeEPNS_8OpKernelEPNS_15OpKernelContextE+0x8d)[0x7fb93f23b1dd]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x63a4bc)[0x7fb93f2844bc]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x63ae2a)[0x7fb93f284e2a]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN5Eigen26NonBlockingThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x21a)[0x7fb93f2f296a]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x32)[0x7fb93f2f1a12]
/home/hyabe/anaconda3/envs/work/bin/../lib/libstdc++.so.6(+0xb8678)[0x7fb9360a2678]
/lib64/libpthread.so.0(+0x7e25)[0x7fb981a0ce25]
/lib64/libc.so.6(clone+0x6d)[0x7fb981736bad]
======= Memory map: ========
200000000-200200000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
200200000-200400000 ---p 00000000 00:00 0 
200400000-200404000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
200404000-200600000 ---p 00000000 00:00 0 
200600000-200a00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
200a00000-201200000 ---p 00000000 00:00 0 
201200000-201204000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201204000-201400000 ---p 00000000 00:00 0 
201400000-201800000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201800000-201804000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201804000-201a00000 ---p 00000000 00:00 0 
201a00000-201e00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201e00000-201e04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201e04000-202000000 ---p 00000000 00:00 0 
202000000-202400000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202400000-202404000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202404000-202600000 ---p 00000000 00:00 0 
202600000-202a00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202a00000-202a04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202a04000-202c00000 ---p 00000000 00:00 0 
202c00000-203000000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203000000-203004000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203004000-203200000 ---p 00000000 00:00 0 
203200000-203600000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203600000-203604000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203604000-203800000 ---p 00000000 00:00 0 
203800000-203c00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203c00000-203c04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203c04000-203e00000 ---p 00000000 00:00 0 
203e00000-204200000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204200000-204204000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204204000-204400000 ---p 00000000 00:00 0 
204400000-204800000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204800000-204804000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204804000-204a00000 ---p 00000000 00:00 0 
204a00000-204e00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204e00000-204e04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204e04000-205000000 ---p 00000000 00:00 0 
205000000-205400000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205400000-205404000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205404000-205600000 ---p 00000000 00:00 0 
205600000-205a00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205a00000-205a04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205a04000-205c00000 ---p 00000000 00:00 0 
205c00000-206000000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206000000-206004000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206004000-206200000 ---p 00000000 00:00 0 
206200000-206600000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206600000-206604000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206604000-206800000 ---p 00000000 00:00 0 
206800000-206c00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206c00000-206e00000 ---p 00000000 00:00 0 
206e00000-207000000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
207000000-300200000 ---p 00000000 00:00 0 
10000000000-10204000000 ---p 00000000 00:00 0 
5596eb5dd000-5596eb89c000 r-xp 00000000 fd:02 86155766                   /home/hyabe/anaconda3/envs/work/bin/python3.6
5596eba9c000-5596eba9f000 r--p 002bf000 fd:02 86155766                   /home/hyabe/anaconda3/envs/work/bin/python3.6
5596eba9f000-5596ebb02000 rw-p 002c2000 fd:02 86155766                   /home/hyabe/anaconda3/envs/work/bin/python3.6
5596ebb02000-5596ebb33000 rw-p 00000000 00:00 0 
5596eda67000-5597016ee000 rw-p 00000000 00:00 0                          [heap]
7fb698000000-7fb69803a000 rw-p 00000000 00:00 0 
7fb69803a000-7fb69c000000 ---p 00000000 00:00 0 
7fb69c000000-7fb69c021000 rw-p 00000000 00:00 0 
7fb69c021000-7fb6a0000000 ---p 00000000 00:00 0 
7fb6a0000000-7fb6a0036000 rw-p 00000000 00:00 0 
7fb6a0036000-7fb6a4000000 ---p 00000000 00:00 0 
7fb6a4000000-7fb6a4049000 rw-p 00000000 00:00 0 
7fb6a4049000-7fb6a8000000 ---p 00000000 00:00 0 
7fb6a8000000-7fb6a8039000 rw-p 00000000 00:00 0 
7fb6a8039000-7fb6ac000000 ---p 00000000 00:00 0 
7fb6b0000000-7fb6b0036000 rw-p 00000000 00:00 0 
7fb6b0036000-7fb6b4000000 ---p 00000000 00:00 0 
7fb6b4000000-7fb6b403b000 rw-p 00000000 00:00 0 
7fb6b403b000-7fb6b8000000 ---p 00000000 00:00 0 
7fb6b8000000-7fb6b8037000 rw-p 00000000 00:00 0 
7fb6b8037000-7fb6bc000000 ---p 00000000 00:00 0 
7fb6c0000000-7fb6c003c000 rw-p 00000000 00:00 0 
7fb6c003c000-7fb6c4000000 ---p 00000000 00:00 0 
7fb6c5400000-7fb6c5a00000 rw-p 00000000 00:00 0 
7fb6c5ad4000-7fb6c5ad5000 ---p 00000000 00:00 0 
7fb6c5ad5000-7fb6c6326000 rw-p 00000000 00:00 0 
7fb6c6326000-7fb6c6327000 ---p 00000000 00:00 0 
7fb6c6327000-7fb6c6b78000 rw-p 00000000 00:00 0 
7fb6c6b78000-7fb6c6b79000 ---p 00000000 00:00 0 
7fb6c6b79000-7fb6d6000000 rw-p 00000000 00:00 0 
7fb6d6000000-7fb8ae800000 ---p 00000000 00:00 0 
7fb8ae800000-7fb8aea00000 rw-s 00000000 00:04 31968                      /dev/zero (deleted)
7fb8aea00000-7fb8aec00000 rw-s 00000000 00:04 29039                      /dev/zero (deleted)
7fb8aec00000-7fb8b0000000 ---p 00000000 00:00 0 
7fb8b0000000-7fb8b0021000 rw-p 00000000 00:00 0 
7fb8b0021000-7fb8b4000000 ---p 00000000 00:00 0 
7fb8b4000000-7fb8b4021000 rw-p 00000000 00:00 0 
7fb8b4021000-7fb8b8000000 ---p 00000000 00:00 0 
7fb8b8000000-7fb8bc000000 ---p 00000000 00:00 0 
7fb8bc000000-7fb8bc021000 rw-p 00000000 00:00 0 
7fb8bc021000-7fb8c0000000 ---p 00000000 00:00 0 
7fb8c0000000-7fb8c0001000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0001000-7fb8c0002000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0002000-7fb8c0003000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0003000-7fb8c0004000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0004000-7fb8c0005000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0005000-7fb8c0006000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0006000-7fb8c0007000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0007000-7fb8c0008000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0008000-7fb8c0009000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0009000-7fb8c000a000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000a000-7fb8c000b000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000b000-7fb8c000c000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000c000-7fb8c000d000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000d000-7fb8c000e000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000e000-7fb8c000f000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000f000-7fb8c0010000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0010000-7fb8d0000000 ---p 00000000 00:00 0 
7fb8d0000000-7fb8d0021000 rw-p 00000000 00:00 0 
7fb8d0021000-7fb8d4000000 ---p 00000000 00:00 0 
7fb8d4200000-7fb8d4600000 rw-p 00000000 00:00 0 
7fb8d470a000-7fb8d470b000 ---p 00000000 00:00 0 
7fb8d470b000-7fb8d4f5c000 rw-p 00000000 00:00 0 
7fb8d4f5c000-7fb8d4f5d000 ---p 00000000 00:00 0 
7fb8d4f5d000-7fb8d57ae000 rw-p 00000000 00:00 0 
7fb8d57ae000-7fb8d57af000 ---p 00000000 00:00 0 
7fb8d57af000-7fb8d6000000 rw-p 00000000 00:00 0 
7fb8d6000000-7fb8d6200000 ---p 00000000 00:00 0 
7fb8d6200000-7fb8d6400000 rw-s 00000000 00:04 31965                      /dev/zero (deleted)
7fb8d6400000-7fb8d6600000 rw-s 00000000 00:05 25687                      /dev/nvidiactl
7fb8d6600000-7fb8d6800000 rw-s 00000000 00:04 31966                      /dev/zero (deleted)
7fb8d6800000-7fb8d6a00000 rw-s 00000000 00:05 25687                      /dev/nvidiactl
7fb8d6a00000-7fb8d6c00000 ---p 00000000 00:00 0 
7fb8d6c00000-7fb8d6ed6000 rw-s 00000000 00:05 25687                      /dev/nvidiactl
7fb8d6ed6000-7fb8d8000000 ---p 00000000 00:00 0 
7fb8d8000000-7fb8d8021000 rw-p 00000000 00:00 0 
7fb8d8021000-7fb8dc000000 ---p 00000000 00:00 0 
7fb8dc200000-7fb8dc600000 rw-p 00000000 00:00 0 
7fb8dc70a000-7fb8dc70b000 ---p 00000000 00:00 0 
7fb8dc70b000-7fb8dcf5c000 rw-p 00000000 00:00 0 
7fb8dcf5c000-7fb8dcf5d000 ---p 00000000 00:00 0 
7fb8dcf5d000-7fb8dd7ae000 rw-p 00000000 00:00 0 
7fb8dd7ae000-7fb8dd7af000 ---p 00000000 00:00 0 
7fb8dd7af000-7fb8de000000 rw-p 00000000 00:00 0 
7fb8de000000-7fb8e4000000 ---p 00000000 00:00 0 
7fb8e4000000-7fb8e4021000 rw-p 00000000 00:00 0 
7fb8e4021000-7fb8e8000000 ---p 00000000 00:00 0 
7fb8e8000000-7fb8e8021000 rw-p 00000000 00:00 0 
7fb8e8021000-7fb8ec000000 ---p 00000000 00:00 0 
7fb8ec000000-7fb8ec021000 rw-p 00000000 00:00 0 
7fb8ec021000-7fb8f0000000 ---p 00000000 00:00 0 
7fb8f0000000-7fb8f0021000 rw-p 00000000 00:00 0 
7fb8f0021000-7fb8f4000000 ---p 00000000 00:00 0 
7fb8f4200000-7fb8f4400000 rw-p 00000000 00:00 0 
7fb8f45c2000-7fb8f45c3000 ---p 00000000 00:00 0 
7fb8f45c3000-7fb8f4e14000 rw-p 00000000 00:00 0 
7fb8f4e14000-7fb8f4e15000 ---p 00000000 00:00 0 
7fb8f4e15000-7fb8f5666000 rw-p 00000000 00:00 0 
7fb8f5666000-7fb8f5667000 ---p 00000000 00:00 0 
7fb8f5667000-7fb8f5eb8000 rw-p 00000000 00:00 0 
7fb8f5eb8000-7fb8f5eb9000 ---p 00000000 00:00 0 
7fb8f5eb9000-7fb8f670a000 rw-p 00000000 00:00 0 
7fb8f670a000-7fb8f670b000 ---p 00000000 00:00 0 
7fb8f670b000-7fb8f6f5c000 rw-p 00000000 00:00 0 
7fb8f6f5c000-7fb8f6f5d000 ---p 00000000 00:00 0 
7fb8f6f5d000-7fb8f77ae000 rw-p 00000000 00:00 0 
7fb8f77ae000-7fb8f77af00Aborted"><pre class="notranslate"><code class="notranslate">$ python nmstest.py
2018-09-28 13:18:42.760545: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-09-28 13:18:43.123328: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-09-28 13:18:43.124208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: 
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 7.93GiB freeMemory: 7.83GiB
2018-09-28 13:18:43.124232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2018-09-28 13:18:43.351813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-28 13:18:43.351847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0 
2018-09-28 13:18:43.351859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N 
2018-09-28 13:18:43.352078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7558 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
#0: (32, 1)
#1: (32, 1)
#2: (32, 1)
#3: (32, 1)
#4: (32, 1)
#5: (32, 1)
#6: (32, 1)
#7: (32, 1)
#8: (32, 1)
#9: (32, 1)
#10: (32, 1)
#11: (32, 1)
#12: (32, 1)
#13: (32, 1)
#14: (32, 1)
#15: (32, 1)
#16: (32, 1)
#17: (32, 1)
#18: (32, 1)
#19: (32, 1)
#20: (32, 1)
#21: (32, 1)
#22: (32, 1)
#23: (32, 1)
#24: (32, 1)
#25: (32, 1)
#26: (32, 1)
#27: (32, 1)
#28: (32, 1)
#29: (32, 1)
#30: (32, 1)
#31: (32, 1)
#32: (32, 1)
#33: (32, 1)
#34: (32, 1)
#35: (32, 1)
*** Error in `python': malloc(): smallbin double linked list corrupted: 0x00007fb6c001c960 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5e4)[0x7fb9816b75e4]
/lib64/libc.so.6(+0x82d00)[0x7fb9816bad00]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7fb9816bd84c]
/home/hyabe/anaconda3/envs/work/bin/../lib/libstdc++.so.6(_Znwm+0x16)[0x7fb936087084]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow15OpKernelContext15allocate_outputEiRKNS_11TensorShapeEPPNS_6TensorENS_19AllocatorAttributesE+0x48)[0x7fb93f0c0608]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow15OpKernelContext15allocate_outputEiRKNS_11TensorShapeEPPNS_6TensorE+0xc5)[0x7fb93f0c0785]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow7MergeOp7ComputeEPNS_15OpKernelContextE+0xa4)[0x7fb9423854b4]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice13ComputeHelperEPNS_8OpKernelEPNS_15OpKernelContextE+0x37d)[0x7fb93f23ac9d]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice7ComputeEPNS_8OpKernelEPNS_15OpKernelContextE+0x8d)[0x7fb93f23b1dd]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x63a4bc)[0x7fb93f2844bc]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x63ae2a)[0x7fb93f284e2a]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN5Eigen26NonBlockingThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x21a)[0x7fb93f2f296a]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x32)[0x7fb93f2f1a12]
/home/hyabe/anaconda3/envs/work/bin/../lib/libstdc++.so.6(+0xb8678)[0x7fb9360a2678]
/lib64/libpthread.so.0(+0x7e25)[0x7fb981a0ce25]
/lib64/libc.so.6(clone+0x6d)[0x7fb981736bad]
======= Memory map: ========
200000000-200200000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
200200000-200400000 ---p 00000000 00:00 0 
200400000-200404000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
200404000-200600000 ---p 00000000 00:00 0 
200600000-200a00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
200a00000-201200000 ---p 00000000 00:00 0 
201200000-201204000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201204000-201400000 ---p 00000000 00:00 0 
201400000-201800000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201800000-201804000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201804000-201a00000 ---p 00000000 00:00 0 
201a00000-201e00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201e00000-201e04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
201e04000-202000000 ---p 00000000 00:00 0 
202000000-202400000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202400000-202404000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202404000-202600000 ---p 00000000 00:00 0 
202600000-202a00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202a00000-202a04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
202a04000-202c00000 ---p 00000000 00:00 0 
202c00000-203000000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203000000-203004000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203004000-203200000 ---p 00000000 00:00 0 
203200000-203600000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203600000-203604000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203604000-203800000 ---p 00000000 00:00 0 
203800000-203c00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203c00000-203c04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
203c04000-203e00000 ---p 00000000 00:00 0 
203e00000-204200000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204200000-204204000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204204000-204400000 ---p 00000000 00:00 0 
204400000-204800000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204800000-204804000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204804000-204a00000 ---p 00000000 00:00 0 
204a00000-204e00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204e00000-204e04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
204e04000-205000000 ---p 00000000 00:00 0 
205000000-205400000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205400000-205404000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205404000-205600000 ---p 00000000 00:00 0 
205600000-205a00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205a00000-205a04000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
205a04000-205c00000 ---p 00000000 00:00 0 
205c00000-206000000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206000000-206004000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206004000-206200000 ---p 00000000 00:00 0 
206200000-206600000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206600000-206604000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206604000-206800000 ---p 00000000 00:00 0 
206800000-206c00000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
206c00000-206e00000 ---p 00000000 00:00 0 
206e00000-207000000 rw-s 00000000 00:05 25687                            /dev/nvidiactl
207000000-300200000 ---p 00000000 00:00 0 
10000000000-10204000000 ---p 00000000 00:00 0 
5596eb5dd000-5596eb89c000 r-xp 00000000 fd:02 86155766                   /home/hyabe/anaconda3/envs/work/bin/python3.6
5596eba9c000-5596eba9f000 r--p 002bf000 fd:02 86155766                   /home/hyabe/anaconda3/envs/work/bin/python3.6
5596eba9f000-5596ebb02000 rw-p 002c2000 fd:02 86155766                   /home/hyabe/anaconda3/envs/work/bin/python3.6
5596ebb02000-5596ebb33000 rw-p 00000000 00:00 0 
5596eda67000-5597016ee000 rw-p 00000000 00:00 0                          [heap]
7fb698000000-7fb69803a000 rw-p 00000000 00:00 0 
7fb69803a000-7fb69c000000 ---p 00000000 00:00 0 
7fb69c000000-7fb69c021000 rw-p 00000000 00:00 0 
7fb69c021000-7fb6a0000000 ---p 00000000 00:00 0 
7fb6a0000000-7fb6a0036000 rw-p 00000000 00:00 0 
7fb6a0036000-7fb6a4000000 ---p 00000000 00:00 0 
7fb6a4000000-7fb6a4049000 rw-p 00000000 00:00 0 
7fb6a4049000-7fb6a8000000 ---p 00000000 00:00 0 
7fb6a8000000-7fb6a8039000 rw-p 00000000 00:00 0 
7fb6a8039000-7fb6ac000000 ---p 00000000 00:00 0 
7fb6b0000000-7fb6b0036000 rw-p 00000000 00:00 0 
7fb6b0036000-7fb6b4000000 ---p 00000000 00:00 0 
7fb6b4000000-7fb6b403b000 rw-p 00000000 00:00 0 
7fb6b403b000-7fb6b8000000 ---p 00000000 00:00 0 
7fb6b8000000-7fb6b8037000 rw-p 00000000 00:00 0 
7fb6b8037000-7fb6bc000000 ---p 00000000 00:00 0 
7fb6c0000000-7fb6c003c000 rw-p 00000000 00:00 0 
7fb6c003c000-7fb6c4000000 ---p 00000000 00:00 0 
7fb6c5400000-7fb6c5a00000 rw-p 00000000 00:00 0 
7fb6c5ad4000-7fb6c5ad5000 ---p 00000000 00:00 0 
7fb6c5ad5000-7fb6c6326000 rw-p 00000000 00:00 0 
7fb6c6326000-7fb6c6327000 ---p 00000000 00:00 0 
7fb6c6327000-7fb6c6b78000 rw-p 00000000 00:00 0 
7fb6c6b78000-7fb6c6b79000 ---p 00000000 00:00 0 
7fb6c6b79000-7fb6d6000000 rw-p 00000000 00:00 0 
7fb6d6000000-7fb8ae800000 ---p 00000000 00:00 0 
7fb8ae800000-7fb8aea00000 rw-s 00000000 00:04 31968                      /dev/zero (deleted)
7fb8aea00000-7fb8aec00000 rw-s 00000000 00:04 29039                      /dev/zero (deleted)
7fb8aec00000-7fb8b0000000 ---p 00000000 00:00 0 
7fb8b0000000-7fb8b0021000 rw-p 00000000 00:00 0 
7fb8b0021000-7fb8b4000000 ---p 00000000 00:00 0 
7fb8b4000000-7fb8b4021000 rw-p 00000000 00:00 0 
7fb8b4021000-7fb8b8000000 ---p 00000000 00:00 0 
7fb8b8000000-7fb8bc000000 ---p 00000000 00:00 0 
7fb8bc000000-7fb8bc021000 rw-p 00000000 00:00 0 
7fb8bc021000-7fb8c0000000 ---p 00000000 00:00 0 
7fb8c0000000-7fb8c0001000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0001000-7fb8c0002000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0002000-7fb8c0003000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0003000-7fb8c0004000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0004000-7fb8c0005000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0005000-7fb8c0006000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0006000-7fb8c0007000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0007000-7fb8c0008000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0008000-7fb8c0009000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0009000-7fb8c000a000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000a000-7fb8c000b000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000b000-7fb8c000c000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000c000-7fb8c000d000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000d000-7fb8c000e000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000e000-7fb8c000f000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c000f000-7fb8c0010000 rw-s 00000000 00:05 25688                      /dev/nvidia0
7fb8c0010000-7fb8d0000000 ---p 00000000 00:00 0 
7fb8d0000000-7fb8d0021000 rw-p 00000000 00:00 0 
7fb8d0021000-7fb8d4000000 ---p 00000000 00:00 0 
7fb8d4200000-7fb8d4600000 rw-p 00000000 00:00 0 
7fb8d470a000-7fb8d470b000 ---p 00000000 00:00 0 
7fb8d470b000-7fb8d4f5c000 rw-p 00000000 00:00 0 
7fb8d4f5c000-7fb8d4f5d000 ---p 00000000 00:00 0 
7fb8d4f5d000-7fb8d57ae000 rw-p 00000000 00:00 0 
7fb8d57ae000-7fb8d57af000 ---p 00000000 00:00 0 
7fb8d57af000-7fb8d6000000 rw-p 00000000 00:00 0 
7fb8d6000000-7fb8d6200000 ---p 00000000 00:00 0 
7fb8d6200000-7fb8d6400000 rw-s 00000000 00:04 31965                      /dev/zero (deleted)
7fb8d6400000-7fb8d6600000 rw-s 00000000 00:05 25687                      /dev/nvidiactl
7fb8d6600000-7fb8d6800000 rw-s 00000000 00:04 31966                      /dev/zero (deleted)
7fb8d6800000-7fb8d6a00000 rw-s 00000000 00:05 25687                      /dev/nvidiactl
7fb8d6a00000-7fb8d6c00000 ---p 00000000 00:00 0 
7fb8d6c00000-7fb8d6ed6000 rw-s 00000000 00:05 25687                      /dev/nvidiactl
7fb8d6ed6000-7fb8d8000000 ---p 00000000 00:00 0 
7fb8d8000000-7fb8d8021000 rw-p 00000000 00:00 0 
7fb8d8021000-7fb8dc000000 ---p 00000000 00:00 0 
7fb8dc200000-7fb8dc600000 rw-p 00000000 00:00 0 
7fb8dc70a000-7fb8dc70b000 ---p 00000000 00:00 0 
7fb8dc70b000-7fb8dcf5c000 rw-p 00000000 00:00 0 
7fb8dcf5c000-7fb8dcf5d000 ---p 00000000 00:00 0 
7fb8dcf5d000-7fb8dd7ae000 rw-p 00000000 00:00 0 
7fb8dd7ae000-7fb8dd7af000 ---p 00000000 00:00 0 
7fb8dd7af000-7fb8de000000 rw-p 00000000 00:00 0 
7fb8de000000-7fb8e4000000 ---p 00000000 00:00 0 
7fb8e4000000-7fb8e4021000 rw-p 00000000 00:00 0 
7fb8e4021000-7fb8e8000000 ---p 00000000 00:00 0 
7fb8e8000000-7fb8e8021000 rw-p 00000000 00:00 0 
7fb8e8021000-7fb8ec000000 ---p 00000000 00:00 0 
7fb8ec000000-7fb8ec021000 rw-p 00000000 00:00 0 
7fb8ec021000-7fb8f0000000 ---p 00000000 00:00 0 
7fb8f0000000-7fb8f0021000 rw-p 00000000 00:00 0 
7fb8f0021000-7fb8f4000000 ---p 00000000 00:00 0 
7fb8f4200000-7fb8f4400000 rw-p 00000000 00:00 0 
7fb8f45c2000-7fb8f45c3000 ---p 00000000 00:00 0 
7fb8f45c3000-7fb8f4e14000 rw-p 00000000 00:00 0 
7fb8f4e14000-7fb8f4e15000 ---p 00000000 00:00 0 
7fb8f4e15000-7fb8f5666000 rw-p 00000000 00:00 0 
7fb8f5666000-7fb8f5667000 ---p 00000000 00:00 0 
7fb8f5667000-7fb8f5eb8000 rw-p 00000000 00:00 0 
7fb8f5eb8000-7fb8f5eb9000 ---p 00000000 00:00 0 
7fb8f5eb9000-7fb8f670a000 rw-p 00000000 00:00 0 
7fb8f670a000-7fb8f670b000 ---p 00000000 00:00 0 
7fb8f670b000-7fb8f6f5c000 rw-p 00000000 00:00 0 
7fb8f6f5c000-7fb8f6f5d000 ---p 00000000 00:00 0 
7fb8f6f5d000-7fb8f77ae000 rw-p 00000000 00:00 0 
7fb8f77ae000-7fb8f77af00Aborted
</code></pre></div>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: archlinux</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>:n/a</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:binary</li>
<li><strong>TensorFlow version (use command below)</strong>:b'v1.9.0-rc2-5276-ge57874169f' 1.12.0-dev20181004</li>
<li><strong>Python version</strong>:3.6</li>
<li><strong>Bazel version (if compiling from source)</strong>:n/a</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:n/a</li>
<li><strong>CUDA/cuDNN version</strong>:9.0</li>
<li><strong>GPU model and memory</strong>:1080Ti</li>
<li><strong>Exact command to reproduce</strong>:below</li>
</ul>
<p dir="auto">This code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import numpy as np

def f(boxes, scores):
    def f(X):
        prob, box = X
        output_shape = tf.shape(prob)
        ids = tf.reshape(tf.where(prob &gt; 0.05), [-1])
        prob = tf.gather(prob, ids)
        box = tf.gather(box, ids)
        # prob = tf.Print(prob, [box, prob], summarize=100, message='boxandprob')
        selection = tf.image.non_max_suppression(box, prob, 100, 0.5)
        selection = tf.to_int32(tf.gather(ids, selection))
        selection = tf.Print(selection, [ids, selection], summarize=100, message='ids_selection_2')
        sorted_selection = -tf.nn.top_k(-selection, k=tf.size(selection))[0]
        mask = tf.sparse_to_dense(
            sparse_indices=sorted_selection,
            output_shape=output_shape,
            sparse_values=True,
            default_value=False)
        return mask

    masks = tf.map_fn(f, (scores, boxes), dtype=tf.bool, parallel_iterations=10)     # #cat x N
    return masks

with tf.device('/gpu:0'):
    boxes = tf.placeholder(tf.float32, (80, None, 4), name='boxes')
    scores = tf.placeholder(tf.float32, (80, None), name='scores')
    outs = f(boxes, scores)

config = tf.ConfigProto()
config.allow_soft_placement = True
sess = tf.Session(config=config)
data = dict(np.load('debug.npz'))
for k in range(1000):
    sess.run(outs, feed_dict={boxes: data['boxes'].transpose(1, 0, 2)[1:, :, :], scores: data['scores'][:, 1:].T})
    print(k)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-k">def</span> <span class="pl-en">f</span>(<span class="pl-s1">boxes</span>, <span class="pl-s1">scores</span>):
    <span class="pl-k">def</span> <span class="pl-en">f</span>(<span class="pl-v">X</span>):
        <span class="pl-s1">prob</span>, <span class="pl-s1">box</span> <span class="pl-c1">=</span> <span class="pl-v">X</span>
        <span class="pl-s1">output_shape</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">shape</span>(<span class="pl-s1">prob</span>)
        <span class="pl-s1">ids</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">tf</span>.<span class="pl-en">where</span>(<span class="pl-s1">prob</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0.05</span>), [<span class="pl-c1">-</span><span class="pl-c1">1</span>])
        <span class="pl-s1">prob</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">gather</span>(<span class="pl-s1">prob</span>, <span class="pl-s1">ids</span>)
        <span class="pl-s1">box</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">gather</span>(<span class="pl-s1">box</span>, <span class="pl-s1">ids</span>)
        <span class="pl-c"># prob = tf.Print(prob, [box, prob], summarize=100, message='boxandprob')</span>
        <span class="pl-s1">selection</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">image</span>.<span class="pl-en">non_max_suppression</span>(<span class="pl-s1">box</span>, <span class="pl-s1">prob</span>, <span class="pl-c1">100</span>, <span class="pl-c1">0.5</span>)
        <span class="pl-s1">selection</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">to_int32</span>(<span class="pl-s1">tf</span>.<span class="pl-en">gather</span>(<span class="pl-s1">ids</span>, <span class="pl-s1">selection</span>))
        <span class="pl-s1">selection</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Print</span>(<span class="pl-s1">selection</span>, [<span class="pl-s1">ids</span>, <span class="pl-s1">selection</span>], <span class="pl-s1">summarize</span><span class="pl-c1">=</span><span class="pl-c1">100</span>, <span class="pl-s1">message</span><span class="pl-c1">=</span><span class="pl-s">'ids_selection_2'</span>)
        <span class="pl-s1">sorted_selection</span> <span class="pl-c1">=</span> <span class="pl-c1">-</span><span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-en">top_k</span>(<span class="pl-c1">-</span><span class="pl-s1">selection</span>, <span class="pl-s1">k</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">size</span>(<span class="pl-s1">selection</span>))[<span class="pl-c1">0</span>]
        <span class="pl-s1">mask</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">sparse_to_dense</span>(
            <span class="pl-s1">sparse_indices</span><span class="pl-c1">=</span><span class="pl-s1">sorted_selection</span>,
            <span class="pl-s1">output_shape</span><span class="pl-c1">=</span><span class="pl-s1">output_shape</span>,
            <span class="pl-s1">sparse_values</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
            <span class="pl-s1">default_value</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
        <span class="pl-k">return</span> <span class="pl-s1">mask</span>

    <span class="pl-s1">masks</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">map_fn</span>(<span class="pl-s1">f</span>, (<span class="pl-s1">scores</span>, <span class="pl-s1">boxes</span>), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">bool</span>, <span class="pl-s1">parallel_iterations</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)     <span class="pl-c"># #cat x N</span>
    <span class="pl-k">return</span> <span class="pl-s1">masks</span>

<span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">device</span>(<span class="pl-s">'/gpu:0'</span>):
    <span class="pl-s1">boxes</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>, (<span class="pl-c1">80</span>, <span class="pl-c1">None</span>, <span class="pl-c1">4</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'boxes'</span>)
    <span class="pl-s1">scores</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>, (<span class="pl-c1">80</span>, <span class="pl-c1">None</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'scores'</span>)
    <span class="pl-s1">outs</span> <span class="pl-c1">=</span> <span class="pl-en">f</span>(<span class="pl-s1">boxes</span>, <span class="pl-s1">scores</span>)

<span class="pl-s1">config</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">ConfigProto</span>()
<span class="pl-s1">config</span>.<span class="pl-s1">allow_soft_placement</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
<span class="pl-s1">sess</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>(<span class="pl-s1">config</span><span class="pl-c1">=</span><span class="pl-s1">config</span>)
<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-en">dict</span>(<span class="pl-s1">np</span>.<span class="pl-en">load</span>(<span class="pl-s">'debug.npz'</span>))
<span class="pl-k">for</span> <span class="pl-s1">k</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">1000</span>):
    <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">outs</span>, <span class="pl-s1">feed_dict</span><span class="pl-c1">=</span>{<span class="pl-s1">boxes</span>: <span class="pl-s1">data</span>[<span class="pl-s">'boxes'</span>].<span class="pl-en">transpose</span>(<span class="pl-c1">1</span>, <span class="pl-c1">0</span>, <span class="pl-c1">2</span>)[<span class="pl-c1">1</span>:, :, :], <span class="pl-s1">scores</span>: <span class="pl-s1">data</span>[<span class="pl-s">'scores'</span>][:, <span class="pl-c1">1</span>:].<span class="pl-v">T</span>})
    <span class="pl-en">print</span>(<span class="pl-s1">k</span>)</pre></div>
<p dir="auto">causes segmentation fault on tf-nightly-gpu, as well as tensorflow-gpu==1.11.0. It works on 1.10.<br>
It needs the data file <code class="notranslate">debug.npz</code> here:<br>
<a href="https://github.com/tensorflow/tensorflow/files/2448247/debug.zip">debug.zip</a></p>
<p dir="auto">Note:</p>
<ol dir="auto">
<li>I tested on two machines, an error happen in &gt;90% runs.</li>
<li>The code was distilled from the bug report about MaskRCNN evaluation <a href="https://github.com/tensorpack/tensorpack/issues/919" data-hovercard-type="issue" data-hovercard-url="/tensorpack/tensorpack/issues/919/hovercard">here</a>. The original bug report does not always segfault, but occasionally crash with other <strong>different</strong> unreasonable TF internal errors, such as:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="InvalidArgumentError (see above for traceback): scores has incompatible shape
         [[node map/while/non_max_suppression/NonMaxSuppressionV3 (defined at bug.py:15)  = NonMaxSuppressionV3[T=DT_FLOAT, _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](map/while/Gather
V2_1/_29, map/while/GatherV2/_31, map/while/non_max_suppression/NonMaxSuppressionV3/max_output_size/_33, map/while/non_max_suppression/iou_threshold/_35, map/while/non_max_suppression/score_thresh
old/_37)]]"><pre class="notranslate"><code class="notranslate">InvalidArgumentError (see above for traceback): scores has incompatible shape
         [[node map/while/non_max_suppression/NonMaxSuppressionV3 (defined at bug.py:15)  = NonMaxSuppressionV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](map/while/Gather
V2_1/_29, map/while/GatherV2/_31, map/while/non_max_suppression/NonMaxSuppressionV3/max_output_size/_33, map/while/non_max_suppression/iou_threshold/_35, map/while/non_max_suppression/score_thresh
old/_37)]]
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2018-10-04 14:59:14.736180: F tensorflow/core/common_runtime/bfc_allocator.cc:458] Check failed: c-&gt;in_use() &amp;&amp; (c-&gt;bin_num == kInvalidBinNum)                                                     "><pre class="notranslate"><code class="notranslate">2018-10-04 14:59:14.736180: F tensorflow/core/common_runtime/bfc_allocator.cc:458] Check failed: c-&gt;in_use() &amp;&amp; (c-&gt;bin_num == kInvalidBinNum)                                                     
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2018-10-04 14:59:49.523436: F tensorflow/core/common_runtime/bfc_allocator.cc:380] Check failed: h != kInvalidChunkHandle "><pre class="notranslate"><code class="notranslate">2018-10-04 14:59:49.523436: F tensorflow/core/common_runtime/bfc_allocator.cc:380] Check failed: h != kInvalidChunkHandle 
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2018-10-05 00:12:03.720295: F ./tensorflow/core/framework/tensor.h:643] Check failed: new_num_elements == NumElements() (39 vs. 0)"><pre class="notranslate"><code class="notranslate">2018-10-05 00:12:03.720295: F ./tensorflow/core/framework/tensor.h:643] Check failed: new_num_elements == NumElements() (39 vs. 0)
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
InvalidArgumentError (see above for traceback): indices[1] = [0] is repeated
         [[{{node map/while/SparseToDense}} = SparseToDense[T=DT_BOOL, Tindices=DT_INT32, _class=[&quot;loc:@map/while/TensorArrayWrite/TensorArrayWriteV3&quot;], validate_indices=true, _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](map/while/Neg_1/_51, map/while/Shape/_53, map/while/SparseToDense/sparse_values/_55, map/while/SparseToDense/default_value/_57)]]
         [[{{node map/while/SparseToDense/sparse_values/_54}} = _Send[T=DT_BOOL, client_terminated=false, recv_device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;, send_device=&quot;/job:localhost/replica:0/task:0/device:GPU:0&quot;, send_device_incarnation=1, tensor_name=&quot;edge_111_map/while/SparseToDense/sparse_values&quot;, _device=&quot;/job:localhost/replica:0/task:0/device:GPU:0&quot;](map/while/SparseToDense/sparse_values)]]"><pre class="notranslate"><code class="notranslate">
InvalidArgumentError (see above for traceback): indices[1] = [0] is repeated
         [[{{node map/while/SparseToDense}} = SparseToDense[T=DT_BOOL, Tindices=DT_INT32, _class=["loc:@map/while/TensorArrayWrite/TensorArrayWriteV3"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](map/while/Neg_1/_51, map/while/Shape/_53, map/while/SparseToDense/sparse_values/_55, map/while/SparseToDense/default_value/_57)]]
         [[{{node map/while/SparseToDense/sparse_values/_54}} = _Send[T=DT_BOOL, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_111_map/while/SparseToDense/sparse_values", _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/SparseToDense/sparse_values)]]
</code></pre></div>
<p dir="auto">After distilled to this small repro, it seems to mostly do segfault. But the above error messages might help. Seems like a memory corruption.</p>