<p dir="auto">I was trying to add profiler to the existing tensorflow object detection API, the following part of code is added to </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/tensorflow/tensorflow/blob/dfa0f87e333c63277b7916a8ac4c56bf61daf1ac/tensorflow/contrib/slim/python/slim/learning.py#L489">tensorflow/tensorflow/contrib/slim/python/slim/learning.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 489
      in
      <a data-pjax="true" class="commit-tease-sha" href="/tensorflow/tensorflow/commit/dfa0f87e333c63277b7916a8ac4c56bf61daf1ac">dfa0f87</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L489" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="489"></td>
          <td id="LC489" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">  </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  if run_metadata is not None:
    logging.info('Added profiling by User')
    ProfileOptionBuilder = profiler.ProfileOptionBuilder
    opts = ProfileOptionBuilder(ProfileOptionBuilder.time_and_memory()).with_step(np_global_step).with_timeline_output(train_step_kwargs['logdir']+'/profile.json').build()
    logging.info(train_step_kwargs['logdir']+'/profile.json')
    profiler.profile(
        ops.get_default_graph(),
        run_meta=run_metadata,
        cmd='graph',
        options=opts)"><pre class="notranslate"><code class="notranslate">  if run_metadata is not None:
    logging.info('Added profiling by User')
    ProfileOptionBuilder = profiler.ProfileOptionBuilder
    opts = ProfileOptionBuilder(ProfileOptionBuilder.time_and_memory()).with_step(np_global_step).with_timeline_output(train_step_kwargs['logdir']+'/profile.json').build()
    logging.info(train_step_kwargs['logdir']+'/profile.json')
    profiler.profile(
        ops.get_default_graph(),
        run_meta=run_metadata,
        cmd='graph',
        options=opts)
</code></pre></div>
<p dir="auto">but when I was trying to run the training for resnet_101_coco with no other changes, the training progress with no error, but the profiler output incompatible shapes:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/pool1/MaxPool incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/shortcut/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/shortcut/MaxPool incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/Pad incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 152, 229) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_4/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node "><pre class="notranslate"><code class="notranslate">Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/pool1/MaxPool incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/shortcut/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/shortcut/MaxPool incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/Pad incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 152, 229) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_4/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node 
</code></pre></div>
<p dir="auto">Hi! I was using the official minist summaries example which uses run_metadata to track model runtime statistics:</p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_with_summaries.py">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_with_summaries.py</a></p>
<p dir="auto">I tried to add simple profiler python API as suggest by</p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/profiler/g3doc/python_api.md">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/profiler/g3doc/python_api.md</a></p>
<p dir="auto">So, my code is:<br>
<a href="https://github.com/MrWanter/mnist-proflie/blob/c0f5b59403fbf3258e8236fa7c983750623354e6/mnist-profile.py#L175-L184">https://github.com/MrWanter/mnist-proflie/blob/c0f5b59403fbf3258e8236fa7c983750623354e6/mnist-profile.py#L175-L184</a></p>
<p dir="auto">but running the model gives</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Added profiling by user
Parsing Inputs...
Incomplete shape.
Incomplete shape.
Options -step=199 not found.
Available steps: 0 
Adding run metadata for 199"><pre class="notranslate"><code class="notranslate">Added profiling by user
Parsing Inputs...
Incomplete shape.
Incomplete shape.
Options -step=199 not found.
Available steps: 0 
Adding run metadata for 199
</code></pre></div>
<p dir="auto">but tensorboard shows the memory consumption of nodes, which suggests runtime statistics are captured:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/18298163/41357872-b3d623fe-6f59-11e8-91c2-539f55340e8a.png"><img src="https://user-images.githubusercontent.com/18298163/41357872-b3d623fe-6f59-11e8-91c2-539f55340e8a.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">what is incomplete shape, the input shape has None for batch size, but since it is runtime statistics, I thought it would work, but it does not, can someone help?</p>
<p dir="auto">As in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="331804891" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/19962" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/19962/hovercard" href="https://github.com/tensorflow/tensorflow/issues/19962">#19962</a>, it is hard to know how to deal with profiler problem since documentation is not enough and    codes are hard to read as it is implemented with C++ and heavily wrapped</p>
<p dir="auto"><strong>UPDATA</strong>:<br>
Th problem is caused by input shape of None for batch size, change it to fixed shape solve the problem, but there would inevitably be a large amount of None in tensors since we have to deal with variable sized inputs, in those cases, it should be reasonable for the profiler to capture runtime statistics, so how can we do it? where am I wrong?</p>
<p dir="auto"><strong>UPDATE</strong><br>
Found a quote on the official profiler doc:</p>
<blockquote>
<p dir="auto">It must have known "shape" information for RegisterStatistics('flops') to calculate the statistics. It is suggested to pass in -run_meta_path if shape is only known during runtime. tfprof can fill in the missing shape with the runtime shape information from RunMetadata. Hence, it is suggested to use -account_displayed_op_only option so that you know the statistics are only for the operations printed out.</p>
</blockquote>
<p dir="auto">seems profiler can fill in the missing shape with runtime shape information, but how to pass in <code class="notranslate">- run_meta_path</code>,  I did not find anything relevant in the profiler options:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="      self._options = {'max_depth': 100,
                       'min_bytes': 0,
                       'min_micros': 0,
                       'min_params': 0,
                       'min_float_ops': 0,
                       'min_occurrence': 0,
                       'order_by': 'name',
                       'account_type_regexes': ['.*'],
                       'start_name_regexes': ['.*'],
                       'trim_name_regexes': [],
                       'show_name_regexes': ['.*'],
                       'hide_name_regexes': [],
                       'account_displayed_op_only': False,
                       'select': ['micros'],
                       'step': -1,
                       'output': 'stdout'}"><pre class="notranslate"><code class="notranslate">      self._options = {'max_depth': 100,
                       'min_bytes': 0,
                       'min_micros': 0,
                       'min_params': 0,
                       'min_float_ops': 0,
                       'min_occurrence': 0,
                       'order_by': 'name',
                       'account_type_regexes': ['.*'],
                       'start_name_regexes': ['.*'],
                       'trim_name_regexes': [],
                       'show_name_regexes': ['.*'],
                       'hide_name_regexes': [],
                       'account_displayed_op_only': False,
                       'select': ['micros'],
                       'step': -1,
                       'output': 'stdout'}
</code></pre></div>