<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-02.html#rel_19-02" rel="nofollow">nvidia-docker <code class="notranslate">tensor:19.02-py3</code> (Ubuntu 16.04)</a></li>
<li>TensorFlow installed from (source or binary): Source</li>
<li>TensorFlow version (use command below): v1.14.0-rc1-22-gaf24dc91b5 1.14.0</li>
<li>Python version: Python 3.5.2</li>
<li>Bazel version (if compiling from source): NA</li>
<li>GCC/Compiler version (if compiling from source): NA</li>
<li>CUDA/cuDNN version: 10.0.130 / 7.4.2</li>
<li>GPU model and memory: Tesla V100 32GB</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I am trying to optimize (decrease) the inference time and model size of my Tiny Yolov3 model. I currently have it as a frozen graph. When I run the timing and evaluation script from <a href="https://github.com/tensorflow/models/blob/master/research/tensorrt/tensorrt.py">here</a>, fps actually decreases as the optimizations are tried, and file sizes stay constant.</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
I expect file sizes and inference time to decrease.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
I used <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-02.html#rel_19-02" rel="nofollow">this</a> nvidia-docker image.</p>
<p dir="auto"><a href="https://github.com/tensorflow/models/blob/master/research/tensorrt/tensorrt.py">This</a> script with a small preprocessing change:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def resize_image(image, target_height, target_width):
    image = image.resize((target_width, target_height), Image.BICUBIC).convert('RGB')
    return np.array(image)

def preprocess_image(file_name, output_height=416, output_width=416,
                     num_channels=3):
    return resize_image(Image.open(file_name), output_height, output_width).astype('float32')"><pre class="notranslate"><code class="notranslate">def resize_image(image, target_height, target_width):
    image = image.resize((target_width, target_height), Image.BICUBIC).convert('RGB')
    return np.array(image)

def preprocess_image(file_name, output_height=416, output_width=416,
                     num_channels=3):
    return resize_image(Image.open(file_name), output_height, output_width).astype('float32')
</code></pre></div>
<p dir="auto">I also removed the printing of predictions, as the script is built for a classifier rather than a detector.</p>
<p dir="auto"><strong>Other info / logs</strong><br>
Command executed (image file is attached; model is <a href="https://drive.google.com/file/d/1Bgj9h6TJLwedtrhnritRs9eYm_iczG4v/view" rel="nofollow">here</a>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python timed.py --frozen_graph=tiny-yolov3_frozen.pb \
--image_file=11075842.jpg \
--native --fp32 --fp16 --int8 \
--output_dir=/workspace \
--input_node=inputs --output_node=output_boxes"><pre class="notranslate"><code class="notranslate">python timed.py --frozen_graph=tiny-yolov3_frozen.pb \
--image_file=11075842.jpg \
--native --fp32 --fp16 --int8 \
--output_dir=/workspace \
--input_node=inputs --output_node=output_boxes
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="==========================
network: native_tiny-yolov3_frozen.pb,   batchsize 256, steps 100
  fps   median: 993.1,  mean: 959.8,    uncertainty: 8.9,       jitter: 7.7
  latency       median: 0.25778,        mean: 0.26986,  99th_p: 0.43523,        99th_uncertainty: 0.05783

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb,       batchsize 256, steps 100
  fps   median: 912.7,  mean: 896.3,    uncertainty: 8.2,       jitter: 4.6
  latency       median: 0.28048,        mean: 0.29338,  99th_p: 0.42830,        99th_uncertainty: 0.43877

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb,       batchsize 256, steps 100
  fps   median: 823.5,  mean: 831.9,    uncertainty: 8.4,       jitter: 96.2
  latency       median: 0.31087,        mean: 0.31129,  99th_p: 0.43070,        99th_uncertainty: 0.07565

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb,       batchsize 256, steps 100
  fps   median: 803.3,  mean: 820.6,    uncertainty: 9.9,       jitter: 19.9
  latency       median: 0.31867,        mean: 0.31814,  99th_p: 0.60985,        99th_uncertainty: 0.01011"><pre class="notranslate"><code class="notranslate">==========================
network: native_tiny-yolov3_frozen.pb,   batchsize 256, steps 100
  fps   median: 993.1,  mean: 959.8,    uncertainty: 8.9,       jitter: 7.7
  latency       median: 0.25778,        mean: 0.26986,  99th_p: 0.43523,        99th_uncertainty: 0.05783

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb,       batchsize 256, steps 100
  fps   median: 912.7,  mean: 896.3,    uncertainty: 8.2,       jitter: 4.6
  latency       median: 0.28048,        mean: 0.29338,  99th_p: 0.42830,        99th_uncertainty: 0.43877

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb,       batchsize 256, steps 100
  fps   median: 823.5,  mean: 831.9,    uncertainty: 8.4,       jitter: 96.2
  latency       median: 0.31087,        mean: 0.31129,  99th_p: 0.43070,        99th_uncertainty: 0.07565

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb,       batchsize 256, steps 100
  fps   median: 803.3,  mean: 820.6,    uncertainty: 9.9,       jitter: 19.9
  latency       median: 0.31867,        mean: 0.31814,  99th_p: 0.60985,        99th_uncertainty: 0.01011
</code></pre></div>
<h2 dir="auto"><strong>System information</strong></h2>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04, <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-02.html#rel_19-02" rel="nofollow">nvcr.io/nvidia/tensorrt:19.02-py3</a></li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): v1.12.1-7024-g24b3e6cf73 1.15.0-dev20190725</li>
<li>Python version: 3.5.2</li>
<li>Bazel version (if compiling from source): NA</li>
<li>GCC/Compiler version (if compiling from source): NA</li>
<li>CUDA/cuDNN version: 10.0.130 / 7.4.2 as per the above linked container</li>
<li>GPU model and memory: Tesla V100 32GB</li>
</ul>
<p dir="auto">Relevant output from <code class="notranslate">pip freeze</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tf-estimator-nightly==1.14.0.dev2019072201
tf-nightly-gpu==1.15.0.dev20190725"><pre class="notranslate"><code class="notranslate">tf-estimator-nightly==1.14.0.dev2019072201
tf-nightly-gpu==1.15.0.dev20190725
</code></pre></div>
<h2 dir="auto"><strong>Describe the current behavior</strong></h2>
<p dir="auto"><strong>Inference Speed (frames per second)</strong></p>
<table role="table">
<thead>
<tr>
<th>Model</th>
<th>Native</th>
<th>FP32</th>
<th>FP16</th>
<th>INT8</th>
</tr>
</thead>
<tbody>
<tr>
<td>tiny-yolo</td>
<td>348</td>
<td>333</td>
<td>402</td>
<td>415</td>
</tr>
<tr>
<td>big-yolo</td>
<td>125</td>
<td>140</td>
<td>243</td>
<td>208</td>
</tr>
</tbody>
</table>
<p dir="auto"><strong>1. Why is there a slowdown for tiny native-&gt;FP32?</strong> (<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pooyadavoodi/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pooyadavoodi">@pooyadavoodi</a> same as <a href="https://github.com/tensorflow/tensorflow/issues/30717#issuecomment-513938562" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/30717/hovercard">here</a>)<br>
<strong>2. Why is this slowdown not consistent for big?</strong><br>
<strong>3. Why is there a slowdown for big FP16-&gt;INT8?</strong></p>
<p dir="auto"><strong>Model Size (megabytes)</strong></p>
<table role="table">
<thead>
<tr>
<th>Model</th>
<th>Native</th>
<th>FP32</th>
<th>FP16</th>
<th>INT8</th>
</tr>
</thead>
<tbody>
<tr>
<td>tiny-yolo</td>
<td>35</td>
<td>67</td>
<td>44</td>
<td>51</td>
</tr>
<tr>
<td>big-yolo</td>
<td>238</td>
<td>439</td>
<td>288</td>
<td>332</td>
</tr>
</tbody>
</table>
<p dir="auto">I understand that there is currently an issue where new graph weights are saved twice to the .pb file (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="468258845" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/30717" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/30717/hovercard" href="https://github.com/tensorflow/tensorflow/issues/30717">#30717</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="468975806" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/30789" data-hovercard-type="pull_request" data-hovercard-url="/tensorflow/tensorflow/pull/30789/hovercard" href="https://github.com/tensorflow/tensorflow/pull/30789">#30789</a>). Once the weights in the table are adjusted for this double weight saving, the resulting sizes for fp32 and fp16 seem reasonable.</p>
<p dir="auto"><strong>1. Why is there an increase in size for fp16-&gt;int8?</strong></p>
<h2 dir="auto"><strong>Describe the expected behavior</strong></h2>
<p dir="auto">I am trying to quantize two different YOLO models (one tiny, one normal) with TensorRT. The goals of this quantization are:</p>
<ol dir="auto">
<li>speed up inference</li>
<li>decrease model size</li>
</ol>
<p dir="auto">As quantization and conversion proceeds from native-&gt;fp32-&gt;fp16-&gt;int8, I expect inference time to decrease (FPS to increase), and model size to decrease.</p>
<h2 dir="auto"><strong>Code to reproduce the issue</strong></h2>
<p dir="auto">I am using <a href="https://github.com/tensorflow/models/blob/master/research/tensorrt/tensorrt.py">this</a> script and a few helper functions from <a href="https://github.com/tensorflow/tensorrt/blob/master/tftrt/examples/object_detection/graph_utils.py">here</a>. The two exact scripts that I use are <a href="https://github.com/tensorflow/tensorflow/files/3432597/do.py.txt">do.py</a> and <a href="https://github.com/tensorflow/tensorflow/files/3432598/utilities.py.txt">utilities.py</a>. Here are the <a href="https://drive.google.com/open?id=1Bgj9h6TJLwedtrhnritRs9eYm_iczG4v" rel="nofollow">tiny model</a> and the <a href="https://drive.google.com/open?id=19BzNDCHpDnxj9wU26BsGM4ewS9bKNmpO" rel="nofollow">big model</a>.</p>
<p dir="auto">My command for running the experiments:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="python do.py \
--frozen_graph=big-yolov3_frozen.pb \ # or tiny-yolov3_frozen.pb
--native --fp32 --fp16 --int8 \
--batch_size=32 \ # or 128 for tiny
--output_dir=/workspace \
--input_node=inputs --output_node=output_boxes"><pre class="notranslate">python do.py \
--frozen_graph=big-yolov3_frozen.pb <span class="pl-cce">\ </span><span class="pl-c"><span class="pl-c">#</span> or tiny-yolov3_frozen.pb</span>
--native --fp32 --fp16 --int8 \
--batch_size=32 <span class="pl-cce">\ </span><span class="pl-c"><span class="pl-c">#</span> or 128 for tiny</span>
--output_dir=/workspace \
--input_node=inputs --output_node=output_boxes</pre></div>
<h2 dir="auto"><strong>Other info / logs</strong></h2>
<p dir="auto">I ran a couple of experiments just to make sure that the results were consistent.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="==========================
network: native_tiny-yolov3_frozen.pb,   batchsize 128, steps 100
  fps   median: 350.2,  mean: 348.0,    uncertainty: 1.4,       jitter: 5.1
  latency       median: 0.36551,        mean: 0.36846,  99th_p: 0.42946,        99th_uncertainty: 0.01453

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 340.9,  mean: 333.4,    uncertainty: 1.5,       jitter: 4.4
  latency       median: 0.37546,        mean: 0.38470,  99th_p: 0.47469,        99th_uncertainty: 0.06110

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 403.3,  mean: 402.3,    uncertainty: 0.6,       jitter: 3.6
  latency       median: 0.31740,        mean: 0.31824,  99th_p: 0.34266,        99th_uncertainty: 0.00263

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 417.7,  mean: 414.9,    uncertainty: 1.1,       jitter: 4.4
  latency       median: 0.30641,        mean: 0.30873,  99th_p: 0.35451,        99th_uncertainty: 0.01511

==========================
network: native_big-yolov3_frozen.pb,    batchsize 32, steps 100
  fps   median: 125.2,  mean: 124.7,    uncertainty: 0.3,       jitter: 1.4
  latency       median: 0.25553,        mean: 0.25677,  99th_p: 0.28257,        99th_uncertainty: 0.00308

==========================
network: tftrt_fp32_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 140.3,  mean: 140.2,    uncertainty: 0.4,       jitter: 1.9
  latency       median: 0.22802,        mean: 0.22839,  99th_p: 0.25419,        99th_uncertainty: 0.00890

==========================
network: tftrt_fp16_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 237.6,  mean: 242.5,    uncertainty: 1.4,       jitter: 5.4
  latency       median: 0.13469,        mean: 0.13245,  99th_p: 0.17733,        99th_uncertainty: 0.04387

==========================
network: tftrt_int8_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 210.1,  mean: 207.5,    uncertainty: 1.5,       jitter: 2.7
  latency       median: 0.15231,        mean: 0.15657,  99th_p: 0.16928,        99th_uncertainty: 0.16613"><pre class="notranslate"><code class="notranslate">==========================
network: native_tiny-yolov3_frozen.pb,   batchsize 128, steps 100
  fps   median: 350.2,  mean: 348.0,    uncertainty: 1.4,       jitter: 5.1
  latency       median: 0.36551,        mean: 0.36846,  99th_p: 0.42946,        99th_uncertainty: 0.01453

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 340.9,  mean: 333.4,    uncertainty: 1.5,       jitter: 4.4
  latency       median: 0.37546,        mean: 0.38470,  99th_p: 0.47469,        99th_uncertainty: 0.06110

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 403.3,  mean: 402.3,    uncertainty: 0.6,       jitter: 3.6
  latency       median: 0.31740,        mean: 0.31824,  99th_p: 0.34266,        99th_uncertainty: 0.00263

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 417.7,  mean: 414.9,    uncertainty: 1.1,       jitter: 4.4
  latency       median: 0.30641,        mean: 0.30873,  99th_p: 0.35451,        99th_uncertainty: 0.01511

==========================
network: native_big-yolov3_frozen.pb,    batchsize 32, steps 100
  fps   median: 125.2,  mean: 124.7,    uncertainty: 0.3,       jitter: 1.4
  latency       median: 0.25553,        mean: 0.25677,  99th_p: 0.28257,        99th_uncertainty: 0.00308

==========================
network: tftrt_fp32_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 140.3,  mean: 140.2,    uncertainty: 0.4,       jitter: 1.9
  latency       median: 0.22802,        mean: 0.22839,  99th_p: 0.25419,        99th_uncertainty: 0.00890

==========================
network: tftrt_fp16_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 237.6,  mean: 242.5,    uncertainty: 1.4,       jitter: 5.4
  latency       median: 0.13469,        mean: 0.13245,  99th_p: 0.17733,        99th_uncertainty: 0.04387

==========================
network: tftrt_int8_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 210.1,  mean: 207.5,    uncertainty: 1.5,       jitter: 2.7
  latency       median: 0.15231,        mean: 0.15657,  99th_p: 0.16928,        99th_uncertainty: 0.16613
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="==========================
network: native_tiny-yolov3_frozen.pb,   batchsize 128, steps 100
  fps   median: 357.4,  mean: 354.3,    uncertainty: 1.7,       jitter: 14.5
  latency       median: 0.35814,        mean: 0.36215,  99th_p: 0.44575,        99th_uncertainty: 0.00629

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 324.9,  mean: 319.4,    uncertainty: 1.4,       jitter: 2.5
  latency       median: 0.39401,        mean: 0.40173,  99th_p: 0.49218,        99th_uncertainty: 0.06484

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 376.1,  mean: 372.8,    uncertainty: 1.1,       jitter: 1.7
  latency       median: 0.34036,        mean: 0.34363,  99th_p: 0.38601,        99th_uncertainty: 0.02051

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 392.1,  mean: 391.3,    uncertainty: 0.5,       jitter: 1.8
  latency       median: 0.32645,        mean: 0.32717,  99th_p: 0.33765,        99th_uncertainty: 0.01967

==========================
network: native_big-yolov3_frozen.pb,    batchsize 32, steps 100
  fps   median: 124.3,  mean: 124.0,    uncertainty: 0.4,       jitter: 1.7
  latency       median: 0.25737,        mean: 0.25842,  99th_p: 0.31292,        99th_uncertainty: 0.00355

==========================
network: tftrt_fp32_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 141.0,  mean: 140.7,    uncertainty: 0.3,       jitter: 0.3
  latency       median: 0.22690,        mean: 0.22761,  99th_p: 0.24239,        99th_uncertainty: 0.00581

==========================
network: tftrt_fp16_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 247.4,  mean: 245.9,    uncertainty: 1.0,       jitter: 4.4
  latency       median: 0.12934,        mean: 0.13044,  99th_p: 0.16018,        99th_uncertainty: 0.02011

==========================
network: tftrt_int8_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 206.0,  mean: 204.5,    uncertainty: 1.4,       jitter: 1.0
  latency       median: 0.15536,        mean: 0.15885,  99th_p: 0.16454,        99th_uncertainty: 0.17423"><pre class="notranslate"><code class="notranslate">==========================
network: native_tiny-yolov3_frozen.pb,   batchsize 128, steps 100
  fps   median: 357.4,  mean: 354.3,    uncertainty: 1.7,       jitter: 14.5
  latency       median: 0.35814,        mean: 0.36215,  99th_p: 0.44575,        99th_uncertainty: 0.00629

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 324.9,  mean: 319.4,    uncertainty: 1.4,       jitter: 2.5
  latency       median: 0.39401,        mean: 0.40173,  99th_p: 0.49218,        99th_uncertainty: 0.06484

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 376.1,  mean: 372.8,    uncertainty: 1.1,       jitter: 1.7
  latency       median: 0.34036,        mean: 0.34363,  99th_p: 0.38601,        99th_uncertainty: 0.02051

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb,       batchsize 128, steps 100
  fps   median: 392.1,  mean: 391.3,    uncertainty: 0.5,       jitter: 1.8
  latency       median: 0.32645,        mean: 0.32717,  99th_p: 0.33765,        99th_uncertainty: 0.01967

==========================
network: native_big-yolov3_frozen.pb,    batchsize 32, steps 100
  fps   median: 124.3,  mean: 124.0,    uncertainty: 0.4,       jitter: 1.7
  latency       median: 0.25737,        mean: 0.25842,  99th_p: 0.31292,        99th_uncertainty: 0.00355

==========================
network: tftrt_fp32_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 141.0,  mean: 140.7,    uncertainty: 0.3,       jitter: 0.3
  latency       median: 0.22690,        mean: 0.22761,  99th_p: 0.24239,        99th_uncertainty: 0.00581

==========================
network: tftrt_fp16_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 247.4,  mean: 245.9,    uncertainty: 1.0,       jitter: 4.4
  latency       median: 0.12934,        mean: 0.13044,  99th_p: 0.16018,        99th_uncertainty: 0.02011

==========================
network: tftrt_int8_big-yolov3_frozen.pb,        batchsize 32, steps 100
  fps   median: 206.0,  mean: 204.5,    uncertainty: 1.4,       jitter: 1.0
  latency       median: 0.15536,        mean: 0.15885,  99th_p: 0.16454,        99th_uncertainty: 0.17423
</code></pre></div>