title: What's the use for converter.build() in TensorRT?
body: <p>The <a href="https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html" rel="nofollow noreferrer">official documentation on TensorRT</a> lists two ways to convert a TensorFlow SavedModel into a TensorRT SavedModel: the first is </p>
  
  <pre class="lang-py prettyprint-override"><code>from tensorflow.python.compiler.tensorrt import trt_convert as trt
  converter = trt.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir)
  converter.convert()
  converter.save(output_saved_model_dir)
  </code></pre>
  
  <p>and the second is </p>
  
  <pre class="lang-py prettyprint-override"><code>import tensorflow as tf
  from tensorflow.python.compiler.tensorrt import trt_convert as trt
  
  conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
  conversion_params = conversion_params._replace(
      max_workspace_size_bytes=(1&lt;&lt;32))
  conversion_params = conversion_params._replace(precision_mode="FP16")
  conversion_params = conversion_params._replace(
      maximum_cached_engiens=100)
  
  converter = trt.TrtGraphConverterV2(
      input_saved_model_dir=input_saved_model_dir,
      conversion_params=conversion_params)
  converter.convert()
  def my_input_fn():
    for _ in range(num_runs):
      Inp1 = np.random.normal(size=(8, 16, 16, 3)).astype(np.float32)
      inp2 = np.random.normal(size=(8, 16, 16, 3)).astype(np.float32)
      yield inp1, inp2
  converter.build(input_fn=my_input_fn)
  converter.save(output_saved_model_dir)
  
  saved_model_loaded = tf.saved_model.load(
      output_saved_model_dir, tags=[tag_constants.SERVING])
  graph_func = saved_model_loaded.signatures[
      signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
  frozen_func = convert_to_constants.convert_variables_to_constants_v2(
      graph_func)
  output = frozen_func(input_data)[0].numpy()
  </code></pre>
  
  <p>Stripping out all of the boilerplate code for imports, inference etc the difference seems to lie in the call to <code>converter.build()</code>. The documentation explains this function as such:</p>
  
  <p>"This method optimizes the converted function (returned by convert()) by building TensorRT engines. This is useful in case the user wants to perform the optimizations before runtime. The optimization is done by running inference on the converted function using the input data received from the argument input_fn. This argument is a generator function that yields input data as a list or tuple. "</p>
  
  <p>What does "before runtime" mean in this context? Will the "optimizations" be performed upon model loading, upon the first inference, or upon every single inference using the converted model? What <em>are</em> those optimizations, even? Isn't converting the model to TensorRT an optimization in itself?</p>
  
  <p>I am asking because if I call <code>converter.build()</code> the conversion seems to fail in unpredictable ways after taking a LOT of time (more than two hours) to run without producing any sensible output, so I was wondering how much am I losing by not calling it and whether there is more comprehensive documentation on using TF2.x SavedModels with TensorRT. </p>
  
  <p>Thanks in advance to whoever can answer!!</p>