We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
🐛 [Bug] Failed to use stoi to get the engine ID in TRTEngine function
##  Bug Description

Got this  error: 
```
INFO: [Torch-TensorRT - Debug Build] - [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +132, now: CPU 0, GPU 540 (MiB)
terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
Aborted (core dumped)
```

Debugged and found that it fails at this line: https://github.com/NVIDIA/Torch-TensorRT/blob/c95229144432d96f4bdaa71fb1d242242d42bc29/core/runtime/TRTEngine.cpp#L63 

## To Reproduce

 MaskRCNN model from detectron2. 

## Expected behavior

should figure out why we cannot find a number from this engine. 




--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `py/torch_tensorrt/ts/_compiler.py`
Content:
```
1 from typing import List, Dict, Any
2 import torch
3 from torch import nn
4 
5 import torch_tensorrt._C.ts as _C
6 from torch_tensorrt import _enums
7 from torch_tensorrt.ts._compile_spec import _parse_compile_spec, _parse_device
8 from torch_tensorrt._Device import Device
9 from types import FunctionType
10 
11 
12 def compile(module: torch.jit.ScriptModule,
13             inputs=[],
14             device=Device._current_device(),
15             disable_tf32=False,
16             sparse_weights=False,
17             enabled_precisions=set(),
18             refit=False,
19             debug=False,
20             capability=_enums.EngineCapability.default,
21             num_min_timing_iters=2,
22             num_avg_timing_iters=1,
23             workspace_size=0,
24             calibrator=None,
25             truncate_long_and_double=False,
26             require_full_compilation=False,
27             min_block_size=3,
28             torch_executed_ops=[],
29             torch_executed_modules=[]) -> torch.jit.ScriptModule:
30     """Compile a TorchScript module for NVIDIA GPUs using TensorRT
31 
32     Takes a existing TorchScript module and a set of settings to configure the compiler
33     and will convert methods to JIT Graphs which call equivalent TensorRT engines
34 
35     Converts specifically the forward method of a TorchScript Module
36 
37     Arguments:
38         module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch
39             ``torch.nn.Module``
40 
41     Keyword Arguments:
42         inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using
43             torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum
44             to select device type. ::
45 
46                 input=[
47                     torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1
48                     torch_tensorrt.Input(
49                         min_shape=(1, 224, 224, 3),
50                         opt_shape=(1, 512, 512, 3),
51                         max_shape=(1, 1024, 1024, 3),
52                         dtype=torch.int32
53                         format=torch.channel_last
54                     ), # Dynamic input shape for input #2
55                     torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings
56                 ]
57 
58         device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on ::
59 
60             device=torch_tensorrt.Device("dla:1", allow_gpu_fallback=True)
61 
62         disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas
63         sparse_weights (bool): Enable sparsity for convolution and fully connected layers.
64         enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels
65         refit (bool): Enable refitting
66         debug (bool): Enable debuggable engine
67         capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels
68         num_min_timing_iters (int): Number of minimization timing iterations used to select kernels
69         num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels
70         workspace_size (int): Maximum size of workspace given to TensorRT
71         truncate_long_and_double (bool): Truncate weights provided in int64 or double (float64) to int32 and float32
72         calibrator (Union(torch_tensorrt._C.IInt8Calibrator, tensorrt.IInt8Calibrator)): Calibrator object which will provide data to the PTQ system for INT8 Calibration
73         require_full_compilation (bool): Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch
74         min_block_size (int): The minimum number of contiguous TensorRT convertable operations in order to run a set of operations in TensorRT
75         torch_executed_ops (List[str]): List of aten operators that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True
76         torch_executed_modules (List[str]): List of modules that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True
77 
78     Returns:
79         torch.jit.ScriptModule: Compiled TorchScript Module, when run it will execute via TensorRT
80     """
81 
82     if isinstance(module, torch.jit.ScriptFunction):
83         raise TypeError(
84             "torch.jit.ScriptFunction currently is not directly supported, wrap the function in a module to compile")
85 
86     if require_full_compilation and (len(torch_executed_modules) > 0 or len(torch_executed_ops) > 0):
87         raise ValueError(
88             "require_full_compilation is enabled however the list of modules and ops to run in torch is not empty. Found: torch_executed_ops: "
89             + torch_executed_ops + ", torch_executed_modules: " + torch_executed_modules)
90 
91     spec = {
92         "inputs": inputs,
93         "device": device,
94         "disable_tf32": disable_tf32,  # Force FP32 layers to use traditional as FP32 format
95         "sparse_weights": sparse_weights,  #Enable sparsity for convolution and fully connected layers.
96         "enabled_precisions": enabled_precisions,  # Enabling FP16 kernels
97         "refit": refit,  # enable refit
98         "debug": debug,  # enable debuggable engine
99         "capability": capability,  # Restrict kernel selection to safe gpu kernels or safe dla kernels
100         "num_min_timing_iters": num_min_timing_iters,  # Number of minimization timing iterations used to select kernels
101         "num_avg_timing_iters": num_avg_timing_iters,  # Number of averaging timing iterations used to select kernels
102         "workspace_size": workspace_size,  # Maximum size of workspace given to TensorRT
103         "calibrator": calibrator,
104         "truncate_long_and_double": truncate_long_and_double,
105         "torch_fallback": {
106             "enabled": not require_full_compilation,
107             "forced_fallback_ops": torch_executed_ops,
108             "forced_fallback_modules": torch_executed_modules,
109             "min_block_size": min_block_size
110         }
111     }
112 
113     compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
114     compiled_module = torch.jit._recursive.wrap_cpp_module(compiled_cpp_mod)
115     return compiled_module
116 
117 
118 def convert_method_to_trt_engine(module: torch.jit.ScriptModule,
119                                  method_name: str,
120                                  inputs=[],
121                                  device=Device._current_device(),
122                                  disable_tf32=False,
123                                  sparse_weights=False,
124                                  enabled_precisions=set(),
125                                  refit=False,
126                                  debug=False,
127                                  capability=_enums.EngineCapability.default,
128                                  num_min_timing_iters=2,
129                                  num_avg_timing_iters=1,
130                                  workspace_size=0,
131                                  truncate_long_and_double=False,
132                                  calibrator=None) -> str:
133     """Convert a TorchScript module method to a serialized TensorRT engine
134 
135     Converts a specified method of a module to a serialized TensorRT engine given a dictionary of conversion settings
136 
137     Arguments:
138         module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch
139             ``torch.nn.Module``
140         method_name (str): Name of method to convert
141 
142     Keyword Args:
143         inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using
144             torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum
145             to select device type. ::
146 
147                 input=[
148                     torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1
149                     torch_tensorrt.Input(
150                         min_shape=(1, 224, 224, 3),
151                         opt_shape=(1, 512, 512, 3),
152                         max_shape=(1, 1024, 1024, 3),
153                         dtype=torch.int32
154                         format=torch.channel_last
155                     ), # Dynamic input shape for input #2
156                     torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings
157                 ]
158 
159         device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on ::
160 
161             device=torch_tensorrt.Device("dla:1", allow_gpu_fallback=True)
162 
163         disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas
164         sparse_weights (bool): Enable sparsity for convolution and fully connected layers.
165         enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels
166         refit (bool): Enable refitting
167         debug (bool): Enable debuggable engine
168         capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels
169         num_min_timing_iters (int): Number of minimization timing iterations used to select kernels
170         num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels
171         workspace_size (int): Maximum size of workspace given to TensorRT
172         truncate_long_and_double (bool): Truncate weights provided in int64 or double (float64) to int32 and float32
173         calibrator (Union(torch_tensorrt._C.IInt8Calibrator, tensorrt.IInt8Calibrator)): Calibrator object which will provide data to the PTQ system for INT8 Calibration
174 
175     Returns:
176         bytes: Serialized TensorRT engine, can either be saved to a file or deserialized via TensorRT APIs
177     """
178     if isinstance(module, torch.jit.ScriptFunction):
179         raise TypeError(
180             "torch.jit.ScriptFunctions currently are not directly supported, wrap the function in a module to compile")
181 
182     compile_spec = {
183         "inputs": inputs,
184         "device": device,
185         "disable_tf32":
186             disable_tf32,  # Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas
187         "sparse_weights": sparse_weights,  #Enable sparsity for convolution and fully connected layers.
188         "enabled_precisions": enabled_precisions,  # Enabling FP16 kernels
189         "refit": refit,  # enable refit
190         "debug": debug,  # enable debuggable engine
191         "capability": capability,  # Restrict kernel selection to safe gpu kernels or safe dla kernels
192         "num_min_timing_iters": num_min_timing_iters,  # Number of minimization timing iterations used to select kernels
193         "num_avg_timing_iters": num_avg_timing_iters,  # Number of averaging timing iterations used to select kernels
194         "workspace_size": workspace_size,  # Maximum size of workspace given to TensorRT
195         "calibrator": calibrator,
196         "truncate_long_and_double": truncate_long_and_double
197     }
198 
199     return _C.convert_graph_to_trt_engine(module._c, method_name, _parse_compile_spec(compile_spec))
200 
201 
202 def embed_engine_in_new_module(serialized_engine: bytes, device=Device._current_device()) -> torch.jit.ScriptModule:
203     """Takes a pre-built serialized TensorRT engine and embeds it within a TorchScript module
204 
205     Takes a pre-built serialied TensorRT engine (as bytes) and embeds it within a TorchScript module.
206     Registers the forward method to execute the TensorRT engine with the function signature:
207 
208         forward(Tensor[]) -> Tensor[]
209 
210     Module can be save with engine embedded with torch.jit.save and moved / loaded according to torch_tensorrt portability rules
211 
212     Arguments:
213         serialized_engine (bytes): Serialized TensorRT engine from either torch_tensorrt or TensorRT APIs
214 
215     Keyword Arguments:
216         device (Union(torch_tensorrt.Device, torch.device, dict)): Target device to run engine on. Must be compatible with engine provided. Default: Current active device
217 
218     Returns:
219         torch.jit.ScriptModule: New TorchScript module with engine embedded
220     """
221     cpp_mod = _C.embed_engine_in_new_module(serialized_engine, _parse_device(device))
222     return torch.jit._recursive.wrap_cpp_module(cpp_mod)
223 
224 
225 def check_method_op_support(module: torch.jit.ScriptModule, method_name: str) -> bool:
226     """Checks to see if a method is fully supported by torch_tensorrt
227 
228     Checks if a method of a TorchScript module can be compiled by torch_tensorrt, if not, a list of operators
229     that are not supported are printed out and the function returns false, else true.
230 
231     Arguments:
232         module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch
233             ``torch.nn.Module``
234         method_name (str): Name of method to check
235 
236     Returns:
237         bool: True if supported Method
238     """
239     return _C.check_method_op_support(module._c, method_name)
240
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/py/torch_tensorrt/ts/_compiler.py b/py/torch_tensorrt/ts/_compiler.py
--- a/py/torch_tensorrt/ts/_compiler.py
+++ b/py/torch_tensorrt/ts/_compiler.py
@@ -207,6 +207,11 @@
 
         forward(Tensor[]) -> Tensor[]
 
+    TensorRT bindings must have names with the following format:
+      - [symbol].[index in input / output array]
+      ex.
+      - [x.0, x.1, x.2] -> [y.0]
+
     Module can be save with engine embedded with torch.jit.save and moved / loaded according to torch_tensorrt portability rules
 
     Arguments:

{"golden_diff": "diff --git a/py/torch_tensorrt/ts/_compiler.py b/py/torch_tensorrt/ts/_compiler.py\n--- a/py/torch_tensorrt/ts/_compiler.py\n+++ b/py/torch_tensorrt/ts/_compiler.py\n@@ -207,6 +207,11 @@\n \n         forward(Tensor[]) -> Tensor[]\n \n+    TensorRT bindings must have names with the following format:\n+      - [symbol].[index in input / output array]\n+      ex.\n+      - [x.0, x.1, x.2] -> [y.0]\n+\n     Module can be save with engine embedded with torch.jit.save and moved / loaded according to torch_tensorrt portability rules\n \n     Arguments:\n", "issue": "\ud83d\udc1b [Bug] Failed to use stoi to get the engine ID in TRTEngine function\n##  Bug Description\r\n\r\nGot this  error: \r\n```\r\nINFO: [Torch-TensorRT - Debug Build] - [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +132, now: CPU 0, GPU 540 (MiB)\r\nterminate called after throwing an instance of 'std::invalid_argument'\r\n  what():  stoi\r\nAborted (core dumped)\r\n```\r\n\r\nDebugged and found that it fails at this line: https://github.com/NVIDIA/Torch-TensorRT/blob/c95229144432d96f4bdaa71fb1d242242d42bc29/core/runtime/TRTEngine.cpp#L63 \r\n\r\n## To Reproduce\r\n\r\n MaskRCNN model from detectron2. \r\n\r\n## Expected behavior\r\n\r\nshould figure out why we cannot find a number from this engine. \r\n\r\n\r\n\n", "before_files": [{"content": "from typing import List, Dict, Any\nimport torch\nfrom torch import nn\n\nimport torch_tensorrt._C.ts as _C\nfrom torch_tensorrt import _enums\nfrom torch_tensorrt.ts._compile_spec import _parse_compile_spec, _parse_device\nfrom torch_tensorrt._Device import Device\nfrom types import FunctionType\n\n\ndef compile(module: torch.jit.ScriptModule,\n            inputs=[],\n            device=Device._current_device(),\n            disable_tf32=False,\n            sparse_weights=False,\n            enabled_precisions=set(),\n            refit=False,\n            debug=False,\n            capability=_enums.EngineCapability.default,\n            num_min_timing_iters=2,\n            num_avg_timing_iters=1,\n            workspace_size=0,\n            calibrator=None,\n            truncate_long_and_double=False,\n            require_full_compilation=False,\n            min_block_size=3,\n            torch_executed_ops=[],\n            torch_executed_modules=[]) -> torch.jit.ScriptModule:\n    \"\"\"Compile a TorchScript module for NVIDIA GPUs using TensorRT\n\n    Takes a existing TorchScript module and a set of settings to configure the compiler\n    and will convert methods to JIT Graphs which call equivalent TensorRT engines\n\n    Converts specifically the forward method of a TorchScript Module\n\n    Arguments:\n        module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch\n            ``torch.nn.Module``\n\n    Keyword Arguments:\n        inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using\n            torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum\n            to select device type. ::\n\n                input=[\n                    torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1\n                    torch_tensorrt.Input(\n                        min_shape=(1, 224, 224, 3),\n                        opt_shape=(1, 512, 512, 3),\n                        max_shape=(1, 1024, 1024, 3),\n                        dtype=torch.int32\n                        format=torch.channel_last\n                    ), # Dynamic input shape for input #2\n                    torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings\n                ]\n\n        device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on ::\n\n            device=torch_tensorrt.Device(\"dla:1\", allow_gpu_fallback=True)\n\n        disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas\n        sparse_weights (bool): Enable sparsity for convolution and fully connected layers.\n        enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels\n        refit (bool): Enable refitting\n        debug (bool): Enable debuggable engine\n        capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels\n        num_min_timing_iters (int): Number of minimization timing iterations used to select kernels\n        num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels\n        workspace_size (int): Maximum size of workspace given to TensorRT\n        truncate_long_and_double (bool): Truncate weights provided in int64 or double (float64) to int32 and float32\n        calibrator (Union(torch_tensorrt._C.IInt8Calibrator, tensorrt.IInt8Calibrator)): Calibrator object which will provide data to the PTQ system for INT8 Calibration\n        require_full_compilation (bool): Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch\n        min_block_size (int): The minimum number of contiguous TensorRT convertable operations in order to run a set of operations in TensorRT\n        torch_executed_ops (List[str]): List of aten operators that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True\n        torch_executed_modules (List[str]): List of modules that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True\n\n    Returns:\n        torch.jit.ScriptModule: Compiled TorchScript Module, when run it will execute via TensorRT\n    \"\"\"\n\n    if isinstance(module, torch.jit.ScriptFunction):\n        raise TypeError(\n            \"torch.jit.ScriptFunction currently is not directly supported, wrap the function in a module to compile\")\n\n    if require_full_compilation and (len(torch_executed_modules) > 0 or len(torch_executed_ops) > 0):\n        raise ValueError(\n            \"require_full_compilation is enabled however the list of modules and ops to run in torch is not empty. Found: torch_executed_ops: \"\n            + torch_executed_ops + \", torch_executed_modules: \" + torch_executed_modules)\n\n    spec = {\n        \"inputs\": inputs,\n        \"device\": device,\n        \"disable_tf32\": disable_tf32,  # Force FP32 layers to use traditional as FP32 format\n        \"sparse_weights\": sparse_weights,  #Enable sparsity for convolution and fully connected layers.\n        \"enabled_precisions\": enabled_precisions,  # Enabling FP16 kernels\n        \"refit\": refit,  # enable refit\n        \"debug\": debug,  # enable debuggable engine\n        \"capability\": capability,  # Restrict kernel selection to safe gpu kernels or safe dla kernels\n        \"num_min_timing_iters\": num_min_timing_iters,  # Number of minimization timing iterations used to select kernels\n        \"num_avg_timing_iters\": num_avg_timing_iters,  # Number of averaging timing iterations used to select kernels\n        \"workspace_size\": workspace_size,  # Maximum size of workspace given to TensorRT\n        \"calibrator\": calibrator,\n        \"truncate_long_and_double\": truncate_long_and_double,\n        \"torch_fallback\": {\n            \"enabled\": not require_full_compilation,\n            \"forced_fallback_ops\": torch_executed_ops,\n            \"forced_fallback_modules\": torch_executed_modules,\n            \"min_block_size\": min_block_size\n        }\n    }\n\n    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))\n    compiled_module = torch.jit._recursive.wrap_cpp_module(compiled_cpp_mod)\n    return compiled_module\n\n\ndef convert_method_to_trt_engine(module: torch.jit.ScriptModule,\n                                 method_name: str,\n                                 inputs=[],\n                                 device=Device._current_device(),\n                                 disable_tf32=False,\n                                 sparse_weights=False,\n                                 enabled_precisions=set(),\n                                 refit=False,\n                                 debug=False,\n                                 capability=_enums.EngineCapability.default,\n                                 num_min_timing_iters=2,\n                                 num_avg_timing_iters=1,\n                                 workspace_size=0,\n                                 truncate_long_and_double=False,\n                                 calibrator=None) -> str:\n    \"\"\"Convert a TorchScript module method to a serialized TensorRT engine\n\n    Converts a specified method of a module to a serialized TensorRT engine given a dictionary of conversion settings\n\n    Arguments:\n        module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch\n            ``torch.nn.Module``\n        method_name (str): Name of method to convert\n\n    Keyword Args:\n        inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using\n            torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum\n            to select device type. ::\n\n                input=[\n                    torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1\n                    torch_tensorrt.Input(\n                        min_shape=(1, 224, 224, 3),\n                        opt_shape=(1, 512, 512, 3),\n                        max_shape=(1, 1024, 1024, 3),\n                        dtype=torch.int32\n                        format=torch.channel_last\n                    ), # Dynamic input shape for input #2\n                    torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings\n                ]\n\n        device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on ::\n\n            device=torch_tensorrt.Device(\"dla:1\", allow_gpu_fallback=True)\n\n        disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas\n        sparse_weights (bool): Enable sparsity for convolution and fully connected layers.\n        enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels\n        refit (bool): Enable refitting\n        debug (bool): Enable debuggable engine\n        capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels\n        num_min_timing_iters (int): Number of minimization timing iterations used to select kernels\n        num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels\n        workspace_size (int): Maximum size of workspace given to TensorRT\n        truncate_long_and_double (bool): Truncate weights provided in int64 or double (float64) to int32 and float32\n        calibrator (Union(torch_tensorrt._C.IInt8Calibrator, tensorrt.IInt8Calibrator)): Calibrator object which will provide data to the PTQ system for INT8 Calibration\n\n    Returns:\n        bytes: Serialized TensorRT engine, can either be saved to a file or deserialized via TensorRT APIs\n    \"\"\"\n    if isinstance(module, torch.jit.ScriptFunction):\n        raise TypeError(\n            \"torch.jit.ScriptFunctions currently are not directly supported, wrap the function in a module to compile\")\n\n    compile_spec = {\n        \"inputs\": inputs,\n        \"device\": device,\n        \"disable_tf32\":\n            disable_tf32,  # Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas\n        \"sparse_weights\": sparse_weights,  #Enable sparsity for convolution and fully connected layers.\n        \"enabled_precisions\": enabled_precisions,  # Enabling FP16 kernels\n        \"refit\": refit,  # enable refit\n        \"debug\": debug,  # enable debuggable engine\n        \"capability\": capability,  # Restrict kernel selection to safe gpu kernels or safe dla kernels\n        \"num_min_timing_iters\": num_min_timing_iters,  # Number of minimization timing iterations used to select kernels\n        \"num_avg_timing_iters\": num_avg_timing_iters,  # Number of averaging timing iterations used to select kernels\n        \"workspace_size\": workspace_size,  # Maximum size of workspace given to TensorRT\n        \"calibrator\": calibrator,\n        \"truncate_long_and_double\": truncate_long_and_double\n    }\n\n    return _C.convert_graph_to_trt_engine(module._c, method_name, _parse_compile_spec(compile_spec))\n\n\ndef embed_engine_in_new_module(serialized_engine: bytes, device=Device._current_device()) -> torch.jit.ScriptModule:\n    \"\"\"Takes a pre-built serialized TensorRT engine and embeds it within a TorchScript module\n\n    Takes a pre-built serialied TensorRT engine (as bytes) and embeds it within a TorchScript module.\n    Registers the forward method to execute the TensorRT engine with the function signature:\n\n        forward(Tensor[]) -> Tensor[]\n\n    Module can be save with engine embedded with torch.jit.save and moved / loaded according to torch_tensorrt portability rules\n\n    Arguments:\n        serialized_engine (bytes): Serialized TensorRT engine from either torch_tensorrt or TensorRT APIs\n\n    Keyword Arguments:\n        device (Union(torch_tensorrt.Device, torch.device, dict)): Target device to run engine on. Must be compatible with engine provided. Default: Current active device\n\n    Returns:\n        torch.jit.ScriptModule: New TorchScript module with engine embedded\n    \"\"\"\n    cpp_mod = _C.embed_engine_in_new_module(serialized_engine, _parse_device(device))\n    return torch.jit._recursive.wrap_cpp_module(cpp_mod)\n\n\ndef check_method_op_support(module: torch.jit.ScriptModule, method_name: str) -> bool:\n    \"\"\"Checks to see if a method is fully supported by torch_tensorrt\n\n    Checks if a method of a TorchScript module can be compiled by torch_tensorrt, if not, a list of operators\n    that are not supported are printed out and the function returns false, else true.\n\n    Arguments:\n        module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch\n            ``torch.nn.Module``\n        method_name (str): Name of method to check\n\n    Returns:\n        bool: True if supported Method\n    \"\"\"\n    return _C.check_method_op_support(module._c, method_name)\n", "path": "py/torch_tensorrt/ts/_compiler.py"}], "after_files": [{"content": "from typing import List, Dict, Any\nimport torch\nfrom torch import nn\n\nimport torch_tensorrt._C.ts as _C\nfrom torch_tensorrt import _enums\nfrom torch_tensorrt.ts._compile_spec import _parse_compile_spec, _parse_device\nfrom torch_tensorrt._Device import Device\nfrom types import FunctionType\n\n\ndef compile(module: torch.jit.ScriptModule,\n            inputs=[],\n            device=Device._current_device(),\n            disable_tf32=False,\n            sparse_weights=False,\n            enabled_precisions=set(),\n            refit=False,\n            debug=False,\n            capability=_enums.EngineCapability.default,\n            num_min_timing_iters=2,\n            num_avg_timing_iters=1,\n            workspace_size=0,\n            calibrator=None,\n            truncate_long_and_double=False,\n            require_full_compilation=False,\n            min_block_size=3,\n            torch_executed_ops=[],\n            torch_executed_modules=[]) -> torch.jit.ScriptModule:\n    \"\"\"Compile a TorchScript module for NVIDIA GPUs using TensorRT\n\n    Takes a existing TorchScript module and a set of settings to configure the compiler\n    and will convert methods to JIT Graphs which call equivalent TensorRT engines\n\n    Converts specifically the forward method of a TorchScript Module\n\n    Arguments:\n        module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch\n            ``torch.nn.Module``\n\n    Keyword Arguments:\n        inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using\n            torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum\n            to select device type. ::\n\n                input=[\n                    torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1\n                    torch_tensorrt.Input(\n                        min_shape=(1, 224, 224, 3),\n                        opt_shape=(1, 512, 512, 3),\n                        max_shape=(1, 1024, 1024, 3),\n                        dtype=torch.int32\n                        format=torch.channel_last\n                    ), # Dynamic input shape for input #2\n                    torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings\n                ]\n\n        device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on ::\n\n            device=torch_tensorrt.Device(\"dla:1\", allow_gpu_fallback=True)\n\n        disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas\n        sparse_weights (bool): Enable sparsity for convolution and fully connected layers.\n        enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels\n        refit (bool): Enable refitting\n        debug (bool): Enable debuggable engine\n        capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels\n        num_min_timing_iters (int): Number of minimization timing iterations used to select kernels\n        num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels\n        workspace_size (int): Maximum size of workspace given to TensorRT\n        truncate_long_and_double (bool): Truncate weights provided in int64 or double (float64) to int32 and float32\n        calibrator (Union(torch_tensorrt._C.IInt8Calibrator, tensorrt.IInt8Calibrator)): Calibrator object which will provide data to the PTQ system for INT8 Calibration\n        require_full_compilation (bool): Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch\n        min_block_size (int): The minimum number of contiguous TensorRT convertable operations in order to run a set of operations in TensorRT\n        torch_executed_ops (List[str]): List of aten operators that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True\n        torch_executed_modules (List[str]): List of modules that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True\n\n    Returns:\n        torch.jit.ScriptModule: Compiled TorchScript Module, when run it will execute via TensorRT\n    \"\"\"\n\n    if isinstance(module, torch.jit.ScriptFunction):\n        raise TypeError(\n            \"torch.jit.ScriptFunction currently is not directly supported, wrap the function in a module to compile\")\n\n    if require_full_compilation and (len(torch_executed_modules) > 0 or len(torch_executed_ops) > 0):\n        raise ValueError(\n            \"require_full_compilation is enabled however the list of modules and ops to run in torch is not empty. Found: torch_executed_ops: \"\n            + torch_executed_ops + \", torch_executed_modules: \" + torch_executed_modules)\n\n    spec = {\n        \"inputs\": inputs,\n        \"device\": device,\n        \"disable_tf32\": disable_tf32,  # Force FP32 layers to use traditional as FP32 format\n        \"sparse_weights\": sparse_weights,  #Enable sparsity for convolution and fully connected layers.\n        \"enabled_precisions\": enabled_precisions,  # Enabling FP16 kernels\n        \"refit\": refit,  # enable refit\n        \"debug\": debug,  # enable debuggable engine\n        \"capability\": capability,  # Restrict kernel selection to safe gpu kernels or safe dla kernels\n        \"num_min_timing_iters\": num_min_timing_iters,  # Number of minimization timing iterations used to select kernels\n        \"num_avg_timing_iters\": num_avg_timing_iters,  # Number of averaging timing iterations used to select kernels\n        \"workspace_size\": workspace_size,  # Maximum size of workspace given to TensorRT\n        \"calibrator\": calibrator,\n        \"truncate_long_and_double\": truncate_long_and_double,\n        \"torch_fallback\": {\n            \"enabled\": not require_full_compilation,\n            \"forced_fallback_ops\": torch_executed_ops,\n            \"forced_fallback_modules\": torch_executed_modules,\n            \"min_block_size\": min_block_size\n        }\n    }\n\n    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))\n    compiled_module = torch.jit._recursive.wrap_cpp_module(compiled_cpp_mod)\n    return compiled_module\n\n\ndef convert_method_to_trt_engine(module: torch.jit.ScriptModule,\n                                 method_name: str,\n                                 inputs=[],\n                                 device=Device._current_device(),\n                                 disable_tf32=False,\n                                 sparse_weights=False,\n                                 enabled_precisions=set(),\n                                 refit=False,\n                                 debug=False,\n                                 capability=_enums.EngineCapability.default,\n                                 num_min_timing_iters=2,\n                                 num_avg_timing_iters=1,\n                                 workspace_size=0,\n                                 truncate_long_and_double=False,\n                                 calibrator=None) -> str:\n    \"\"\"Convert a TorchScript module method to a serialized TensorRT engine\n\n    Converts a specified method of a module to a serialized TensorRT engine given a dictionary of conversion settings\n\n    Arguments:\n        module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch\n            ``torch.nn.Module``\n        method_name (str): Name of method to convert\n\n    Keyword Args:\n        inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using\n            torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum\n            to select device type. ::\n\n                input=[\n                    torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1\n                    torch_tensorrt.Input(\n                        min_shape=(1, 224, 224, 3),\n                        opt_shape=(1, 512, 512, 3),\n                        max_shape=(1, 1024, 1024, 3),\n                        dtype=torch.int32\n                        format=torch.channel_last\n                    ), # Dynamic input shape for input #2\n                    torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings\n                ]\n\n        device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on ::\n\n            device=torch_tensorrt.Device(\"dla:1\", allow_gpu_fallback=True)\n\n        disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas\n        sparse_weights (bool): Enable sparsity for convolution and fully connected layers.\n        enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels\n        refit (bool): Enable refitting\n        debug (bool): Enable debuggable engine\n        capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels\n        num_min_timing_iters (int): Number of minimization timing iterations used to select kernels\n        num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels\n        workspace_size (int): Maximum size of workspace given to TensorRT\n        truncate_long_and_double (bool): Truncate weights provided in int64 or double (float64) to int32 and float32\n        calibrator (Union(torch_tensorrt._C.IInt8Calibrator, tensorrt.IInt8Calibrator)): Calibrator object which will provide data to the PTQ system for INT8 Calibration\n\n    Returns:\n        bytes: Serialized TensorRT engine, can either be saved to a file or deserialized via TensorRT APIs\n    \"\"\"\n    if isinstance(module, torch.jit.ScriptFunction):\n        raise TypeError(\n            \"torch.jit.ScriptFunctions currently are not directly supported, wrap the function in a module to compile\")\n\n    compile_spec = {\n        \"inputs\": inputs,\n        \"device\": device,\n        \"disable_tf32\":\n            disable_tf32,  # Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas\n        \"sparse_weights\": sparse_weights,  #Enable sparsity for convolution and fully connected layers.\n        \"enabled_precisions\": enabled_precisions,  # Enabling FP16 kernels\n        \"refit\": refit,  # enable refit\n        \"debug\": debug,  # enable debuggable engine\n        \"capability\": capability,  # Restrict kernel selection to safe gpu kernels or safe dla kernels\n        \"num_min_timing_iters\": num_min_timing_iters,  # Number of minimization timing iterations used to select kernels\n        \"num_avg_timing_iters\": num_avg_timing_iters,  # Number of averaging timing iterations used to select kernels\n        \"workspace_size\": workspace_size,  # Maximum size of workspace given to TensorRT\n        \"calibrator\": calibrator,\n        \"truncate_long_and_double\": truncate_long_and_double\n    }\n\n    return _C.convert_graph_to_trt_engine(module._c, method_name, _parse_compile_spec(compile_spec))\n\n\ndef embed_engine_in_new_module(serialized_engine: bytes, device=Device._current_device()) -> torch.jit.ScriptModule:\n    \"\"\"Takes a pre-built serialized TensorRT engine and embeds it within a TorchScript module\n\n    Takes a pre-built serialied TensorRT engine (as bytes) and embeds it within a TorchScript module.\n    Registers the forward method to execute the TensorRT engine with the function signature:\n\n        forward(Tensor[]) -> Tensor[]\n\n    TensorRT bindings must have names with the following format:\n      - [symbol].[index in input / output array]\n      ex.\n      - [x.0, x.1, x.2] -> [y.0]\n\n    Module can be save with engine embedded with torch.jit.save and moved / loaded according to torch_tensorrt portability rules\n\n    Arguments:\n        serialized_engine (bytes): Serialized TensorRT engine from either torch_tensorrt or TensorRT APIs\n\n    Keyword Arguments:\n        device (Union(torch_tensorrt.Device, torch.device, dict)): Target device to run engine on. Must be compatible with engine provided. Default: Current active device\n\n    Returns:\n        torch.jit.ScriptModule: New TorchScript module with engine embedded\n    \"\"\"\n    cpp_mod = _C.embed_engine_in_new_module(serialized_engine, _parse_device(device))\n    return torch.jit._recursive.wrap_cpp_module(cpp_mod)\n\n\ndef check_method_op_support(module: torch.jit.ScriptModule, method_name: str) -> bool:\n    \"\"\"Checks to see if a method is fully supported by torch_tensorrt\n\n    Checks if a method of a TorchScript module can be compiled by torch_tensorrt, if not, a list of operators\n    that are not supported are printed out and the function returns false, else true.\n\n    Arguments:\n        module (torch.jit.ScriptModule): Source module, a result of tracing or scripting a PyTorch\n            ``torch.nn.Module``\n        method_name (str): Name of method to check\n\n    Returns:\n        bool: True if supported Method\n    \"\"\"\n    return _C.check_method_op_support(module._c, method_name)\n", "path": "py/torch_tensorrt/ts/_compiler.py"}]}