<h2 dir="auto">❓ Questions and Help</h2>
<p dir="auto">I tried to convert a PyTorch model that has a grid_sample operation to ONNX but it was not supported.<br>
I'm using PyTorch version 1.6.0. Please help add support!</p>
<p dir="auto">Issue was also described but not resolved in:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="320140182" data-permission-text="Title is private" data-url="https://github.com/onnx/onnx/issues/883" data-hovercard-type="issue" data-hovercard-url="/onnx/onnx/issues/883/hovercard" href="https://github.com/onnx/onnx/issues/883">onnx/onnx#883</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="501445631" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/27212" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/27212/hovercard" href="https://github.com/pytorch/pytorch/issues/27212">#27212</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="501452260" data-permission-text="Title is private" data-url="https://github.com/onnx/onnx/issues/2383" data-hovercard-type="issue" data-hovercard-url="/onnx/onnx/issues/2383/hovercard" href="https://github.com/onnx/onnx/issues/2383">onnx/onnx#2383</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="558563403" data-permission-text="Title is private" data-url="https://github.com/microsoft/onnxruntime/issues/2962" data-hovercard-type="issue" data-hovercard-url="/microsoft/onnxruntime/issues/2962/hovercard" href="https://github.com/microsoft/onnxruntime/issues/2962">microsoft/onnxruntime#2962</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="596024212" data-permission-text="Title is private" data-url="https://github.com/AliaksandrSiarohin/first-order-model/issues/92" data-hovercard-type="issue" data-hovercard-url="/AliaksandrSiarohin/first-order-model/issues/92/hovercard" href="https://github.com/AliaksandrSiarohin/first-order-model/issues/92">AliaksandrSiarohin/first-order-model#92</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="496632562" data-permission-text="Title is private" data-url="https://github.com/NVIDIA/TensorRT/issues/130" data-hovercard-type="issue" data-hovercard-url="/NVIDIA/TensorRT/issues/130/hovercard" href="https://github.com/NVIDIA/TensorRT/issues/130">NVIDIA/TensorRT#130</a></p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/houseroad/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/houseroad">@houseroad</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/spandantiwari/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/spandantiwari">@spandantiwari</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lara-hdr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lara-hdr">@lara-hdr</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/BowenBao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/BowenBao">@BowenBao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/neginraoof/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/neginraoof">@neginraoof</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>
<p dir="auto">Add support for conversion of <code class="notranslate">grid_sample</code> layer into ONNX.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">It could be really hard to perform geometric deformations on image such as object stretch or horizontal flip with convolutional layers, especially using relatively small models.</p>
<p dir="auto"><a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" rel="nofollow">Spatial Transformer Networks tutorial by PyTorch</a> as one more instance of motivation for proposed feature.</p>
<h2 dir="auto">Pitch</h2>
<p dir="auto">PyTorch supports <code class="notranslate">grid_sample</code> layer. It would be great to have an ability to convert models with this layer in ONNX for further usage.</p>
<h2 dir="auto">Alternatives</h2>
<h2 dir="auto">Additional context</h2>
<p dir="auto">Minimal code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch

import torch.nn.functional as F


class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel,self).__init__()

    def forward(self, x, m):
        return F.grid_sample(x, m)


x = torch.rand(1, 1, 10, 10)
m = torch.Tensor([[[1, 0, 0], [0, 1, 0]]])
m = F.affine_grid(m, x.size()).type_as(x)

model = MyModel()

torch.onnx.export(
    model,
    (x, m),
    &quot;grid_sample.onnx&quot;,
    verbose=True,
    input_names=['input', 'grid'],
    output_names=['output']
)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>

<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">functional</span> <span class="pl-k">as</span> <span class="pl-v">F</span>


<span class="pl-k">class</span> <span class="pl-v">MyModel</span>(<span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>):
        <span class="pl-en">super</span>(<span class="pl-v">MyModel</span>,<span class="pl-s1">self</span>).<span class="pl-en">__init__</span>()

    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>, <span class="pl-s1">m</span>):
        <span class="pl-k">return</span> <span class="pl-v">F</span>.<span class="pl-en">grid_sample</span>(<span class="pl-s1">x</span>, <span class="pl-s1">m</span>)


<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">10</span>, <span class="pl-c1">10</span>)
<span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>([[[<span class="pl-c1">1</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>], [<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">0</span>]]])
<span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-v">F</span>.<span class="pl-en">affine_grid</span>(<span class="pl-s1">m</span>, <span class="pl-s1">x</span>.<span class="pl-en">size</span>()).<span class="pl-en">type_as</span>(<span class="pl-s1">x</span>)

<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">MyModel</span>()

<span class="pl-s1">torch</span>.<span class="pl-s1">onnx</span>.<span class="pl-en">export</span>(
    <span class="pl-s1">model</span>,
    (<span class="pl-s1">x</span>, <span class="pl-s1">m</span>),
    <span class="pl-s">"grid_sample.onnx"</span>,
    <span class="pl-s1">verbose</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
    <span class="pl-s1">input_names</span><span class="pl-c1">=</span>[<span class="pl-s">'input'</span>, <span class="pl-s">'grid'</span>],
    <span class="pl-s1">output_names</span><span class="pl-c1">=</span>[<span class="pl-s">'output'</span>]
)</pre></div>
<p dir="auto"><a href="https://github.com/onnx/onnx/issues/2383" data-hovercard-type="issue" data-hovercard-url="/onnx/onnx/issues/2383/hovercard">Reference</a> to my issue in ONNX repository to add spatial transformer operator.</p>