<p dir="auto">Didn't find engine for operation quantized::conv_prepack NoQEngine (operator () at ..\aten\src\ATen\native\quantized\cpu\qconv_prepack.cpp:264)<br>
(no backtrace available)</p>
<p dir="auto">win10<br>
python3.6<br>
pytorch1.3</p>
<p dir="auto">code:<br>
from torch.quantization import convert<br>
model_quantized_and_trained = convert(model_ft_tuned, inplace=True)</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/peterjc123/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/peterjc123">@peterjc123</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jerryzh168/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jerryzh168">@jerryzh168</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jianyuh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jianyuh">@jianyuh</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dzhulgakov/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dzhulgakov">@dzhulgakov</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/raghuramank100/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/raghuramank100">@raghuramank100</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jamesr66a/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jamesr66a">@jamesr66a</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature Request [Solved]</h2>
<p dir="auto">The <code class="notranslate">torch::jit::load</code> works well for loading both quantized model and un-quantized model on Linux and MacOS. But it will throw an error on Windows when loading the quantized model.</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="cookie" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f36a.png">🍪</g-emoji> Currently, official FBGEMM can be compiled with PyTorch for Windows.</h2>
<p dir="auto">PR:  <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="564301894" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/33250" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/33250/hovercard" href="https://github.com/pytorch/pytorch/pull/33250">#33250</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="cookie" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f36a.png">🍪</g-emoji>  <del>Current Unofficial Method</del></h2>
<p dir="auto">Currently we compile <code class="notranslate">libtorch-1.3.1</code> with <a href="https://github.com/marian-nmt/FBGEMM">FBGEMM marian-nmt version</a> by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ykim362/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ykim362">@ykim362</a> on windows successfully. And the quantized model works pretty well on windows (200 ms -&gt; 57 ms). Thanks for the effort from <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ykim362/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ykim362">@ykim362</a>. Hope <code class="notranslate">fbgemm</code> for windows can be aggregated in the <strong>future/next</strong> release of PyTorch. You know, the need of quantization for windows is as much as for Linux and ARM.</p>
<p dir="auto">Some details:</p>
<ul dir="auto">
<li>PyTorch-1.3.1 : <a href="https://github.com/pytorch/pytorch/tree/v1.3.1">released branch with tag-1.3.1</a></li>
<li>FBGEMM : <a href="https://github.com/marian-nmt/FBGEMM">marian-nmt version</a> by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ykim362/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ykim362">@ykim362</a></li>
<li><code class="notranslate">set CAFFE2=OFF</code> when compiling on windows</li>
</ul>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Quick steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Use a model under <a href="https://github.com/pytorch/vision/tree/master/torchvision/models/quantization">torchvision/models/quantization</a>, for example, mobilenet.</li>
<li>In model definition py file, add</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if __name__ == '__main__':
    model = mobilenet_v2(pretrained=False, progress=True, quantize=True)
    torch.jit.save(torch.jit.script(model), 'mov2.pt')"><pre class="notranslate"><span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:
    <span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-en">mobilenet_v2</span>(<span class="pl-s1">pretrained</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">progress</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">quantize</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
    <span class="pl-s1">torch</span>.<span class="pl-s1">jit</span>.<span class="pl-en">save</span>(<span class="pl-s1">torch</span>.<span class="pl-s1">jit</span>.<span class="pl-en">script</span>(<span class="pl-s1">model</span>), <span class="pl-s">'mov2.pt'</span>)</pre></div>
<ol start="3" dir="auto">
<li>Load the scripted model in Linux, MacOS, and Windows</li>
</ol>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt; 
#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
    torch::jit::script::Module quant_model;
    quant_model = torch::jit::load(argv[1]);
}"><pre class="notranslate">#<span class="pl-k">include</span> <span class="pl-s"><span class="pl-pds">&lt;</span>torch/torch.h<span class="pl-pds">&gt;</span></span>
#<span class="pl-k">include</span> <span class="pl-s"><span class="pl-pds">&lt;</span>torch/script.h<span class="pl-pds">&gt;</span></span> 
#<span class="pl-k">include</span> <span class="pl-s"><span class="pl-pds">&lt;</span>iostream<span class="pl-pds">&gt;</span></span>
#<span class="pl-k">include</span> <span class="pl-s"><span class="pl-pds">&lt;</span>memory<span class="pl-pds">&gt;</span></span>

<span class="pl-k">int</span> <span class="pl-en">main</span>(<span class="pl-k">int</span> argc, <span class="pl-k">const</span> <span class="pl-k">char</span>* argv[]) {
    torch::jit::script::Module quant_model;
    quant_model = <span class="pl-c1">torch::jit::load</span>(argv[<span class="pl-c1">1</span>]);
}</pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">The <code class="notranslate">torch::jit::load</code> works well for loading both quantized model and un-quantized model on Linux and MacOS. But it will throw an error on Windows when loading the quantized model:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="The field 'training' was left unitialized after __setstate__, but expected a value of type 'bool' (postSetStateValidate at ..\..\torch\csrc\jit\import.cpp:52)
(no backtrace available)"><pre class="notranslate">The field <span class="pl-s"><span class="pl-pds">'</span>training<span class="pl-pds">'</span></span> was left unitialized after __setstate__, but expected a value of <span class="pl-c1">type</span> <span class="pl-s"><span class="pl-pds">'</span>bool<span class="pl-pds">'</span></span> (postSetStateValidate at ..<span class="pl-cce">\.</span>.<span class="pl-cce">\t</span>orch<span class="pl-cce">\c</span>src<span class="pl-cce">\j</span>it<span class="pl-cce">\i</span>mport.cpp:52)
(no backtrace available)</pre></div>
<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>PyTorch Version: 1.3.1</li>
<li>LibTorch Version: 1.3.1</li>
<li>OS: Ubuntu, MacOS, and Windows10</li>
</ul>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/suo/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/suo">@suo</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jerryzh168/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jerryzh168">@jerryzh168</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jianyuh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jianyuh">@jianyuh</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dzhulgakov/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dzhulgakov">@dzhulgakov</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/raghuramank100/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/raghuramank100">@raghuramank100</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jamesr66a/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jamesr66a">@jamesr66a</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/soumith/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/soumith">@soumith</a></p>