<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">LSTM cell with bias produces "expected scalar type Half but found Float" error when called within an autocast context.</p>
<h2 dir="auto">To Reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import torch.nn as nn
from torch.cuda.amp import *
dev = 'cuda'
net = nn.LSTMCell(3, 3, bias=True).to(dev)
x = torch.randn(2, 3).to(dev)
with autocast():    
    h, c = net(x.half())"><pre class="notranslate"><code class="notranslate">import torch
import torch.nn as nn
from torch.cuda.amp import *
dev = 'cuda'
net = nn.LSTMCell(3, 3, bias=True).to(dev)
x = torch.randn(2, 3).to(dev)
with autocast():    
    h, c = net(x.half())
</code></pre></div>
<p dir="auto">Error produced:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-16-1d68da89e14f&gt; in &lt;module&gt;
      4 
      5 with autocast():
----&gt; 6     h, c = net(x.half())

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    967             input, hx,
    968             self.weight_ih, self.weight_hh,
--&gt; 969             self.bias_ih, self.bias_hh,
    970         )
    971 

RuntimeError: expected scalar type Half but found Float"><pre class="notranslate"><code class="notranslate">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-16-1d68da89e14f&gt; in &lt;module&gt;
      4 
      5 with autocast():
----&gt; 6     h, c = net(x.half())

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    967             input, hx,
    968             self.weight_ih, self.weight_hh,
--&gt; 969             self.bias_ih, self.bias_hh,
    970         )
    971 

RuntimeError: expected scalar type Half but found Float
</code></pre></div>
<p dir="auto">Note that there is no problem while setting <code class="notranslate">bias=False</code>.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">No error.</p>
<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.0): 1.6.0</li>
<li>OS (e.g., Linux): Ubuntu 18.04</li>
<li>How you installed PyTorch (<code class="notranslate">conda</code>, <code class="notranslate">pip</code>, source): conda</li>
<li>Build command you used (if compiling from source):</li>
<li>Python version: 3.7.6</li>
<li>CUDA/cuDNN version: 10.2</li>
<li>GPU models and configuration: GTX 1060</li>
</ul>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="670241707" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/42385" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/42385/hovercard" href="https://github.com/pytorch/pytorch/pull/42385">#42385</a> fixes cudnn RNNs with <code class="notranslate">torch.cuda.amp.autocast</code> but it appears the cell-based RNN API is also broken:<br>
<a href="https://discuss.pytorch.org/t/runtimeerror-expected-scalar-type-half-but-found-float/91628" rel="nofollow">https://discuss.pytorch.org/t/runtimeerror-expected-scalar-type-half-but-found-float/91628</a></p>
<p dir="auto">I thought cell-based RNNs were implemented in terms of autograd-exposed primitive ops (gemms, etc) already covered by autocast, but LSTMCell and GRUCell have <a href="https://github.com/pytorch/pytorch/blob/db52cd7322ccf3f2121f1a6e592ddb436968aa50/tools/autograd/derivatives.yaml#L1756-L1761">dedicated autograd functions</a>.  I should add FP16 cast wrappers for these.  I may need to add wrappers at a higher level instead, ie wrap the Python-exposed <code class="notranslate">at::lstm_cell</code> and <code class="notranslate">at::gru_cell</code>, depending where in the call chain input type checking occurs.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mcarilli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mcarilli">@mcarilli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ppwwyyxx/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ppwwyyxx">@ppwwyyxx</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a></p>