<p dir="auto"><a href="https://github.com/huggingface/transformers/blob/53ce3854a16ad2a715bc6ac8af3e30c18b5a1d11/src/transformers/activations.py#L21">gelu = getattr(F, "gelu", _gelu_python)</a></p>
<p dir="auto">should be changed to:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if torch.__version__ &lt; '1.4.0':
    gelu = _gelu_python
else:
    gelu = getattr(F, &quot;gelu&quot;, _gelu_python)"><pre class="notranslate"><span class="pl-k">if</span> <span class="pl-s1">torch</span>.<span class="pl-s1">__version__</span> <span class="pl-c1">&lt;</span> <span class="pl-s">'1.4.0'</span>:
    <span class="pl-s1">gelu</span> <span class="pl-c1">=</span> <span class="pl-s1">_gelu_python</span>
<span class="pl-k">else</span>:
    <span class="pl-s1">gelu</span> <span class="pl-c1">=</span> <span class="pl-en">getattr</span>(<span class="pl-v">F</span>, <span class="pl-s">"gelu"</span>, <span class="pl-s1">_gelu_python</span>)</pre></div>
<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<p dir="auto">BERT breaks during FP16 training due to the gelu function.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="File &quot;/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py&quot;, line 407, in forward

hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask

File &quot;/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py&quot;, line 547, in __call__

result = self.forward(*input, **kwargs)

File &quot;/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py&quot;, line 379, in forward

intermediate_output = self.intermediate(attention_output)

File &quot;/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py&quot;, line 547, in __call__

result = self.forward(*input, **kwargs)

File &quot;/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py&quot;, line 332, in forward

hidden_states = self.intermediate_act_fn(hidden_states)

File &quot;/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py&quot;, line 1125, in gelu

return torch._C._nn.gelu(input)

RuntimeError: &quot;GeluCUDAKernelImpl&quot; not implemented for 'Half'"><pre class="notranslate"><code class="notranslate">File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 407, in forward

hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__

result = self.forward(*input, **kwargs)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 379, in forward

intermediate_output = self.intermediate(attention_output)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__

result = self.forward(*input, **kwargs)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 332, in forward

hidden_states = self.intermediate_act_fn(hidden_states)

File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 1125, in gelu

return torch._C._nn.gelu(input)

RuntimeError: "GeluCUDAKernelImpl" not implemented for 'Half'
</code></pre></div>
<h2 dir="auto">Information</h2>
<p dir="auto">The reason this is happening is because in <code class="notranslate">modeling_bert.py</code>, in 2.4.1 we had:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def gelu(x):
    &quot;&quot;&quot; Original Implementation of the gelu activation function in Google Bert repo when initially created.
        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
        Also see https://arxiv.org/abs/1606.08415
    &quot;&quot;&quot;
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))

...

ACT2FN = {&quot;gelu&quot;: gelu, &quot;relu&quot;: torch.nn.functional.relu, &quot;swish&quot;: swish, &quot;gelu_new&quot;: gelu_new, &quot;mish&quot;: mish}"><pre class="notranslate"><code class="notranslate">def gelu(x):
    """ Original Implementation of the gelu activation function in Google Bert repo when initially created.
        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
        Also see https://arxiv.org/abs/1606.08415
    """
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))

...

ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish, "gelu_new": gelu_new, "mish": mish}
</code></pre></div>
<p dir="auto">whereas in 2.5.0 we have:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ACT2FN = {&quot;gelu&quot;: gelu, &quot;relu&quot;: torch.nn.functional.relu, &quot;swish&quot;: swish, &quot;gelu_new&quot;: gelu_new, &quot;mish&quot;: mish}"><pre class="notranslate"><code class="notranslate">ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish, "gelu_new": gelu_new, "mish": mish}
</code></pre></div>
<p dir="auto">where <code class="notranslate">gelu</code> now comes from <code class="notranslate">activations.py</code> as:<br>
<code class="notranslate">gelu = getattr(F, "gelu", _gelu_python)</code> on line 21.</p>
<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version:</li>
<li>Platform: Linux</li>
<li>Python version: 3.6</li>
<li>PyTorch version (GPU?): 1.2.0 CUDA 10.0</li>
<li>Tensorflow version (GPU?): 2.5.0</li>
<li>Using GPU in script?: V100</li>
<li>Using distributed or parallel set-up in script?: No, but using Apex FP16 training</li>
</ul>