We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
RuntimeError: "GeluCUDAKernelImpl" not implemented for 'Half'
PyTorch 1.2 introduced the `gelu` activation function. Unfortunately, this leads to terminal errors when using with AMP.

Trace (`self.activation` is `gelu`): 

```
Traceback (most recent call last):
  File "predict.py", line 282, in <module>
    predictor.predict()
  File "predict.py", line 74, in predict
    fig = trainer.train()
  File "/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/TransformerTrainer.py", line 232, in train
    self._process('train', epoch)
  File "/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/TransformerTrainer.py", line 124, in _process
    preds = self.model(input_ids, attention_mask=input_mask)
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 442, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/models.py", line 140, in forward
    cls_output = self.activation(cls_output)
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/functional.py", line 1126, in gelu
    return torch._C._nn.gelu(input)
RuntimeError: "GeluCUDAKernelImpl" not implemented for 'Half'
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `apex/amp/lists/functional_overrides.py`
Content:
```
1 
2 # TODO: think about the following two. They do weird things.
3 # - torch.nn.utils.clip_grad (but it should always be fp32 anyway)
4 # - torch.nn.utils.weight_norm
5 
6 # Notes:
7 # F.instance_norm uses batch_norm internally. Which correctly handles
8 #   fp16 in/out with fp32 weights. So we shouldn't do anything for
9 #   either of these.
10 # F.normalize calls `input.norm()` internally, so it's redundant, but
11 #   kept here in case impl. changes.
12 # F.cosine_similarity is same: calls `x.norm()` internally.
13 
14 import torch.nn.functional
15 
16 MODULE = torch.nn.functional
17 
18 FP16_FUNCS = [
19     'conv1d',
20     'conv2d',
21     'conv3d',
22     'conv_transpose1d',
23     'conv_transpose2d',
24     'conv_transpose3d',
25     'conv_tbc', # Undocumented / maybe new?
26     'linear',
27 ]
28 
29 FP32_FUNCS = [
30 
31     # Interpolation/Upsampling TODO:  Remove for 1.2
32     'interpolate',
33     'grid_sample',
34 
35     # Pointwise
36     'softplus',
37     'softmin',
38     'log_softmax',
39     'softmax',
40 
41     # Normalization
42     'layer_norm',
43     'group_norm',
44     'local_response_norm',
45     'normalize',
46     'cosine_similarity',
47 
48     # Loss functions
49     # TODO: which of these can be fp16?
50     'poisson_nll_loss',
51     'cosine_embedding_loss',
52     'cross_entropy',
53     'hinge_embedding_loss',
54     'kl_div',
55     'l1_loss',
56     'mse_loss',
57     'margin_ranking_loss',
58     'multilabel_margin_loss',
59     'multilabel_soft_margin_loss',
60     'multi_margin_loss',
61     'nll_loss',
62     'binary_cross_entropy_with_logits',
63     'smooth_l1_loss',
64     'soft_margin_loss',
65     'triplet_margin_loss'
66 ]
67 
68 BANNED_FUNCS = [
69     ('binary_cross_entropy',
70      ("\namp does not work out-of-the-box with `F.binary_cross_entropy` or `torch.nn.BCELoss.` "
71       "It requires that the output of the previous function be already a FloatTensor. \n\n"
72       "Most models have a Sigmoid right before BCELoss. In that case, you can use\n"
73       "    torch.nn.BCEWithLogitsLoss\nto combine Sigmoid+BCELoss into a single layer "
74       "that is compatible with amp.\nAnother option is to add\n"
75       "    amp.register_float_function(torch, 'sigmoid')\nbefore calling `amp.init()`.\n"
76       "If you _really_ know what you are doing, you can disable this warning by passing "
77       "allow_banned=True to `amp.init()`."))
78 ]
79
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/apex/amp/lists/functional_overrides.py b/apex/amp/lists/functional_overrides.py
--- a/apex/amp/lists/functional_overrides.py
+++ b/apex/amp/lists/functional_overrides.py
@@ -37,7 +37,8 @@
     'softmin',
     'log_softmax',
     'softmax',
-
+    'gelu',
+    
     # Normalization
     'layer_norm',
     'group_norm',

{"golden_diff": "diff --git a/apex/amp/lists/functional_overrides.py b/apex/amp/lists/functional_overrides.py\n--- a/apex/amp/lists/functional_overrides.py\n+++ b/apex/amp/lists/functional_overrides.py\n@@ -37,7 +37,8 @@\n     'softmin',\n     'log_softmax',\n     'softmax',\n-\n+    'gelu',\n+    \n     # Normalization\n     'layer_norm',\n     'group_norm',\n", "issue": "RuntimeError: \"GeluCUDAKernelImpl\" not implemented for 'Half'\nPyTorch 1.2 introduced the `gelu` activation function. Unfortunately, this leads to terminal errors when using with AMP.\r\n\r\nTrace (`self.activation` is `gelu`): \r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"predict.py\", line 282, in <module>\r\n    predictor.predict()\r\n  File \"predict.py\", line 74, in predict\r\n    fig = trainer.train()\r\n  File \"/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/TransformerTrainer.py\", line 232, in train\r\n    self._process('train', epoch)\r\n  File \"/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/TransformerTrainer.py\", line 124, in _process\r\n    preds = self.model(input_ids, attention_mask=input_mask)\r\n  File \"/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 541, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/parallel/distributed.py\", line 442, in forward\r\n    output = self.module(*inputs[0], **kwargs[0])\r\n  File \"/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 541, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/models.py\", line 140, in forward\r\n    cls_output = self.activation(cls_output)\r\n  File \"/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/functional.py\", line 1126, in gelu\r\n    return torch._C._nn.gelu(input)\r\nRuntimeError: \"GeluCUDAKernelImpl\" not implemented for 'Half'\r\n```\n", "before_files": [{"content": "\n# TODO: think about the following two. They do weird things.\n# - torch.nn.utils.clip_grad (but it should always be fp32 anyway)\n# - torch.nn.utils.weight_norm\n\n# Notes:\n# F.instance_norm uses batch_norm internally. Which correctly handles\n#   fp16 in/out with fp32 weights. So we shouldn't do anything for\n#   either of these.\n# F.normalize calls `input.norm()` internally, so it's redundant, but\n#   kept here in case impl. changes.\n# F.cosine_similarity is same: calls `x.norm()` internally.\n\nimport torch.nn.functional\n\nMODULE = torch.nn.functional\n\nFP16_FUNCS = [\n    'conv1d',\n    'conv2d',\n    'conv3d',\n    'conv_transpose1d',\n    'conv_transpose2d',\n    'conv_transpose3d',\n    'conv_tbc', # Undocumented / maybe new?\n    'linear',\n]\n\nFP32_FUNCS = [\n\n    # Interpolation/Upsampling TODO:  Remove for 1.2\n    'interpolate',\n    'grid_sample',\n\n    # Pointwise\n    'softplus',\n    'softmin',\n    'log_softmax',\n    'softmax',\n\n    # Normalization\n    'layer_norm',\n    'group_norm',\n    'local_response_norm',\n    'normalize',\n    'cosine_similarity',\n\n    # Loss functions\n    # TODO: which of these can be fp16?\n    'poisson_nll_loss',\n    'cosine_embedding_loss',\n    'cross_entropy',\n    'hinge_embedding_loss',\n    'kl_div',\n    'l1_loss',\n    'mse_loss',\n    'margin_ranking_loss',\n    'multilabel_margin_loss',\n    'multilabel_soft_margin_loss',\n    'multi_margin_loss',\n    'nll_loss',\n    'binary_cross_entropy_with_logits',\n    'smooth_l1_loss',\n    'soft_margin_loss',\n    'triplet_margin_loss'\n]\n\nBANNED_FUNCS = [\n    ('binary_cross_entropy',\n     (\"\\namp does not work out-of-the-box with `F.binary_cross_entropy` or `torch.nn.BCELoss.` \"\n      \"It requires that the output of the previous function be already a FloatTensor. \\n\\n\"\n      \"Most models have a Sigmoid right before BCELoss. In that case, you can use\\n\"\n      \"    torch.nn.BCEWithLogitsLoss\\nto combine Sigmoid+BCELoss into a single layer \"\n      \"that is compatible with amp.\\nAnother option is to add\\n\"\n      \"    amp.register_float_function(torch, 'sigmoid')\\nbefore calling `amp.init()`.\\n\"\n      \"If you _really_ know what you are doing, you can disable this warning by passing \"\n      \"allow_banned=True to `amp.init()`.\"))\n]\n", "path": "apex/amp/lists/functional_overrides.py"}], "after_files": [{"content": "\n# TODO: think about the following two. They do weird things.\n# - torch.nn.utils.clip_grad (but it should always be fp32 anyway)\n# - torch.nn.utils.weight_norm\n\n# Notes:\n# F.instance_norm uses batch_norm internally. Which correctly handles\n#   fp16 in/out with fp32 weights. So we shouldn't do anything for\n#   either of these.\n# F.normalize calls `input.norm()` internally, so it's redundant, but\n#   kept here in case impl. changes.\n# F.cosine_similarity is same: calls `x.norm()` internally.\n\nimport torch.nn.functional\n\nMODULE = torch.nn.functional\n\nFP16_FUNCS = [\n    'conv1d',\n    'conv2d',\n    'conv3d',\n    'conv_transpose1d',\n    'conv_transpose2d',\n    'conv_transpose3d',\n    'conv_tbc', # Undocumented / maybe new?\n    'linear',\n]\n\nFP32_FUNCS = [\n\n    # Interpolation/Upsampling TODO:  Remove for 1.2\n    'interpolate',\n    'grid_sample',\n\n    # Pointwise\n    'softplus',\n    'softmin',\n    'log_softmax',\n    'softmax',\n    'gelu',\n    \n    # Normalization\n    'layer_norm',\n    'group_norm',\n    'local_response_norm',\n    'normalize',\n    'cosine_similarity',\n\n    # Loss functions\n    # TODO: which of these can be fp16?\n    'poisson_nll_loss',\n    'cosine_embedding_loss',\n    'cross_entropy',\n    'hinge_embedding_loss',\n    'kl_div',\n    'l1_loss',\n    'mse_loss',\n    'margin_ranking_loss',\n    'multilabel_margin_loss',\n    'multilabel_soft_margin_loss',\n    'multi_margin_loss',\n    'nll_loss',\n    'binary_cross_entropy_with_logits',\n    'smooth_l1_loss',\n    'soft_margin_loss',\n    'triplet_margin_loss'\n]\n\nBANNED_FUNCS = [\n    ('binary_cross_entropy',\n     (\"\\namp does not work out-of-the-box with `F.binary_cross_entropy` or `torch.nn.BCELoss.` \"\n      \"It requires that the output of the previous function be already a FloatTensor. \\n\\n\"\n      \"Most models have a Sigmoid right before BCELoss. In that case, you can use\\n\"\n      \"    torch.nn.BCEWithLogitsLoss\\nto combine Sigmoid+BCELoss into a single layer \"\n      \"that is compatible with amp.\\nAnother option is to add\\n\"\n      \"    amp.register_float_function(torch, 'sigmoid')\\nbefore calling `amp.init()`.\\n\"\n      \"If you _really_ know what you are doing, you can disable this warning by passing \"\n      \"allow_banned=True to `amp.init()`.\"))\n]\n", "path": "apex/amp/lists/functional_overrides.py"}]}