We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Call to torch.pow() passing integer as exponent isn't per PyTorch docs
# 🐛 Bug

## Information

I am running the GPT2 pytest and it fails to load in [PyTorch Glow](https://github.com/pytorch/glow) because the model calls _torch.pow()_ with an integer for the _exponent_ parameter.

Per the PyTorch documentation (https://pytorch.org/docs/master/torch.html?highlight=torch%20pow#torch.pow):

> exponent can be either a single float number or a Tensor with the same number of elements as input.

and

>exponent (float or tensor) – the exponent value

The test was run with the following modifications to enable Glow:

```
diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py
index 12013996..d6f39007 100644
--- a/src/transformers/modeling_gpt2.py
+++ b/src/transformers/modeling_gpt2.py
@@ -265,7 +265,7 @@ class GPT2PreTrainedModel(PreTrainedModel):
         if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):
             # Slightly different from the TF version which uses truncated_normal for initialization
             # cf https://github.com/pytorch/pytorch/pull/5617
-            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            module.weight.data.normal_(mean=0.0, std=0.02) #self.config.initializer_range)
             if isinstance(module, (nn.Linear, Conv1D)) and module.bias is not None:
                 module.bias.data.zero_()
         elif isinstance(module, nn.LayerNorm):
diff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py
index 1d11ef8c..9df209f7 100644
--- a/tests/test_modeling_common.py
+++ b/tests/test_modeling_common.py
@@ -27,6 +27,7 @@ from .utils import require_torch, slow, torch_device
 
 if is_torch_available():
     import torch
+    import torch_glow
     import numpy as np
 
     from transformers import (
@@ -209,6 +210,8 @@ class ModelTesterMixin:
             inputs = inputs_dict["input_ids"]  # Let's keep only input_ids
 
             try:
+                torch_glow.enableFusionPass()
+                torch_glow.setGlowBackend('Interpreter')
                 traced_gpt2 = torch.jit.trace(model, inputs)
             except RuntimeError:
                 self.fail("Couldn't trace module.")
```

## To reproduce

Steps to reproduce the behavior:

1. python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py

## Expected behavior

Expect exponent to be passed as a float per the documentation so that model loaders adhering to the docs will be able to load the model.

## Environment info
     
- `transformers` version: 2.8.0                                                                                                                                                                                                         
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.8                                                                                                                                                                                                                 
- PyTorch version (GPU?): 1.5.0a0+8eaafbd (False)
- Tensorflow version (GPU?): not installed (NA)                                                                                                                                                                                         
- Using GPU in script?: no                                                                                  
- Using distributed or parallel set-up in script?: no 

Call to torch.pow() passing integer as exponent isn't per PyTorch docs
# 🐛 Bug

## Information

I am running the GPT2 pytest and it fails to load in [PyTorch Glow](https://github.com/pytorch/glow) because the model calls _torch.pow()_ with an integer for the _exponent_ parameter.

Per the PyTorch documentation (https://pytorch.org/docs/master/torch.html?highlight=torch%20pow#torch.pow):

> exponent can be either a single float number or a Tensor with the same number of elements as input.

and

>exponent (float or tensor) – the exponent value

The test was run with the following modifications to enable Glow:

```
diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py
index 12013996..d6f39007 100644
--- a/src/transformers/modeling_gpt2.py
+++ b/src/transformers/modeling_gpt2.py
@@ -265,7 +265,7 @@ class GPT2PreTrainedModel(PreTrainedModel):
         if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):
             # Slightly different from the TF version which uses truncated_normal for initialization
             # cf https://github.com/pytorch/pytorch/pull/5617
-            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            module.weight.data.normal_(mean=0.0, std=0.02) #self.config.initializer_range)
             if isinstance(module, (nn.Linear, Conv1D)) and module.bias is not None:
                 module.bias.data.zero_()
         elif isinstance(module, nn.LayerNorm):
diff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py
index 1d11ef8c..9df209f7 100644
--- a/tests/test_modeling_common.py
+++ b/tests/test_modeling_common.py
@@ -27,6 +27,7 @@ from .utils import require_torch, slow, torch_device
 
 if is_torch_available():
     import torch
+    import torch_glow
     import numpy as np
 
     from transformers import (
@@ -209,6 +210,8 @@ class ModelTesterMixin:
             inputs = inputs_dict["input_ids"]  # Let's keep only input_ids
 
             try:
+                torch_glow.enableFusionPass()
+                torch_glow.setGlowBackend('Interpreter')
                 traced_gpt2 = torch.jit.trace(model, inputs)
             except RuntimeError:
                 self.fail("Couldn't trace module.")
```

## To reproduce

Steps to reproduce the behavior:

1. python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py

## Expected behavior

Expect exponent to be passed as a float per the documentation so that model loaders adhering to the docs will be able to load the model.

## Environment info
     
- `transformers` version: 2.8.0                                                                                                                                                                                                         
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.8                                                                                                                                                                                                                 
- PyTorch version (GPU?): 1.5.0a0+8eaafbd (False)
- Tensorflow version (GPU?): not installed (NA)                                                                                                                                                                                         
- Using GPU in script?: no                                                                                  
- Using distributed or parallel set-up in script?: no 


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/transformers/activations.py`
Content:
```
1 import logging
2 import math
3 
4 import torch
5 import torch.nn.functional as F
6 
7 
8 logger = logging.getLogger(__name__)
9 
10 
11 def swish(x):
12     return x * torch.sigmoid(x)
13 
14 
15 def _gelu_python(x):
16     """ Original Implementation of the gelu activation function in Google Bert repo when initially created.
17         For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
18         0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
19         This is now written in C in torch.nn.functional
20         Also see https://arxiv.org/abs/1606.08415
21     """
22     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
23 
24 
25 def gelu_new(x):
26     """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
27         Also see https://arxiv.org/abs/1606.08415
28     """
29     return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
30 
31 
32 if torch.__version__ < "1.4.0":
33     gelu = _gelu_python
34 else:
35     gelu = F.gelu
36     try:
37         import torch_xla  # noqa F401
38 
39         logger.warning(
40             "The torch_xla package was detected in the python environment. PyTorch/XLA and JIT is untested,"
41             " no activation function will be traced with JIT."
42         )
43     except ImportError:
44         gelu_new = torch.jit.script(gelu_new)
45 
46 ACT2FN = {
47     "relu": F.relu,
48     "swish": swish,
49     "gelu": gelu,
50     "tanh": torch.tanh,
51     "gelu_new": gelu_new,
52 }
53 
54 
55 def get_activation(activation_string):
56     if activation_string in ACT2FN:
57         return ACT2FN[activation_string]
58     else:
59         raise KeyError("function {} not found in ACT2FN mapping {}".format(activation_string, list(ACT2FN.keys())))
60
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/transformers/activations.py b/src/transformers/activations.py
--- a/src/transformers/activations.py
+++ b/src/transformers/activations.py
@@ -26,7 +26,7 @@
     """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
         Also see https://arxiv.org/abs/1606.08415
     """
-    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
 
 
 if torch.__version__ < "1.4.0":

{"golden_diff": "diff --git a/src/transformers/activations.py b/src/transformers/activations.py\n--- a/src/transformers/activations.py\n+++ b/src/transformers/activations.py\n@@ -26,7 +26,7 @@\n     \"\"\" Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).\n         Also see https://arxiv.org/abs/1606.08415\n     \"\"\"\n-    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))\n+    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))\n \n \n if torch.__version__ < \"1.4.0\":\n", "issue": "Call to torch.pow() passing integer as exponent isn't per PyTorch docs\n# \ud83d\udc1b Bug\r\n\r\n## Information\r\n\r\nI am running the GPT2 pytest and it fails to load in [PyTorch Glow](https://github.com/pytorch/glow) because the model calls _torch.pow()_ with an integer for the _exponent_ parameter.\r\n\r\nPer the PyTorch documentation (https://pytorch.org/docs/master/torch.html?highlight=torch%20pow#torch.pow):\r\n\r\n> exponent can be either a single float number or a Tensor with the same number of elements as input.\r\n\r\nand\r\n\r\n>exponent (float or tensor) \u2013 the exponent value\r\n\r\nThe test was run with the following modifications to enable Glow:\r\n\r\n```\r\ndiff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py\r\nindex 12013996..d6f39007 100644\r\n--- a/src/transformers/modeling_gpt2.py\r\n+++ b/src/transformers/modeling_gpt2.py\r\n@@ -265,7 +265,7 @@ class GPT2PreTrainedModel(PreTrainedModel):\r\n         if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):\r\n             # Slightly different from the TF version which uses truncated_normal for initialization\r\n             # cf https://github.com/pytorch/pytorch/pull/5617\r\n-            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\r\n+            module.weight.data.normal_(mean=0.0, std=0.02) #self.config.initializer_range)\r\n             if isinstance(module, (nn.Linear, Conv1D)) and module.bias is not None:\r\n                 module.bias.data.zero_()\r\n         elif isinstance(module, nn.LayerNorm):\r\ndiff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py\r\nindex 1d11ef8c..9df209f7 100644\r\n--- a/tests/test_modeling_common.py\r\n+++ b/tests/test_modeling_common.py\r\n@@ -27,6 +27,7 @@ from .utils import require_torch, slow, torch_device\r\n \r\n if is_torch_available():\r\n     import torch\r\n+    import torch_glow\r\n     import numpy as np\r\n \r\n     from transformers import (\r\n@@ -209,6 +210,8 @@ class ModelTesterMixin:\r\n             inputs = inputs_dict[\"input_ids\"]  # Let's keep only input_ids\r\n \r\n             try:\r\n+                torch_glow.enableFusionPass()\r\n+                torch_glow.setGlowBackend('Interpreter')\r\n                 traced_gpt2 = torch.jit.trace(model, inputs)\r\n             except RuntimeError:\r\n                 self.fail(\"Couldn't trace module.\")\r\n```\r\n\r\n## To reproduce\r\n\r\nSteps to reproduce the behavior:\r\n\r\n1. python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py\r\n\r\n## Expected behavior\r\n\r\nExpect exponent to be passed as a float per the documentation so that model loaders adhering to the docs will be able to load the model.\r\n\r\n## Environment info\r\n     \r\n- `transformers` version: 2.8.0                                                                                                                                                                                                         \r\n- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core\r\n- Python version: 3.6.8                                                                                                                                                                                                                 \r\n- PyTorch version (GPU?): 1.5.0a0+8eaafbd (False)\r\n- Tensorflow version (GPU?): not installed (NA)                                                                                                                                                                                         \r\n- Using GPU in script?: no                                                                                  \r\n- Using distributed or parallel set-up in script?: no \r\n\nCall to torch.pow() passing integer as exponent isn't per PyTorch docs\n# \ud83d\udc1b Bug\r\n\r\n## Information\r\n\r\nI am running the GPT2 pytest and it fails to load in [PyTorch Glow](https://github.com/pytorch/glow) because the model calls _torch.pow()_ with an integer for the _exponent_ parameter.\r\n\r\nPer the PyTorch documentation (https://pytorch.org/docs/master/torch.html?highlight=torch%20pow#torch.pow):\r\n\r\n> exponent can be either a single float number or a Tensor with the same number of elements as input.\r\n\r\nand\r\n\r\n>exponent (float or tensor) \u2013 the exponent value\r\n\r\nThe test was run with the following modifications to enable Glow:\r\n\r\n```\r\ndiff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py\r\nindex 12013996..d6f39007 100644\r\n--- a/src/transformers/modeling_gpt2.py\r\n+++ b/src/transformers/modeling_gpt2.py\r\n@@ -265,7 +265,7 @@ class GPT2PreTrainedModel(PreTrainedModel):\r\n         if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):\r\n             # Slightly different from the TF version which uses truncated_normal for initialization\r\n             # cf https://github.com/pytorch/pytorch/pull/5617\r\n-            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\r\n+            module.weight.data.normal_(mean=0.0, std=0.02) #self.config.initializer_range)\r\n             if isinstance(module, (nn.Linear, Conv1D)) and module.bias is not None:\r\n                 module.bias.data.zero_()\r\n         elif isinstance(module, nn.LayerNorm):\r\ndiff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py\r\nindex 1d11ef8c..9df209f7 100644\r\n--- a/tests/test_modeling_common.py\r\n+++ b/tests/test_modeling_common.py\r\n@@ -27,6 +27,7 @@ from .utils import require_torch, slow, torch_device\r\n \r\n if is_torch_available():\r\n     import torch\r\n+    import torch_glow\r\n     import numpy as np\r\n \r\n     from transformers import (\r\n@@ -209,6 +210,8 @@ class ModelTesterMixin:\r\n             inputs = inputs_dict[\"input_ids\"]  # Let's keep only input_ids\r\n \r\n             try:\r\n+                torch_glow.enableFusionPass()\r\n+                torch_glow.setGlowBackend('Interpreter')\r\n                 traced_gpt2 = torch.jit.trace(model, inputs)\r\n             except RuntimeError:\r\n                 self.fail(\"Couldn't trace module.\")\r\n```\r\n\r\n## To reproduce\r\n\r\nSteps to reproduce the behavior:\r\n\r\n1. python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py\r\n\r\n## Expected behavior\r\n\r\nExpect exponent to be passed as a float per the documentation so that model loaders adhering to the docs will be able to load the model.\r\n\r\n## Environment info\r\n     \r\n- `transformers` version: 2.8.0                                                                                                                                                                                                         \r\n- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core\r\n- Python version: 3.6.8                                                                                                                                                                                                                 \r\n- PyTorch version (GPU?): 1.5.0a0+8eaafbd (False)\r\n- Tensorflow version (GPU?): not installed (NA)                                                                                                                                                                                         \r\n- Using GPU in script?: no                                                                                  \r\n- Using distributed or parallel set-up in script?: no \r\n\n", "before_files": [{"content": "import logging\nimport math\n\nimport torch\nimport torch.nn.functional as F\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef swish(x):\n    return x * torch.sigmoid(x)\n\n\ndef _gelu_python(x):\n    \"\"\" Original Implementation of the gelu activation function in Google Bert repo when initially created.\n        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):\n        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))\n        This is now written in C in torch.nn.functional\n        Also see https://arxiv.org/abs/1606.08415\n    \"\"\"\n    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))\n\n\ndef gelu_new(x):\n    \"\"\" Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).\n        Also see https://arxiv.org/abs/1606.08415\n    \"\"\"\n    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))\n\n\nif torch.__version__ < \"1.4.0\":\n    gelu = _gelu_python\nelse:\n    gelu = F.gelu\n    try:\n        import torch_xla  # noqa F401\n\n        logger.warning(\n            \"The torch_xla package was detected in the python environment. PyTorch/XLA and JIT is untested,\"\n            \" no activation function will be traced with JIT.\"\n        )\n    except ImportError:\n        gelu_new = torch.jit.script(gelu_new)\n\nACT2FN = {\n    \"relu\": F.relu,\n    \"swish\": swish,\n    \"gelu\": gelu,\n    \"tanh\": torch.tanh,\n    \"gelu_new\": gelu_new,\n}\n\n\ndef get_activation(activation_string):\n    if activation_string in ACT2FN:\n        return ACT2FN[activation_string]\n    else:\n        raise KeyError(\"function {} not found in ACT2FN mapping {}\".format(activation_string, list(ACT2FN.keys())))\n", "path": "src/transformers/activations.py"}], "after_files": [{"content": "import logging\nimport math\n\nimport torch\nimport torch.nn.functional as F\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef swish(x):\n    return x * torch.sigmoid(x)\n\n\ndef _gelu_python(x):\n    \"\"\" Original Implementation of the gelu activation function in Google Bert repo when initially created.\n        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):\n        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))\n        This is now written in C in torch.nn.functional\n        Also see https://arxiv.org/abs/1606.08415\n    \"\"\"\n    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))\n\n\ndef gelu_new(x):\n    \"\"\" Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).\n        Also see https://arxiv.org/abs/1606.08415\n    \"\"\"\n    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))\n\n\nif torch.__version__ < \"1.4.0\":\n    gelu = _gelu_python\nelse:\n    gelu = F.gelu\n    try:\n        import torch_xla  # noqa F401\n\n        logger.warning(\n            \"The torch_xla package was detected in the python environment. PyTorch/XLA and JIT is untested,\"\n            \" no activation function will be traced with JIT.\"\n        )\n    except ImportError:\n        gelu_new = torch.jit.script(gelu_new)\n\nACT2FN = {\n    \"relu\": F.relu,\n    \"swish\": swish,\n    \"gelu\": gelu,\n    \"tanh\": torch.tanh,\n    \"gelu_new\": gelu_new,\n}\n\n\ndef get_activation(activation_string):\n    if activation_string in ACT2FN:\n        return ACT2FN[activation_string]\n    else:\n        raise KeyError(\"function {} not found in ACT2FN mapping {}\".format(activation_string, list(ACT2FN.keys())))\n", "path": "src/transformers/activations.py"}]}