We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
clip_grad_global_norm doc needs notice on usage (was: clip_grad_global_norm produces problematic results)
I tried to use mulitple gpus to train the language model (e.g., AWD-LSTM), but the behaviour is not expected. I paste the training logs in the first 2 epochs as follows. I set hyper-parameters `alpha` and `beta` to zeros.

The logs with gradient clipping:

4 GPUS:
```
[Epoch 0 Batch 200/372] current loss 8.72, ppl 6128.99, throughput 660.63 samples/s, lr 29.57
[Epoch 0] throughput 45928.01 samples/s
[Epoch 0] time cost 52.74s, valid loss 8.34, valid ppl 4199.37，lr 30.00
[Epoch 0] test loss 8.31, test ppl 4053.50
[Epoch 1 Batch 200/372] current loss 8.47, ppl 4790.62, throughput 701.91 samples/s, lr 15.00
[Epoch 1] throughput 47520.37 samples/s
[Epoch 1] time cost 51.10s, valid loss 8.82, valid ppl 6737.68，lr 30.00
```

1 GPU:
```
[Epoch 0 Batch 200/372] current loss 7.70, ppl 2205.38, throughput 295.53 samples/s, lr 29.57
[Epoch 0] throughput 19927.64 samples/s
[Epoch 0] time cost 112.08s, valid loss 6.81, valid ppl 907.20，lr 30.00
[Epoch 0] test loss 6.74, test ppl 849.29
[Epoch 1 Batch 200/372] current loss 7.02, ppl 1116.47, throughput 302.28 samples/s, lr 15.00
[Epoch 1] throughput 20606.80 samples/s
[Epoch 1] time cost 108.55s, valid loss 6.51, valid ppl 671.14，lr 30.00
```

The logs without gradient clipping:

4 GPUS:
```
[Epoch 0 Batch 200/372] current loss 7.67, ppl 2153.44, throughput 775.13 samples/s, lr 29.57
[Epoch 0] throughput 53775.66 samples/s
[Epoch 0] time cost 46.28s, valid loss 6.78, valid ppl 881.91，lr 30.00
[Epoch 0] test loss 6.71, test ppl 821.79
[Epoch 1 Batch 200/372] current loss 7.00, ppl 1099.21, throughput 831.20 samples/s, lr 15.00
[Epoch 1] throughput 56021.61 samples/s
[Epoch 1] time cost 44.62s, valid loss 6.48, valid ppl 650.45，lr 30.00
```

1 GPU:
```
[Epoch 0 Batch 200/372] current loss 7.69, ppl 2182.02, throughput 309.02 samples/s, lr 29.57
[Epoch 0] throughput 20760.28 samples/s
[Epoch 0] time cost 107.76s, valid loss 6.76, valid ppl 865.22，lr 30.00
[Epoch 0] test loss 6.70, test ppl 809.79
[Epoch 1 Batch 200/372] current loss 7.01, ppl 1110.89, throughput 307.27 samples/s, lr 15.00
[Epoch 1] throughput 20919.05 samples/s
[Epoch 1] time cost 106.92s, valid loss 6.51, valid ppl 673.24，lr 30.00
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/gluonnlp/utils/parameter.py`
Content:
```
1 # coding: utf-8
2 
3 # Licensed to the Apache Software Foundation (ASF) under one
4 # or more contributor license agreements.  See the NOTICE file
5 # distributed with this work for additional information
6 # regarding copyright ownership.  The ASF licenses this file
7 # to you under the Apache License, Version 2.0 (the
8 # "License"); you may not use this file except in compliance
9 # with the License.  You may obtain a copy of the License at
10 #
11 #   http://www.apache.org/licenses/LICENSE-2.0
12 #
13 # Unless required by applicable law or agreed to in writing,
14 # software distributed under the License is distributed on an
15 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16 # KIND, either express or implied.  See the License for the
17 # specific language governing permissions and limitations
18 # under the License.
19 """Utility functions for parameters."""
20 
21 __all__ = ['clip_grad_global_norm']
22 
23 import warnings
24 
25 import numpy as np
26 from mxnet import nd
27 
28 def clip_grad_global_norm(parameters, max_norm, check_isfinite=True):
29     """Rescales gradients of parameters so that the sum of their 2-norm is smaller than `max_norm`.
30     If gradients exist for more than one context for a parameter, user needs to explicitly call
31     ``trainer.allreduce_grads`` so that the gradients are summed first before calculating
32     the 2-norm.
33 
34     .. note::
35 
36         This function is only for use when `update_on_kvstore` is set to False in trainer.
37 
38     Example::
39 
40         trainer = Trainer(net.collect_params(), update_on_kvstore=False, ...)
41         for x, y in mx.gluon.utils.split_and_load(X, [mx.gpu(0), mx.gpu(1)]):
42             with mx.autograd.record():
43                 y = net(x)
44                 loss = loss_fn(y, label)
45             loss.backward()
46         trainer.allreduce_grads()
47         nlp.utils.clip_grad_global_norm(net.collect_params().values(), max_norm)
48         trainer.update(batch_size)
49         ...
50 
51     Parameters
52     ----------
53     parameters : list of Parameters
54     max_norm : float
55     check_isfinite : bool, default True
56          If True, check that the total_norm is finite (not nan or inf). This
57          requires a blocking .asscalar() call.
58 
59     Returns
60     -------
61     NDArray or float
62       Total norm. Return type is NDArray of shape (1,) if check_isfinite is
63       False. Otherwise a float is returned.
64 
65     """
66     def _norm(array):
67         if array.stype == 'default':
68             x = array.reshape((-1))
69             return nd.dot(x, x)
70         return array.norm().square()
71 
72     arrays = []
73     i = 0
74     for p in parameters:
75         if p.grad_req != 'null':
76             grad_list = p.list_grad()
77             arrays.append(grad_list[i % len(grad_list)])
78             i += 1
79     assert len(arrays) > 0, 'No parameter found available for gradient norm clipping.'
80     ctx, dtype = arrays[0].context, arrays[0].dtype
81     total_norm = nd.add_n(*[_norm(arr).as_in_context(ctx) for arr in arrays])
82     total_norm = nd.sqrt(total_norm)
83     if check_isfinite:
84         total_norm = total_norm.asscalar()
85         if not np.isfinite(total_norm):
86             warnings.warn(
87                 UserWarning('nan or inf is detected. '
88                             'Clipping results will be undefined.'), stacklevel=2)
89     scale = max_norm / (total_norm + 1e-8)
90     if check_isfinite:
91         scale = nd.array([scale], dtype=dtype, ctx=ctx)
92     scale = nd.min(nd.concat(scale, nd.ones((1,), dtype=dtype, ctx=ctx), dim=0))
93     for p in parameters:
94         if p.grad_req != 'null':
95             for arr in p.list_grad():
96                 arr *= scale.as_in_context(arr.context)
97     return total_norm
98
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/gluonnlp/utils/parameter.py b/src/gluonnlp/utils/parameter.py
--- a/src/gluonnlp/utils/parameter.py
+++ b/src/gluonnlp/utils/parameter.py
@@ -34,6 +34,9 @@
     .. note::
 
         This function is only for use when `update_on_kvstore` is set to False in trainer.
+        In cases where training happens on multiple contexts, this method should be used in
+        conjunction with ``trainer.allreduce_grads()`` and ``trainer.update()``.
+        (**not** ``trainer.step()``)
 
     Example::

{"golden_diff": "diff --git a/src/gluonnlp/utils/parameter.py b/src/gluonnlp/utils/parameter.py\n--- a/src/gluonnlp/utils/parameter.py\n+++ b/src/gluonnlp/utils/parameter.py\n@@ -34,6 +34,9 @@\n     .. note::\n \n         This function is only for use when `update_on_kvstore` is set to False in trainer.\n+        In cases where training happens on multiple contexts, this method should be used in\n+        conjunction with ``trainer.allreduce_grads()`` and ``trainer.update()``.\n+        (**not** ``trainer.step()``)\n \n     Example::\n", "issue": "clip_grad_global_norm doc needs notice on usage (was: clip_grad_global_norm produces problematic results)\nI tried to use mulitple gpus to train the language model (e.g., AWD-LSTM), but the behaviour is not expected. I paste the training logs in the first 2 epochs as follows. I set hyper-parameters `alpha` and `beta` to zeros.\r\n\r\nThe logs with gradient clipping:\r\n\r\n4 GPUS:\r\n```\r\n[Epoch 0 Batch 200/372] current loss 8.72, ppl 6128.99, throughput 660.63 samples/s, lr 29.57\r\n[Epoch 0] throughput 45928.01 samples/s\r\n[Epoch 0] time cost 52.74s, valid loss 8.34, valid ppl 4199.37\uff0clr 30.00\r\n[Epoch 0] test loss 8.31, test ppl 4053.50\r\n[Epoch 1 Batch 200/372] current loss 8.47, ppl 4790.62, throughput 701.91 samples/s, lr 15.00\r\n[Epoch 1] throughput 47520.37 samples/s\r\n[Epoch 1] time cost 51.10s, valid loss 8.82, valid ppl 6737.68\uff0clr 30.00\r\n```\r\n\r\n1 GPU:\r\n```\r\n[Epoch 0 Batch 200/372] current loss 7.70, ppl 2205.38, throughput 295.53 samples/s, lr 29.57\r\n[Epoch 0] throughput 19927.64 samples/s\r\n[Epoch 0] time cost 112.08s, valid loss 6.81, valid ppl 907.20\uff0clr 30.00\r\n[Epoch 0] test loss 6.74, test ppl 849.29\r\n[Epoch 1 Batch 200/372] current loss 7.02, ppl 1116.47, throughput 302.28 samples/s, lr 15.00\r\n[Epoch 1] throughput 20606.80 samples/s\r\n[Epoch 1] time cost 108.55s, valid loss 6.51, valid ppl 671.14\uff0clr 30.00\r\n```\r\n\r\nThe logs without gradient clipping:\r\n\r\n4 GPUS:\r\n```\r\n[Epoch 0 Batch 200/372] current loss 7.67, ppl 2153.44, throughput 775.13 samples/s, lr 29.57\r\n[Epoch 0] throughput 53775.66 samples/s\r\n[Epoch 0] time cost 46.28s, valid loss 6.78, valid ppl 881.91\uff0clr 30.00\r\n[Epoch 0] test loss 6.71, test ppl 821.79\r\n[Epoch 1 Batch 200/372] current loss 7.00, ppl 1099.21, throughput 831.20 samples/s, lr 15.00\r\n[Epoch 1] throughput 56021.61 samples/s\r\n[Epoch 1] time cost 44.62s, valid loss 6.48, valid ppl 650.45\uff0clr 30.00\r\n```\r\n\r\n1 GPU:\r\n```\r\n[Epoch 0 Batch 200/372] current loss 7.69, ppl 2182.02, throughput 309.02 samples/s, lr 29.57\r\n[Epoch 0] throughput 20760.28 samples/s\r\n[Epoch 0] time cost 107.76s, valid loss 6.76, valid ppl 865.22\uff0clr 30.00\r\n[Epoch 0] test loss 6.70, test ppl 809.79\r\n[Epoch 1 Batch 200/372] current loss 7.01, ppl 1110.89, throughput 307.27 samples/s, lr 15.00\r\n[Epoch 1] throughput 20919.05 samples/s\r\n[Epoch 1] time cost 106.92s, valid loss 6.51, valid ppl 673.24\uff0clr 30.00\r\n```\n", "before_files": [{"content": "# coding: utf-8\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\"\"\"Utility functions for parameters.\"\"\"\n\n__all__ = ['clip_grad_global_norm']\n\nimport warnings\n\nimport numpy as np\nfrom mxnet import nd\n\ndef clip_grad_global_norm(parameters, max_norm, check_isfinite=True):\n    \"\"\"Rescales gradients of parameters so that the sum of their 2-norm is smaller than `max_norm`.\n    If gradients exist for more than one context for a parameter, user needs to explicitly call\n    ``trainer.allreduce_grads`` so that the gradients are summed first before calculating\n    the 2-norm.\n\n    .. note::\n\n        This function is only for use when `update_on_kvstore` is set to False in trainer.\n\n    Example::\n\n        trainer = Trainer(net.collect_params(), update_on_kvstore=False, ...)\n        for x, y in mx.gluon.utils.split_and_load(X, [mx.gpu(0), mx.gpu(1)]):\n            with mx.autograd.record():\n                y = net(x)\n                loss = loss_fn(y, label)\n            loss.backward()\n        trainer.allreduce_grads()\n        nlp.utils.clip_grad_global_norm(net.collect_params().values(), max_norm)\n        trainer.update(batch_size)\n        ...\n\n    Parameters\n    ----------\n    parameters : list of Parameters\n    max_norm : float\n    check_isfinite : bool, default True\n         If True, check that the total_norm is finite (not nan or inf). This\n         requires a blocking .asscalar() call.\n\n    Returns\n    -------\n    NDArray or float\n      Total norm. Return type is NDArray of shape (1,) if check_isfinite is\n      False. Otherwise a float is returned.\n\n    \"\"\"\n    def _norm(array):\n        if array.stype == 'default':\n            x = array.reshape((-1))\n            return nd.dot(x, x)\n        return array.norm().square()\n\n    arrays = []\n    i = 0\n    for p in parameters:\n        if p.grad_req != 'null':\n            grad_list = p.list_grad()\n            arrays.append(grad_list[i % len(grad_list)])\n            i += 1\n    assert len(arrays) > 0, 'No parameter found available for gradient norm clipping.'\n    ctx, dtype = arrays[0].context, arrays[0].dtype\n    total_norm = nd.add_n(*[_norm(arr).as_in_context(ctx) for arr in arrays])\n    total_norm = nd.sqrt(total_norm)\n    if check_isfinite:\n        total_norm = total_norm.asscalar()\n        if not np.isfinite(total_norm):\n            warnings.warn(\n                UserWarning('nan or inf is detected. '\n                            'Clipping results will be undefined.'), stacklevel=2)\n    scale = max_norm / (total_norm + 1e-8)\n    if check_isfinite:\n        scale = nd.array([scale], dtype=dtype, ctx=ctx)\n    scale = nd.min(nd.concat(scale, nd.ones((1,), dtype=dtype, ctx=ctx), dim=0))\n    for p in parameters:\n        if p.grad_req != 'null':\n            for arr in p.list_grad():\n                arr *= scale.as_in_context(arr.context)\n    return total_norm\n", "path": "src/gluonnlp/utils/parameter.py"}], "after_files": [{"content": "# coding: utf-8\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\"\"\"Utility functions for parameters.\"\"\"\n\n__all__ = ['clip_grad_global_norm']\n\nimport warnings\n\nimport numpy as np\nfrom mxnet import nd\n\ndef clip_grad_global_norm(parameters, max_norm, check_isfinite=True):\n    \"\"\"Rescales gradients of parameters so that the sum of their 2-norm is smaller than `max_norm`.\n    If gradients exist for more than one context for a parameter, user needs to explicitly call\n    ``trainer.allreduce_grads`` so that the gradients are summed first before calculating\n    the 2-norm.\n\n    .. note::\n\n        This function is only for use when `update_on_kvstore` is set to False in trainer.\n        In cases where training happens on multiple contexts, this method should be used in\n        conjunction with ``trainer.allreduce_grads()`` and ``trainer.update()``.\n        (**not** ``trainer.step()``)\n\n    Example::\n\n        trainer = Trainer(net.collect_params(), update_on_kvstore=False, ...)\n        for x, y in mx.gluon.utils.split_and_load(X, [mx.gpu(0), mx.gpu(1)]):\n            with mx.autograd.record():\n                y = net(x)\n                loss = loss_fn(y, label)\n            loss.backward()\n        trainer.allreduce_grads()\n        nlp.utils.clip_grad_global_norm(net.collect_params().values(), max_norm)\n        trainer.update(batch_size)\n        ...\n\n    Parameters\n    ----------\n    parameters : list of Parameters\n    max_norm : float\n    check_isfinite : bool, default True\n         If True, check that the total_norm is finite (not nan or inf). This\n         requires a blocking .asscalar() call.\n\n    Returns\n    -------\n    NDArray or float\n      Total norm. Return type is NDArray of shape (1,) if check_isfinite is\n      False. Otherwise a float is returned.\n\n    \"\"\"\n    def _norm(array):\n        if array.stype == 'default':\n            x = array.reshape((-1))\n            return nd.dot(x, x)\n        return array.norm().square()\n\n    arrays = []\n    i = 0\n    for p in parameters:\n        if p.grad_req != 'null':\n            grad_list = p.list_grad()\n            arrays.append(grad_list[i % len(grad_list)])\n            i += 1\n    assert len(arrays) > 0, 'No parameter found available for gradient norm clipping.'\n    ctx, dtype = arrays[0].context, arrays[0].dtype\n    total_norm = nd.add_n(*[_norm(arr).as_in_context(ctx) for arr in arrays])\n    total_norm = nd.sqrt(total_norm)\n    if check_isfinite:\n        total_norm = total_norm.asscalar()\n        if not np.isfinite(total_norm):\n            warnings.warn(\n                UserWarning('nan or inf is detected. '\n                            'Clipping results will be undefined.'), stacklevel=2)\n    scale = max_norm / (total_norm + 1e-8)\n    if check_isfinite:\n        scale = nd.array([scale], dtype=dtype, ctx=ctx)\n    scale = nd.min(nd.concat(scale, nd.ones((1,), dtype=dtype, ctx=ctx), dim=0))\n    for p in parameters:\n        if p.grad_req != 'null':\n            for arr in p.list_grad():\n                arr *= scale.as_in_context(arr.context)\n    return total_norm\n", "path": "src/gluonnlp/utils/parameter.py"}]}