We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
numerical_grad caution
I was implementing a differentiable Transpose function. 

```
class Transpose(Function):
    def forward(self, inputs):
        x = inputs[0]
        return x.transpose(),
    def backward(self, inputs, grads):
        return grads[0].transpose(),
```

While testing the gradient with numerical_grad, 

```
    def test_numerical_grad_cpu(self):
        x = np.random.randn(1, 10)
        x_var = Variable(x)
        y_var = self.f(x_var)
        y_var.grad = np.random.rand(10, 1)
        y_var.backward()
        cl = lambda: self.f.forward((x,))
        gx, = gradient_check.numerical_grad(cl, (x,), (y_var.grad,))
        gradient_check.assert_allclose(gx, x_var.grad)
```

(here `self.f = Transpose()`)
the numerical gradient `gx` keeps coming back as 0. After much frustration, I finally figured out that I was returning a view of `x` in the above code, and in `numerical_grad_cpu`,

```
            flat_x[i] = orig + eps
            ys1 = f()
            flat_x[i] = orig - eps
            ys2 = f()
            flat_x[i] = orig
```

`ys1` and `ys2` end up being equal after the last line resetting `flat_x[i]` to the original value. I solved my problem by changing `cl = lambda: self.f.forward((x,))` to `cl = lambda: np.copy(self.f.forward((x,)))`.

I'm not sure how frequent this phenomenon could occur outside of transpose, but I just wanted to put this out here so that there could be a discussion. Perhaps a passing note in the documentation suffices here. Or doing `ys1 = np.copy(f())` instead might work as well.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `chainer/gradient_check.py`
Content:
```
1 import numpy
2 import six
3 
4 from chainer import cuda
5 from chainer import utils
6 
7 
8 def numerical_grad_cpu(f, inputs, grad_outputs, eps=1e-3):
9     grads = tuple(numpy.zeros_like(x) for x in inputs)
10     for x, gx in zip(inputs, grads):
11         flat_x = x.ravel()
12         flat_gx = gx.ravel()
13         for i in six.moves.range(flat_x.size):
14             orig = flat_x[i]
15             flat_x[i] = orig + eps
16             ys1 = f()
17             flat_x[i] = orig - eps
18             ys2 = f()
19             flat_x[i] = orig
20 
21             for y1, y2, gy in zip(ys1, ys2, grad_outputs):
22                 if gy is not None:
23                     dot = float(sum(((y1 - y2) * gy).ravel()))
24                     flat_gx[i] += dot / (2 * eps)
25 
26     return grads
27 
28 
29 def numerical_grad_gpu(f, inputs, grad_outputs, eps=1e-3):
30     grads = tuple(cuda.zeros_like(x) for x in inputs)
31     for x, gx in zip(inputs, grads):
32         x = x.ravel()
33         gx = gx.ravel()
34         x_cpu = x.get()
35         gx_cpu = gx.get()
36         for i in six.moves.range(x_cpu.size):
37             orig = x_cpu[i]
38             x_cpu[i] = orig + eps
39             x.set(x_cpu)
40             ys1 = f()
41             x_cpu[i] = orig - eps
42             x.set(x_cpu)
43             ys2 = f()
44             x_cpu[i] = orig
45             x.set(x_cpu)
46 
47             for y1, y2, gy in zip(ys1, ys2, grad_outputs):
48                 if gy is not None:
49                     dot = sum(((y1 - y2) * gy).ravel()).get()
50                     gx_cpu[i] += dot / (2 * eps)
51         gx.set(gx_cpu)
52 
53     return grads
54 
55 
56 def numerical_grad(f, inputs, grad_outputs, eps=1e-3):
57     """Computes numerical gradient by finite differences.
58 
59     This function is used to implement gradient check. For usage example, see
60     unit tests of :mod:`chainer.functions`.
61 
62     Args:
63         f (function): Python function with no arguments that runs forward
64             computation and returns the result.
65         inputs (tuple of arrays): Tuple of arrays that should be treated as
66             inputs. Each element of them is slightly modified to realize
67             numerical gradient by finite differences.
68         grad_outputs (tuple of arrays): Tuple of arrays that are treated as
69             output gradients.
70         eps (float): Epsilon value of finite differences.
71 
72     Returns:
73         tuple: Numerical gradient arrays corresponding to ``inputs``.
74 
75     """
76     assert eps > 0
77     inputs = tuple(inputs)
78     grad_outputs = tuple(grad_outputs)
79     gpu = any(isinstance(x, cuda.ndarray) for x in inputs + grad_outputs)
80 
81     cpu = any(isinstance(x, numpy.ndarray) for x in inputs + grad_outputs)
82 
83     if gpu and cpu:
84         raise RuntimeError('Do not mix GPU and CPU arrays in `numerical_grad`')
85     elif gpu:
86         return numerical_grad_gpu(f, inputs, grad_outputs, eps)
87     else:
88         return numerical_grad_cpu(f, inputs, grad_outputs, eps)
89 
90 
91 def assert_allclose(x, y, atol=1e-5, rtol=1e-4, verbose=True):
92     """Asserts if some corresponding element of x and y differs too much.
93 
94     This function can handle both CPU and GPU arrays simultaneously.
95 
96     Args:
97         x: Left-hand-side array.
98         y: Right-hand-side array.
99         atol (float): Absolute tolerance.
100         rtol (float): Relative tolerance.
101         verbose (bool): If True, it outputs verbose messages on error.
102 
103     """
104     x = cuda.to_cpu(utils.force_array(x))
105     y = cuda.to_cpu(utils.force_array(y))
106     try:
107         numpy.testing.assert_allclose(
108             x, y, atol=atol, rtol=rtol, verbose=verbose)
109     except Exception:
110         print('error:', numpy.abs(x - y).max())
111         raise
112
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/chainer/gradient_check.py b/chainer/gradient_check.py
--- a/chainer/gradient_check.py
+++ b/chainer/gradient_check.py
@@ -5,6 +5,11 @@
 from chainer import utils
 
 
+def _copy_arrays(xs):
+    xp = cuda.get_array_module(*xs)
+    return tuple(xp.copy(x) for x in xs)
+
+
 def numerical_grad_cpu(f, inputs, grad_outputs, eps=1e-3):
     grads = tuple(numpy.zeros_like(x) for x in inputs)
     for x, gx in zip(inputs, grads):
@@ -13,9 +18,9 @@
         for i in six.moves.range(flat_x.size):
             orig = flat_x[i]
             flat_x[i] = orig + eps
-            ys1 = f()
+            ys1 = _copy_arrays(f())
             flat_x[i] = orig - eps
-            ys2 = f()
+            ys2 = _copy_arrays(f())
             flat_x[i] = orig
 
             for y1, y2, gy in zip(ys1, ys2, grad_outputs):
@@ -37,10 +42,10 @@
             orig = x_cpu[i]
             x_cpu[i] = orig + eps
             x.set(x_cpu)
-            ys1 = f()
+            ys1 = _copy_arrays(f())
             x_cpu[i] = orig - eps
             x.set(x_cpu)
-            ys2 = f()
+            ys2 = _copy_arrays(f())
             x_cpu[i] = orig
             x.set(x_cpu)

{"golden_diff": "diff --git a/chainer/gradient_check.py b/chainer/gradient_check.py\n--- a/chainer/gradient_check.py\n+++ b/chainer/gradient_check.py\n@@ -5,6 +5,11 @@\n from chainer import utils\n \n \n+def _copy_arrays(xs):\n+    xp = cuda.get_array_module(*xs)\n+    return tuple(xp.copy(x) for x in xs)\n+\n+\n def numerical_grad_cpu(f, inputs, grad_outputs, eps=1e-3):\n     grads = tuple(numpy.zeros_like(x) for x in inputs)\n     for x, gx in zip(inputs, grads):\n@@ -13,9 +18,9 @@\n         for i in six.moves.range(flat_x.size):\n             orig = flat_x[i]\n             flat_x[i] = orig + eps\n-            ys1 = f()\n+            ys1 = _copy_arrays(f())\n             flat_x[i] = orig - eps\n-            ys2 = f()\n+            ys2 = _copy_arrays(f())\n             flat_x[i] = orig\n \n             for y1, y2, gy in zip(ys1, ys2, grad_outputs):\n@@ -37,10 +42,10 @@\n             orig = x_cpu[i]\n             x_cpu[i] = orig + eps\n             x.set(x_cpu)\n-            ys1 = f()\n+            ys1 = _copy_arrays(f())\n             x_cpu[i] = orig - eps\n             x.set(x_cpu)\n-            ys2 = f()\n+            ys2 = _copy_arrays(f())\n             x_cpu[i] = orig\n             x.set(x_cpu)\n", "issue": "numerical_grad caution\nI was implementing a differentiable Transpose function. \n\n```\nclass Transpose(Function):\n    def forward(self, inputs):\n        x = inputs[0]\n        return x.transpose(),\n    def backward(self, inputs, grads):\n        return grads[0].transpose(),\n```\n\nWhile testing the gradient with numerical_grad, \n\n```\n    def test_numerical_grad_cpu(self):\n        x = np.random.randn(1, 10)\n        x_var = Variable(x)\n        y_var = self.f(x_var)\n        y_var.grad = np.random.rand(10, 1)\n        y_var.backward()\n        cl = lambda: self.f.forward((x,))\n        gx, = gradient_check.numerical_grad(cl, (x,), (y_var.grad,))\n        gradient_check.assert_allclose(gx, x_var.grad)\n```\n\n(here `self.f = Transpose()`)\nthe numerical gradient `gx` keeps coming back as 0. After much frustration, I finally figured out that I was returning a view of `x` in the above code, and in `numerical_grad_cpu`,\n\n```\n            flat_x[i] = orig + eps\n            ys1 = f()\n            flat_x[i] = orig - eps\n            ys2 = f()\n            flat_x[i] = orig\n```\n\n`ys1` and `ys2` end up being equal after the last line resetting `flat_x[i]` to the original value. I solved my problem by changing `cl = lambda: self.f.forward((x,))` to `cl = lambda: np.copy(self.f.forward((x,)))`.\n\nI'm not sure how frequent this phenomenon could occur outside of transpose, but I just wanted to put this out here so that there could be a discussion. Perhaps a passing note in the documentation suffices here. Or doing `ys1 = np.copy(f())` instead might work as well.\n\n", "before_files": [{"content": "import numpy\nimport six\n\nfrom chainer import cuda\nfrom chainer import utils\n\n\ndef numerical_grad_cpu(f, inputs, grad_outputs, eps=1e-3):\n    grads = tuple(numpy.zeros_like(x) for x in inputs)\n    for x, gx in zip(inputs, grads):\n        flat_x = x.ravel()\n        flat_gx = gx.ravel()\n        for i in six.moves.range(flat_x.size):\n            orig = flat_x[i]\n            flat_x[i] = orig + eps\n            ys1 = f()\n            flat_x[i] = orig - eps\n            ys2 = f()\n            flat_x[i] = orig\n\n            for y1, y2, gy in zip(ys1, ys2, grad_outputs):\n                if gy is not None:\n                    dot = float(sum(((y1 - y2) * gy).ravel()))\n                    flat_gx[i] += dot / (2 * eps)\n\n    return grads\n\n\ndef numerical_grad_gpu(f, inputs, grad_outputs, eps=1e-3):\n    grads = tuple(cuda.zeros_like(x) for x in inputs)\n    for x, gx in zip(inputs, grads):\n        x = x.ravel()\n        gx = gx.ravel()\n        x_cpu = x.get()\n        gx_cpu = gx.get()\n        for i in six.moves.range(x_cpu.size):\n            orig = x_cpu[i]\n            x_cpu[i] = orig + eps\n            x.set(x_cpu)\n            ys1 = f()\n            x_cpu[i] = orig - eps\n            x.set(x_cpu)\n            ys2 = f()\n            x_cpu[i] = orig\n            x.set(x_cpu)\n\n            for y1, y2, gy in zip(ys1, ys2, grad_outputs):\n                if gy is not None:\n                    dot = sum(((y1 - y2) * gy).ravel()).get()\n                    gx_cpu[i] += dot / (2 * eps)\n        gx.set(gx_cpu)\n\n    return grads\n\n\ndef numerical_grad(f, inputs, grad_outputs, eps=1e-3):\n    \"\"\"Computes numerical gradient by finite differences.\n\n    This function is used to implement gradient check. For usage example, see\n    unit tests of :mod:`chainer.functions`.\n\n    Args:\n        f (function): Python function with no arguments that runs forward\n            computation and returns the result.\n        inputs (tuple of arrays): Tuple of arrays that should be treated as\n            inputs. Each element of them is slightly modified to realize\n            numerical gradient by finite differences.\n        grad_outputs (tuple of arrays): Tuple of arrays that are treated as\n            output gradients.\n        eps (float): Epsilon value of finite differences.\n\n    Returns:\n        tuple: Numerical gradient arrays corresponding to ``inputs``.\n\n    \"\"\"\n    assert eps > 0\n    inputs = tuple(inputs)\n    grad_outputs = tuple(grad_outputs)\n    gpu = any(isinstance(x, cuda.ndarray) for x in inputs + grad_outputs)\n\n    cpu = any(isinstance(x, numpy.ndarray) for x in inputs + grad_outputs)\n\n    if gpu and cpu:\n        raise RuntimeError('Do not mix GPU and CPU arrays in `numerical_grad`')\n    elif gpu:\n        return numerical_grad_gpu(f, inputs, grad_outputs, eps)\n    else:\n        return numerical_grad_cpu(f, inputs, grad_outputs, eps)\n\n\ndef assert_allclose(x, y, atol=1e-5, rtol=1e-4, verbose=True):\n    \"\"\"Asserts if some corresponding element of x and y differs too much.\n\n    This function can handle both CPU and GPU arrays simultaneously.\n\n    Args:\n        x: Left-hand-side array.\n        y: Right-hand-side array.\n        atol (float): Absolute tolerance.\n        rtol (float): Relative tolerance.\n        verbose (bool): If True, it outputs verbose messages on error.\n\n    \"\"\"\n    x = cuda.to_cpu(utils.force_array(x))\n    y = cuda.to_cpu(utils.force_array(y))\n    try:\n        numpy.testing.assert_allclose(\n            x, y, atol=atol, rtol=rtol, verbose=verbose)\n    except Exception:\n        print('error:', numpy.abs(x - y).max())\n        raise\n", "path": "chainer/gradient_check.py"}], "after_files": [{"content": "import numpy\nimport six\n\nfrom chainer import cuda\nfrom chainer import utils\n\n\ndef _copy_arrays(xs):\n    xp = cuda.get_array_module(*xs)\n    return tuple(xp.copy(x) for x in xs)\n\n\ndef numerical_grad_cpu(f, inputs, grad_outputs, eps=1e-3):\n    grads = tuple(numpy.zeros_like(x) for x in inputs)\n    for x, gx in zip(inputs, grads):\n        flat_x = x.ravel()\n        flat_gx = gx.ravel()\n        for i in six.moves.range(flat_x.size):\n            orig = flat_x[i]\n            flat_x[i] = orig + eps\n            ys1 = _copy_arrays(f())\n            flat_x[i] = orig - eps\n            ys2 = _copy_arrays(f())\n            flat_x[i] = orig\n\n            for y1, y2, gy in zip(ys1, ys2, grad_outputs):\n                if gy is not None:\n                    dot = float(sum(((y1 - y2) * gy).ravel()))\n                    flat_gx[i] += dot / (2 * eps)\n\n    return grads\n\n\ndef numerical_grad_gpu(f, inputs, grad_outputs, eps=1e-3):\n    grads = tuple(cuda.zeros_like(x) for x in inputs)\n    for x, gx in zip(inputs, grads):\n        x = x.ravel()\n        gx = gx.ravel()\n        x_cpu = x.get()\n        gx_cpu = gx.get()\n        for i in six.moves.range(x_cpu.size):\n            orig = x_cpu[i]\n            x_cpu[i] = orig + eps\n            x.set(x_cpu)\n            ys1 = _copy_arrays(f())\n            x_cpu[i] = orig - eps\n            x.set(x_cpu)\n            ys2 = _copy_arrays(f())\n            x_cpu[i] = orig\n            x.set(x_cpu)\n\n            for y1, y2, gy in zip(ys1, ys2, grad_outputs):\n                if gy is not None:\n                    dot = sum(((y1 - y2) * gy).ravel()).get()\n                    gx_cpu[i] += dot / (2 * eps)\n        gx.set(gx_cpu)\n\n    return grads\n\n\ndef numerical_grad(f, inputs, grad_outputs, eps=1e-3):\n    \"\"\"Computes numerical gradient by finite differences.\n\n    This function is used to implement gradient check. For usage example, see\n    unit tests of :mod:`chainer.functions`.\n\n    Args:\n        f (function): Python function with no arguments that runs forward\n            computation and returns the result.\n        inputs (tuple of arrays): Tuple of arrays that should be treated as\n            inputs. Each element of them is slightly modified to realize\n            numerical gradient by finite differences.\n        grad_outputs (tuple of arrays): Tuple of arrays that are treated as\n            output gradients.\n        eps (float): Epsilon value of finite differences.\n\n    Returns:\n        tuple: Numerical gradient arrays corresponding to ``inputs``.\n\n    \"\"\"\n    assert eps > 0\n    inputs = tuple(inputs)\n    grad_outputs = tuple(grad_outputs)\n    gpu = any(isinstance(x, cuda.ndarray) for x in inputs + grad_outputs)\n\n    cpu = any(isinstance(x, numpy.ndarray) for x in inputs + grad_outputs)\n\n    if gpu and cpu:\n        raise RuntimeError('Do not mix GPU and CPU arrays in `numerical_grad`')\n    elif gpu:\n        return numerical_grad_gpu(f, inputs, grad_outputs, eps)\n    else:\n        return numerical_grad_cpu(f, inputs, grad_outputs, eps)\n\n\ndef assert_allclose(x, y, atol=1e-5, rtol=1e-4, verbose=True):\n    \"\"\"Asserts if some corresponding element of x and y differs too much.\n\n    This function can handle both CPU and GPU arrays simultaneously.\n\n    Args:\n        x: Left-hand-side array.\n        y: Right-hand-side array.\n        atol (float): Absolute tolerance.\n        rtol (float): Relative tolerance.\n        verbose (bool): If True, it outputs verbose messages on error.\n\n    \"\"\"\n    x = cuda.to_cpu(utils.force_array(x))\n    y = cuda.to_cpu(utils.force_array(y))\n    try:\n        numpy.testing.assert_allclose(\n            x, y, atol=atol, rtol=rtol, verbose=verbose)\n    except Exception:\n        print('error:', numpy.abs(x - y).max())\n        raise\n", "path": "chainer/gradient_check.py"}]}