<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">After register a backward hook <code class="notranslate">myhook(module, grad_input, grad_output)</code> of <code class="notranslate">torch.nn.Linear</code>, and call <code class="notranslate">myhook</code>, it seems that the <code class="notranslate">grad_input</code> has an unexpected order/existence of gradient tensors of <code class="notranslate">(grad_of_linear_output, grad_of_linear_input, grad_of_linear_weight)</code>, which are respectively gradient tensors wrt outputs of <code class="notranslate">Linear</code>, inputs of <code class="notranslate">Linear</code>, and weights of <code class="notranslate">Linear</code>. This is unexpected and biases' gradient is missing. It is also inconsistent with <code class="notranslate">Conv2d</code>, <code class="notranslate">BatchNorm2d</code> and <code class="notranslate">ReLU</code>, whose <code class="notranslate">grad_input</code> return something like <code class="notranslate">(grad_of_input, grad_of_parameter1, grad_of_parameter2)</code>.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:<br>
Code fragments</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# define a hook
    def myhook(m, grad_input, grad_output, name=None):
            print(name, '======================')
            print('--')
            for grad in grad_input:
                if grad is not None:
                    print(grad.size())
                else:
                    print('None')
            print('--')
            for grad in grad_output:
                if grad is not None:
                    print(grad.size())
                else:
                    print('None')"><pre class="notranslate"><code class="notranslate"># define a hook
    def myhook(m, grad_input, grad_output, name=None):
            print(name, '======================')
            print('--')
            for grad in grad_input:
                if grad is not None:
                    print(grad.size())
                else:
                    print('None')
            print('--')
            for grad in grad_output:
                if grad is not None:
                    print(grad.size())
                else:
                    print('None')
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# hook up
        for idx, m in enumerate(model.named_modules()):
            if isinstance(m[1], torch.nn.Linear):
                logger.info('\t{} registering backward hook...'.format(m[0]))
                h = m[1].register_backward_hook(hook=partial(myhook, name=m[0]))"><pre class="notranslate"><code class="notranslate"># hook up
        for idx, m in enumerate(model.named_modules()):
            if isinstance(m[1], torch.nn.Linear):
                logger.info('\t{} registering backward hook...'.format(m[0]))
                h = m[1].register_backward_hook(hook=partial(myhook, name=m[0]))
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# build a model with
linear = nn.Linear(64, 10)"><pre class="notranslate"><code class="notranslate"># build a model with
linear = nn.Linear(64, 10)
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# run with batch size 128 and get
linear ======================
--
(128, 10)
(128, 64)
(64, 10)
--
(128, 10)"><pre class="notranslate"><code class="notranslate"># run with batch size 128 and get
linear ======================
--
(128, 10)
(128, 64)
(64, 10)
--
(128, 10)
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Based on the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_backward_hook" rel="nofollow">doc</a> and consistency with <code class="notranslate">Conv2d</code>, the shapes should be</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="linear ======================
--
(128, 64)
(64, 10)
(10,)
--
(128, 10)"><pre class="notranslate"><code class="notranslate">linear ======================
--
(128, 64)
(64, 10)
(10,)
--
(128, 10)
</code></pre></div>
<p dir="auto">The shapes are normal for <code class="notranslate">Conv2d</code>, <code class="notranslate">BatchNorm2d</code> and <code class="notranslate">ReLU</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="layer1.0.relu ======================
--
(128, 16, 32, 32)
--
(128, 16, 32, 32)
layer1.0.bn1 ======================
--
(128, 16, 32, 32)
(16,)
(16,)
--
(128, 16, 32, 32)
layer2.0.conv1 ======================
--
(128, 16, 32, 32)
(32, 16, 3, 3)
None # no biases in the tested conv
--
(128, 32, 16, 16) # stride is 2"><pre class="notranslate"><code class="notranslate">layer1.0.relu ======================
--
(128, 16, 32, 32)
--
(128, 16, 32, 32)
layer1.0.bn1 ======================
--
(128, 16, 32, 32)
(16,)
(16,)
--
(128, 16, 32, 32)
layer2.0.conv1 ======================
--
(128, 16, 32, 32)
(32, 16, 3, 3)
None # no biases in the tested conv
--
(128, 32, 16, 16) # stride is 2
</code></pre></div>
<p dir="auto">For consistency, we should correct this or highlight it in the doc if compatibility is a concern.</p>
<h2 dir="auto">Environment</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 0.4.1.post2
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 2.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: GeForce GTX 1080
GPU 1: GeForce GTX 1080
GPU 2: GeForce GTX 1080
GPU 3: GeForce GTX 1080

Nvidia driver version: 396.37
cuDNN version: Probably one of the following:
/usr/local/cuda-9.2/lib64/libcudnn.so
/usr/local/cuda-9.2/lib64/libcudnn.so.7
/usr/local/cuda-9.2/lib64/libcudnn.so.7.1.1
/usr/local/cuda-9.2/lib64/libcudnn.so.7.2.1
/usr/local/cuda-9.2/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] Could not collect
[conda] blas                      1.0                         mkl
[conda] mkl                       2018.0.2                      1
[conda] mkl-service               1.1.2            py27hb2d42c5_4
[conda] mkl_fft                   1.0.1            py27h3010b51_0
[conda] mkl_random                1.0.1            py27h629b387_0
[conda] pytorch                   0.4.1           py27__9.0.176_7.1.2_2    pytorch
[conda] torchvision               0.2.1                    py27_1    pytorch"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 0.4.1.post2
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 2.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: GeForce GTX 1080
GPU 1: GeForce GTX 1080
GPU 2: GeForce GTX 1080
GPU 3: GeForce GTX 1080

Nvidia driver version: 396.37
cuDNN version: Probably one of the following:
/usr/local/cuda-9.2/lib64/libcudnn.so
/usr/local/cuda-9.2/lib64/libcudnn.so.7
/usr/local/cuda-9.2/lib64/libcudnn.so.7.1.1
/usr/local/cuda-9.2/lib64/libcudnn.so.7.2.1
/usr/local/cuda-9.2/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] Could not collect
[conda] blas                      1.0                         mkl
[conda] mkl                       2018.0.2                      1
[conda] mkl-service               1.1.2            py27hb2d42c5_4
[conda] mkl_fft                   1.0.1            py27h3010b51_0
[conda] mkl_random                1.0.1            py27h629b387_0
[conda] pytorch                   0.4.1           py27__9.0.176_7.1.2_2    pytorch
[conda] torchvision               0.2.1                    py27_1    pytorch
</code></pre></div>
<p dir="auto">Maxim Naumov <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mnaumovfb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mnaumovfb">@mnaumovfb</a> sends in the following feedback:</p>
<hr>
<p dir="auto">I thought I would share some feedback from my recent work with register_backward_hook function. I will describe some some inconsistencies in it that may be worth addressing in the future.</p>
<h2 dir="auto">Background</h2>
<p dir="auto">PyTorch supports different hook functions, including register_hook, register_forward_hook and register_backward_hook. The former is applied to a tensor variable, while the latter two are applied to a layer module. I will discuss the latest form here. It has the following signature <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow">https://pytorch.org/docs/stable/nn.html</a> <code class="notranslate">func(layer, grad_input, grad_output)</code></p>
<h2 dir="auto">Details</h2>
<p dir="auto">It is extremely important to understand the meaning of gradients to register_backwards_hook. Let a layer be defined as</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" output z  (grad_output)
     ____|____
    |__layer__|
         |
 input x  (grad_input)"><pre class="notranslate"><code class="notranslate"> output z  (grad_output)
     ____|____
    |__layer__|
         |
 input x  (grad_input)
</code></pre></div>
<p dir="auto">with overall loss E, error gradient dE/dz^(k) and weight gradient dE/dw^(k).<br>
First, let us assume the simplest case: a layer with no bias. Then,</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="grad_output= [dE/dz^(k)]
grad_input = [dE/dz^(k-1), dE/dw^(k)]"><pre class="notranslate"><code class="notranslate">grad_output= [dE/dz^(k)]
grad_input = [dE/dz^(k-1), dE/dw^(k)]
</code></pre></div>
<h2 dir="auto">Inconsistencies</h2>
<p dir="auto">It seems that there are some inconsistencies in how gradients for different layers are handled by this function.</p>
<ol dir="auto">
<li>Shape
<ul dir="auto">
<li>in convolution layers the weight gradient has the same shape as the weights</li>
<li>in fully connected layers the weight gradient is transpose of the weights</li>
</ul>
</li>
<li>Bias
<ul dir="auto">
<li>in convolution layers bias gradient are appended: grad_input = [dE/dz^(k-1), dE/dw^(k), dE/db^(k)]</li>
<li>in fully connected layers bias gradient are prepended: grad_input = [dE/db^(k), dE/dz^(k-1), dE/dw^(k)]</li>
</ul>
</li>
<li>Batch size &gt; 1
<ul dir="auto">
<li>in convolution layers bias gradient corresponds to the gradient over the entire batch: grad_input = [dE/dz^(k-1), dE/dw^(k), dE/db^(k)]</li>
<li>in fully connected layers bias gradient corresponds to the gradient per data point j=1,...,r in the batch (therefore it needs to be added to get the gradient over the entire batch): grad_input = [[dE/db^(k,1),...,dE/db^(k,r)], dE/dz^(k-1), dE/dw^(k)]</li>
</ul>
</li>
</ol>
<p dir="auto">These discrepancies can make handling of different layers, bias and batch sizes quite cumbersome in the code. It would help if they were done more consistently in the future.</p>