<p dir="auto">Maxim Naumov <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mnaumovfb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mnaumovfb">@mnaumovfb</a> sends in the following feedback:</p>
<hr>
<p dir="auto">I thought I would share some feedback from my recent work with register_backward_hook function. I will describe some some inconsistencies in it that may be worth addressing in the future.</p>
<h2 dir="auto">Background</h2>
<p dir="auto">PyTorch supports different hook functions, including register_hook, register_forward_hook and register_backward_hook. The former is applied to a tensor variable, while the latter two are applied to a layer module. I will discuss the latest form here. It has the following signature <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow">https://pytorch.org/docs/stable/nn.html</a> <code class="notranslate">func(layer, grad_input, grad_output)</code></p>
<h2 dir="auto">Details</h2>
<p dir="auto">It is extremely important to understand the meaning of gradients to register_backwards_hook. Let a layer be defined as</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" output z  (grad_output)
     ____|____
    |__layer__|
         |
 input x  (grad_input)"><pre class="notranslate"><code class="notranslate"> output z  (grad_output)
     ____|____
    |__layer__|
         |
 input x  (grad_input)
</code></pre></div>
<p dir="auto">with overall loss E, error gradient dE/dz^(k) and weight gradient dE/dw^(k).<br>
First, let us assume the simplest case: a layer with no bias. Then,</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="grad_output= [dE/dz^(k)]
grad_input = [dE/dz^(k-1), dE/dw^(k)]"><pre class="notranslate"><code class="notranslate">grad_output= [dE/dz^(k)]
grad_input = [dE/dz^(k-1), dE/dw^(k)]
</code></pre></div>
<h2 dir="auto">Inconsistencies</h2>
<p dir="auto">It seems that there are some inconsistencies in how gradients for different layers are handled by this function.</p>
<ol dir="auto">
<li>Shape
<ul dir="auto">
<li>in convolution layers the weight gradient has the same shape as the weights</li>
<li>in fully connected layers the weight gradient is transpose of the weights</li>
</ul>
</li>
<li>Bias
<ul dir="auto">
<li>in convolution layers bias gradient are appended: grad_input = [dE/dz^(k-1), dE/dw^(k), dE/db^(k)]</li>
<li>in fully connected layers bias gradient are prepended: grad_input = [dE/db^(k), dE/dz^(k-1), dE/dw^(k)]</li>
</ul>
</li>
<li>Batch size &gt; 1
<ul dir="auto">
<li>in convolution layers bias gradient corresponds to the gradient over the entire batch: grad_input = [dE/dz^(k-1), dE/dw^(k), dE/db^(k)]</li>
<li>in fully connected layers bias gradient corresponds to the gradient per data point j=1,...,r in the batch (therefore it needs to be added to get the gradient over the entire batch): grad_input = [[dE/db^(k,1),...,dE/db^(k,r)], dE/dz^(k-1), dE/dw^(k)]</li>
</ul>
</li>
</ol>
<p dir="auto">These discrepancies can make handling of different layers, bias and batch sizes quite cumbersome in the code. It would help if they were done more consistently in the future.</p>
<p dir="auto">Hi,</p>
<p dir="auto">there is something strange in the <code class="notranslate">backward</code> step (or maybe something I don't understand). If I define a Module that takes 3 inputs, the <code class="notranslate">grad_input</code> has to be of size 3, right ? But this is not the case here (from the backward_hook point of view):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

def bh(m,go,gi):
    print(&quot;Grad Input&quot;)
    print(go)
    print(&quot;Grad Output&quot;)
    print(gi)

class M(nn.Module):
    def __init__(self):
        super(M,self).__init__()
        self.register_backward_hook(bh)


    def forward(self,x,y,z):
        return (x+y+z)

x=Variable(torch.randn(1,5),requires_grad=True)
y=Variable(torch.randn(1,5),requires_grad=True)
z=Variable(torch.randn(1,5),requires_grad=True)

criterion=nn.MSELoss()
mod=M()
out=mod(x,y,z)
loss=criterion(out,Variable(torch.randn(1,5)))
loss.backward()```

In that case, when I print grad_input throught the hook function, it is just composed of two elements... Could you tell me where am I wrong ? But `x.grad, y.grad and z.grad` seem correctly computed

cc @ezyang @gchanan @zou3519 @SsnL @albanD @gqchen"><pre lang="import" class="notranslate"><code class="notranslate">from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

def bh(m,go,gi):
    print("Grad Input")
    print(go)
    print("Grad Output")
    print(gi)

class M(nn.Module):
    def __init__(self):
        super(M,self).__init__()
        self.register_backward_hook(bh)


    def forward(self,x,y,z):
        return (x+y+z)

x=Variable(torch.randn(1,5),requires_grad=True)
y=Variable(torch.randn(1,5),requires_grad=True)
z=Variable(torch.randn(1,5),requires_grad=True)

criterion=nn.MSELoss()
mod=M()
out=mod(x,y,z)
loss=criterion(out,Variable(torch.randn(1,5)))
loss.backward()```

In that case, when I print grad_input throught the hook function, it is just composed of two elements... Could you tell me where am I wrong ? But `x.grad, y.grad and z.grad` seem correctly computed

cc @ezyang @gchanan @zou3519 @SsnL @albanD @gqchen
</code></pre></div>