How to attach hooks to ReLUs in Inception V3 from torchvision
I am using Inception v3 from torchvision. I tried to find the ReLUs within the model:

def recursively_find_submodules(model, submodule_type):
    module_list = []
    q = [model]    
    while q:
        child = q.pop()
        if isinstance(child, submodule_type):
            module_list.append(child)
        q.extend(list(child.children()))                 
    return module_list

inception = torch.hub.load('pytorch/vision:v0.6.0', 'inception_v3', pretrained=True)
l = recursively_find_submodules(inception, torch.nn.ReLU) # l is empty!


So the ReLUs are not children of any module within the torch model. Upon closer inspection I found the ReLUs in the source code of torchvision but not as modules. In inception.py I found the following:

class BasicConv2d(nn.Module):

    def __init__(self, in_channels, out_channels, **kwargs):
        super(BasicConv2d, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs)
        self.bn = nn.BatchNorm2d(out_channels, eps=0.001)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        return F.relu(x, inplace=True)


So the BasicConv2d module uses the ReLU function to clamp it's output instead of the module (torch.nn.ReLU). I guess there is no way to hook up to ReLU functions and modify their input / output without modifying the whole model to use ReLU modules or is there a way to do this?

You can hook to the batch-norm layer preceding the ReLU and attach there, taking into account you observe the inputs to the ReLU rather that the features after the activation. 
