CNN + RNN architecture for video recognition
I am trying to replicate the ConvNet + LSTM approach presented in this paper using pytorch. But I am struggling to find the correct way to combine the CNN and the LSTM in my model. Here is my attempt :
class VideoRNN(nn.Module):
  def __init__(self, hidden_size, n_classes):
    super(VideoRNN, self).__init__()

    self.hidden_size = hidden_size

    vgg = models.vgg16(pretrained=True)
    embed = nn.Sequential(*list(vgg.classifier.children())[:-1])
    vgg.classifier = embed
    
    for param in vgg.parameters():
      param.requires_grad = False


    self.embedding = vgg
    self.GRU = nn.GRU(4096, hidden_size)


  def forward(self, input, hidden=None):
    embedded = self.embedding(input)
    output, hidden = self.gru(output, hidden)
    output = self.classifier(output.view(-1, 4096))

    return output, hidden

As my videos have variable length, I provide a PackedSequence as an input. It is created from a Tensor with shape (M,B,C,H,W) where M is the maximum sequence length and B the batch size. The C,H,W are the channels, height and width of each frame.
I want the pre-trained CNN to be part of the model as I may later unfreeze some layer to finetune the CNN for my task. That's why I didn't compute the embedding of the images separately.
My questions are then the following :

Is the shape of my input data correct in order to handle batches of videos in my context or should I use something else than a PackedSequence?

In my forward function, how can I handle the batch of sequences of images with my VGG and my GRU unit ? I cannot feed directly the PackedSequence as an input to my VGG so how can I proceed?

Does this approach seem to respect the &quot;pytorch way of doing things&quot; or should is my approach flawed?



I finally found the solution to make it works. Here is a simplified yet complete example of how I managed to create a VideoRNN able to use packedSequence as an input :
    class VideoRNN(nn.Module):
    def __init__(self, n_classes, batch_size, device):
        super(VideoRNN, self).__init__()

        self.batch = batch_size
        self.device = device

        # Loading a VGG16
        vgg = models.vgg16(pretrained=True)

        # Removing last layer of vgg 16
        embed = nn.Sequential(*list(vgg.classifier.children())[:-1])
        vgg.classifier = embed

        # Freezing the model 3 last layers
        for param in vgg.parameters():
            param.requires_grad = False

        self.embedding = vgg
        self.gru = nn.LSTM(4096, 2048, bidirectional=True)

        # Classification layer (*2 because bidirectionnal)
        self.classifier = nn.Sequential(
            nn.Linear(2048 * 2, 256),
            nn.ReLU(),
            nn.Linear(256, n_classes),
        )

    def forward(self, input):
        hidden = torch.zeros(2, self.batch , 2048).to(
            self.device
        )

        c_0 = torch.zeros(self.num_layer * 2, self.batch, 2048).to(
            self.device
        )

        embedded = self.simple_elementwise_apply(self.embedding, input)
        output, hidden = self.gru(embedded, (hidden, c_0))
        hidden = hidden[0].view(-1, 2048 * 2)

        output = self.classifier(hidden)

        return output

    def simple_elementwise_apply(self, fn, packed_sequence):
        return torch.nn.utils.rnn.PackedSequence(
            fn(packed_sequence.data), packed_sequence.batch_sizes
        )

the key is the simple_elementwise_apply methods allowing to feed the PackedSequence in the CNN networks and to retrieve a new PackedSequence made of embedding as an output.
I hope you'll find it useful.
