Weighted summation of embeddings in pytorch
I have a sequence of 12 words which I represent using a 12x256 matrix (using word embeddings). Let us refer to these as . I wish to take this as input and output a 1x256 vector. However I don't want to use a (12x256) x 256 dense layer. Instead I want to create the output embedding using a weighted summation of the 12 embeddings



where the wi s are scalars (thus there is weight sharing).

How can I create trainable wi s in pytorch? I am new and only familiar with the standard modules like nn.Linear.

You can implement this via 1D convolution with kernel_size = 1

import torch

batch_size=2

inputs = torch.randn(batch_size, 12, 256)
aggregation_layer = torch.nn.Conv1d(in_channels=12, out_channels=1, kernel_size=1)
weighted_sum = aggregation_layer(inputs)


Such convolution will have 12 parameters. Each parameter will be a equal to e_i in formula you provided. 

In other words this convolution will ran over dimetion with size 256 and sum it with learnable weights.
