Pytorch Luong global attention: what is the shape of the alignment vector supposed to be? I am looking at the Luong paper on Attention models and global attention. I understand how the alignment vector is computed from a dot product of the encoder hidden state and the decoder hidden state. So that all makes sense.
My question is, what should the dimensions of the alignment score tensor be? If I have my data in batches, I basically compute a single score for each timestep in the hidden state, right. So should the alignment vector be of dimension [sequence length, 1], or something like that? I would then softmax this alignment vector and multiply it by every element in the batch to compute the context vector, right.
Again, my key question is what the dimensions of the alignment score vector or tensor should be. Thanks.
You answered yourself [sequence length, 1] is correct assuming you work with a single sentence. (Or actually, the 1 dimension depends on implementation.)
In practice, the data is typically batched, so it will be [batch, sequence length 1]. This can be element-wise multiplied with the encoder states of dimension [batch, sequence length, hidden size] and summed in the middle dimension to get a context vector of dimension [batch, hidden_size].
Note also that if the sentences do not have the same length, you need to deal with padding, i.e., set the padded positions to -inf before softmax.