cosine similarity for attention decoder in nmt
I am implementing a neural machine translation model, and for the decoder part (with attention mechanism) I would like to calculate the cosine similarity for finding the scores.
Here is the function:

score(a,b) =  / ||a|| ||b|| 

In my case: 

        a = htilde_t (N, H)

        b = h (S, N, H)

        the output should be (S, N)


My confusion is about their dimensions and I don't know how to solve that in pytorch. 

See here: https://pytorch.org/docs/master/nn.html?highlight=cosine#torch.nn.CosineSimilarity

cos = nn.CosineSimilarity(dim=2, eps=1e-6)
output = cos(a.unsqueeze(0),b)


you need to unsqueeze to add a ghost dimension to have both input of same dim:

    Input1: (∗1,D,∗2) where D is at position dim

    Input2: (∗1,D,∗2) , same shape as the Input1

    Output: (∗1,∗2)

