Is a dense layer required for implementing Bahdanau attention?
A Dense layer inside the Attention layer logic is a must. This is typically a single unit, single layer MLP (though in case of Additive Attention, there is a provision to use more units). This layer helps us derive the attention weights during training
This Dense layer transforms the input (each of the encoder LSTM layer output) keeping the dimensions the same. This is then multiplied individually with the last decoder hidden state &amp; softmaxed to generate the attention weights.
Multiplying the attention weights with the encoder hidden states help to magnify or diminish the relevant input information and this context can then be used for Predictions. (In Additive layer addition is used instead of multiplication)
Without the Dense layer in the Attention layer, we don't get the Attention weights. Without Attention weights, there is no context. Without context there is no Attention.
So in order to incorporate Attention, a single layer Dense MLP is mandatory. The model learns the weights of this layer during training. These layer weights are then used to derive the attention weights.