How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer?
First to equation (7): $s_{i-1}$ is a vector, not a matrix. When you multiply it with a matrix $W$, you get another vector of which as the length of the intermediate attention projection, let us call it $d_a$. The shape of $W$ is thus $1024 \times d_a$. Similarly, the shape of $V$ is $512 \times d_a$ and bias $b$ is a vector of length $d_a$. The vector $w$ has $d_a$ dimensions and projects the intermediate projection into a scalar number.

You would probably like to compute everything with large matrix multiplications, which is possible only at training time, and which might the reason for your confusion about the dimensions.


If you have all decoder states in a matrix $S$ of shape $n \times 1024$. You can project it using $W$ into the shape $n \times d_a$.
You can also project all encoder states $H$ of shape $m \times 512$  using $V$ and get a matrix of shape $m \times d_a$ for input and output lengths $n$ and $m$.*
Now, you need to get a tensor of size $n \times m \times d_a$ for the values of $e_{i,j}$.


There is a simple way to implement it, you just reshape the first matrix to $n \times 1 \times d_a$, the second one to $1 \times m \times d_a$ and sum them together. The deep learning/linear algebra libraries will broadcast the tensor (i.e., copy along the singleton dimension), so it will give you a tensor of shape $n \times m \times d_a$. When you multiply this with $w$, you get a table of attention energies of shape $n \times m$, after applying softmax on the last dimension, you get $\alpha_{i,j}$ There is no unknown dimension anywhere.

Equation (8) is just a fancy formulation of the weighted average. In $F$, you have $k$ vectors of dimension $r$, the distribution $\alpha_i$ has length $k$:

$$f_i = \sum_{j=0}^k f_{i,j} \cdot \alpha_{i-1,j}$$

Note that $\alpha_{i-1,j}$ is scalar. This is the context vector form the previous decoding step, i.e., a weighted average of $h_j$ based on the probability distribution  $a_{i-1,j}$.

Equation (9) is basically the same as (7).