How do LSTMs work if the following two matrices are not able to be multiplied? In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance:
The hidden state at timestep t-1 ($h_{t-1}$) has shape $(na, m)$
The input data at timestep t ($x_{t}$) has shape $(nx, m)$
$Z_{t}$ has shape $(na+nx, m)$ since the hidden state and input data are concatenated in LSTMs.
$W_{c}$ has shape $(na, na+nx)$
$W_{c}$ • $Z_{t}$ has shape $(na, m)$ = $i_{t}$
$W_{i}$ • $Z_{t}$ has shape $(na, m)$ = $ĉ_{t}$
When working through the network to the point $i_{t}$ and $ĉ_{t}$, how can these two be dot producted when the multiplication is not of the form (m x n)(n x p) as per the matrix multiplication definition?:
Turns out the reason is because, for places where a dot is shown in the image above, they're actually element-wise multiplications, not dot products. A lot of sources use an X or . to denote multiplication, but don't clearly indicate they mean element-wise multiplication.