Matrix multiplication in recurrent neural networks I was looking at a tutorial for recurrent neural networks in Python, and I have a question in regards to multiplying matrices of different sizes. Specifically, why does S[t] have 100 elements in it?


  s[t] = np.tanh(self.U[:,x[t]] + self.W.dot(s[t-1]))


Earlier in the tutorial, the author lists the dimensions for each variable:


  \begin{aligned} x_t &amp; \in \mathbb{R}^{8000} \\ o_t &amp; \in \mathbb{R}^{8000} \\ s_t &amp; \in \mathbb{R}^{100} \\ U &amp; \in \mathbb{R}^{100 \times 8000} \\ V &amp; \in \mathbb{R}^{8000 \times 100} \\ W &amp; \in \mathbb{R}^{100 \times 100} \\ \end{aligned} 


From how I understand the above line of code, it multiplies U by x[t] and adds it to the product of W and s[t-1], then computes tanh for each element.

Sources like this say that you cannot add matrices of different dimensions, however that seems to be what is happening here (because multiplication is just repeated addition). In fact, it seems that U is 2D and x[t] is 1D. How are these added? Also, how is the sum of the two products then 100 elements?
Matrix-by-matrix multiplication is very different from scalar-by-scalar. It has no connection to repeated addition, and in fact isn't even commutative: it's entirely possible that $AB \neq BA$. It's only called multiplication because it has some similar properties to repeated addition of scalars.

Matrix multiplication requires that the inner dimensions match. Nothing more, nothing less. You can multiply an (a×b) matrix by a (b×c) matrix, for instance, and the result is an (a×c) matrix. For the full details, Wikipedia has a good summary.

In addition, that code isn't adding or multiplying $U$ and $x[t]$. Rather, it's removing one dimension from $U$, taking only the columns (or rows depending on your definitions) which correspond to 1s in $x[t]$.