What is the weight matrix in self-attention? I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about &quot;weights&quot; in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.
From this article, http://peterbloem.nl/blog/transformers, in the additional tricks section, it mentions,
The query is the dot product of the query weight matrix and the word vector,
ie, q = W(q)x and the key is the dot product of the key weight matrix and the word vector, k = W(k)x and similarly for the value it is v = W(v)x. So my question is, where do the weight matrices come from?
The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes &quot;normal&quot; weights of a neural network.
This is also the reason why in the original paper the authors tested several setting with single and multiple attention heads. If these matrices were somehow &quot;special&quot; or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.