What is the weight matrix in self-attention?
The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes &quot;normal&quot; weights of a neural network.
This is also the reason why in the original paper the authors tested several setting with single and multiple attention heads. If these matrices were somehow &quot;special&quot; or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.