Transformers - is the self attention matrix softmax output (layer 1) symmetric?
I compared my results visually to a second implementation known to be working - &quot;The annotated transformer&quot;.  I compared the pytorch calculation results of the attention-method to my implementation results.
The answe is - the softmax is applied row by row. Therefore the resulting matrix p-attn is not equal to its transposed version.