What is the purpose of Decoder mask (triangular mask) in Transformer?
The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens.
So in order to predict next token, you have to make sure that only previous token are attended. (If not, this would be a cheating because model already knows whats next).
So attention mask would be like this
[0, 1, 1, 1, 1]
[0, 0, 1, 1, 1]
[0, 0, 0, 1, 1]
[0, 0, 0, 0, 1]
[0, 0, 0, 0, 0]
For example: If you are translating English to Spanish
Input: How are you ?
Target: &lt; start &gt; Como estas ? &lt; end &gt;
Then decoder will predict something like this
&lt; start &gt; (it will be given to decoder as initial token)
&lt; start &gt; Como
&lt; start &gt; Como estas
&lt; start &gt; Como estas ?
&lt; start &gt; Como estas ? &lt; end &gt;
Now compare this step by step prediction sequences to attention mask given above, It would make sense now to you