The invalid connections are formed by the training process, and the triangle mask is shaped to mask them out. To understand why one may be needed, let's take a look at how and why triangle masks are used.

Let's assume the attention model above is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens.

So in order to predict next token, you have to make sure that only previous tokens are attended. (If not, this would be somewhat like cheating while training because model already knows what's next).

So an attention mask would be like this
```block
[0, 1, 1, 1, 1]
[0, 0, 1, 1, 1]
[0, 0, 0, 1, 1]
[0, 0, 0, 0, 1]
[0, 0, 0, 0, 0]
```

For example: If you are translating English to Spanish
Input: How are you ?
Target: ```< start > Como estas ? < end >```
Then the decoder will predict something like this (start will be given to decoder as initial token)
```block
< start > 
< start > Como
< start > Como estas
< start > Como estas ?
< start > Como estas ? < end >
```

Now compare this step by step prediction sequences to attention mask matrix given above, see how the ones correspond to valid connections we want to pay attention to as the sequence is built, and the zeros correspond to the triangle we want to mask out, the invalid connections we want to remove from attention.  So, yes, in this sense, invalid connections are not only common, but inherent.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence