Decoder Transformer feedforward
There are some problems with your description:

During training, the decoder receives all the shifted target tokens, prepending the BOS token. You removed sole. The actual input would be: [&lt;bos&gt;, io, amo, il, sole]. Note that the output at the position of sole would be the end-of-sequence token &lt;eos&gt;.

During training, there is a single forward pass (not one per token), and all the output tokens are predicted at once. Therefore, only the last one of your attention masks is used.

During inference, we don't have the target tokens (because that is what we are trying to predict). In this case, we have one pass per generated token, starting with &lt;bos&gt;. This way, the decoder input in the first step would just be the sequence [&lt;bos&gt;], and we would predict the first token: io. Then, we would prepare the input for the next timestep as [&lt;bos&gt;, io], and then we would obtain the prediction for the second token. And so on. Note that, at each timestep, we are repeating the computations for the past positions; in real implementations, these states are cached instead of re-computed each timestep.


About some piece of Python code illustrating how the Transformer works, I suggest The annotated Transformer, which is a nice guide through a real implementation. You may be most interested in the function run_epoch for the training and in the function greedy_decode for the inference.