Does the transformer decoder reuse previous tokens' intermediate states like GPT2? I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point :
He explains that the decoder of GPT-2 processes input tokens one at a time, only actively processing the last input token, the past tokens being saved in memory already and "passively" reused without reevaluation.

From my understanding of the transformer architecture, I had the impression that the decoder reevaluates every token generated at each generation. Is this then a difference between the decoder from GPT-2 or does the decoder from a "classical' transformer also work this way ?

Intuitively I would think that it would make more sense the reevaluate everything at each iteration since new dependencies between words can appear that weren't there at the beginning and would then not be taken into account if past processed words are passively reused.

I hope I am making sense, can someone with knowledge about the GPT2 architecture help me clarify this ?
The past token internal states are reused both in GPT-2 and any other Transformer decoder.

For example, in fairseq's implementation of the transformer, these previous states are received in TransformerDecoder.forward in parameter incremental_state(see the source code).

Remember that there is a mask in the self-attention blocks in the decoder that prevents the predictions and intermediate states to attend to positions equal to or greater than the current one, which means that the internal state won't change even if you recomputed them at every decoding step.

Update: of course it is technically possible to recompute past tokens attending future tokens, but then, what do you do with the future tokens after you re-compute the past ones? Do you recompute them? This is a totally different beast, which has been studied to some degree and is referred to as "iterative refinement". An example can be found in article "Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement". AFAIK, this kind of approach has not studied in autoregressive models, only in non-autoregressive ones.