How Transformer is Bidirectional - Machine Learning
It is the encoder part of the Transformer model that is bidirectional in nature, not the whole model.
The full Transformer model has two parts: encoder and decoder. This encoder-decoder model is used for sequence-to-sequence tasks, like machine translation.
There are other tasks, however, that do not need the full model, but only one of its parts. For instance, for causal language modeling (e.g. GPT-2) we need the decoder. For masked language modeling (e.g. BERT), we need the encoder.
The decoder is designed so that each predicted token can only depend on the previous tokens. This is achieved with self-attention masking, and it is what makes the decoder unidirectional.
The encoder does not have self-attention masking. Therefore is designed not to have any dependency limitation: the token representation obtained at one position depends on all the tokens in the input. This is what makes the Transformer encoder bidirectional.