RoBERTa, or BERT only use the encoding half of the transformer, as the decoding half is not required to translate the input sequence into another language. In Figure , the "BERT-RoBERTa" unit includes the input embedding unit, positional encoding unit, multi-head attention units, and normalisation layer, followed by layers of feed-forward neural network.