One model, X-MOL , uses a Transformer encoder with a combined fully-visible and autoregressive attention mask. This allows the model to be applied to both discriminative and sequenceto-sequence tasks. However, this is very resource intensive for the latter since the amount of memory and computation required grows quadratically with the length of the sequence . Additionally, X-MOL does not approach pre-training from a language-modelling perspective and it explores only a single pre-training task.