We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder. Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right. bi-directional language model to augment the sequence to sequence encoder 
 Question: What language model architectures are used?
uni-directional model to augment the decoder