In our final system, after pre-training the forward and backward LMs separately, we remove the top layer softmax and concatenate the forward and backward LM embeddings to form bidirectional LM embeddings, i.e., INLINEFORM0 .  
 Question: how are the bidirectional lms obtained?
They pre-train forward and backward LMs separately, remove top layer softmax, and concatenate to obtain the bidirectional LMs.