Implementation Details. Pytorch was used as the deep learning framework to implement our models and APEX 1 for the distributed training. We used the RAdam optimizer with learning rate 0.001 and weight decay for 0.011. Gradient was accumulated for 10 batches. Dropout was set to 0.1 and the early stopping steps was 30. Detailed hyper-parameters for model training can be found supporting materials.