We use the additional INLINEFORM0 training articles labeled by publisher as an unsupervised data set to further train the BERT model. We first investigate the impact of pre-training on BERT-BASE's performance.  On the same computer, fine tuning the model on the small training set took only about 35 minutes for sequence length 100.  
 Question: How are the two different models trained?
They pre-train the models using 600000 articles as an unsupervised dataset and then fine-tune the models on small training set.