text: 5.4 Fine-tuning BERT 
  
  Pre-training + ﬁne-tuning workﬂow is a crucial 
  part of BERT. The former is supposed to provide 
  task-independent knowledge, and the latter would 
  presumably teach the model to rely more on the 
  representations useful for the task at hand. 
  
  Kovaleva et al. (2019) did not ﬁnd that to be the 
  case for BERT ﬁne-tuned on GLUE tasks5: dur- 
  ing ﬁne-tuning, the most changes for 3 epochs oc- 
  curred in the last two layers of the models, but those 
  changes caused self-attention to focus on [SEP] 
  rather than on linguistically interpretable patterns. 
  It is understandable why ﬁne-tuning would increase 
  the attention to [CLS], but not [SEP]. If Clark 
  et al. (2019) are correct that [SEP] serves as "no- 
  op" indicator, ﬁne-tuning basically tells BERT what 
  to ignore. 
  
  Figure 5: Pre-trained weights help BERT ﬁnd wider 
  optima in ﬁne-tuning on MRPC (right) than training 
  from scratch (left) (Hao et al., 2019) 
  
  beddings as input for training BERT, while Po- 
  erner et al. (2019) adapt entity vectors to BERT 
  representations. As mentioned above, Wang et al. 
  (2020c) integrate knowledge not through entity em- 
  beddings, but through additional pre-training ob- 
  jective of knowledge base completion. Sun et al. 
  (2019b,c) modify the standard MLM task to mask 
  named entities rather than random words, and Yin 
  et al. (2020) train with MLM objective over both 
  text and linearized table data. Wang et al. (2020a) 
  enhance RoBERTa with both linguistic and factual 
  knowledge with task-speciﬁc adapters. 
  
  Several studies explored the possibilities of im- 
  
  proving the ﬁne-tuning of BERT: 