text: be successfully approximated with adapter mod- 
  ules. They achieve competitive performance on 
  26 classiﬁcation tasks at a fraction of the computa- 
  tional cost. Adapters in BERT were also used for 
  multi-task learning (Stickland and Murray, 2019) 
  and cross-lingual transfer (Artetxe et al., 2019). An 
  alternative to ﬁne-tuning is extracting features from 
  frozen representations, but ﬁne-tuning works better 
  for BERT (Peters et al., 2019b). 
  
  Depending on the task, some BERT heads/layers 
  are not only redundant (Kao et al., 2020), but also 
  harmful to the downstream task performance. Pos- 
  itive effect from head disabling was reported for 
  machine translation (Michel et al., 2019), abstrac- 
  tive summarization (Baan et al., 2019), and GLUE 
  tasks (Kovaleva et al., 2019). Additionally, Ten- 
  ney et al. (2019a) examine the cumulative gains of 
  their structural probing classiﬁer, observing that in 
  5 out of 8 probing tasks some layers cause a drop 
  in scores (typically in the ﬁnal layers). Gordon 
  et al. (2020) ﬁnd that 30–40% of the weights can 
  be pruned without impact on downstream tasks. 