of BERT ﬁne-tuning even for a small number
of training examples (Lee et al., 2019).
With large models, even ﬁne-tuning becomes ex-
pensive, but Houlsby et al. (2019) show that it can
5Kondratyuk and Straka (2019) suggest that ﬁne-tuning on
Universal Dependencies does result in syntactically meaning-
ful attention patterns, but there was no quantitative evaluation.