2020) and dynamic elimination of intermediate en-
coder outputs (Goyal et al., 2020). See Ganesh et al.
(2020) for a more detailed discussion of compres-
sion methods.
6.3 Pruning and model analysis
There is a nascent discussion around pruning as a
model analysis technique. The basic idea is that
a compressed model a priori consists of elements
that are useful for prediction; therefore by ﬁnding
out what they do we may ﬁnd out what the whole
network does. For instance, BERT has heads that
seem to encode frame-semantic relations, but dis-
abling them might not hurt downstream task per-
formance Kovaleva et al. (2019); this suggests that
this knowledge is not actually used.
For the base Transformer, V oita et al. (2019b)
identify the functions of self-attention heads and