
Freeze Training: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. NeurIPS 2017
Efficient Training of BERT by Progressively Stacking. ICML 2019
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping. NeurIPS 2020.
On the Transformer Growth for Progressive BERT Training. NACCL 2021





Figure 2. Interpretable Freeze Training: DNNs converge bottom-up (Results on CIFAR10 using ResNet). Each pane shows layer-by-layer similarity using SVCCA [17][18]