Transkimmer: Transformer Learns to Layer-wise Skim
Yong Liang Guan|Zhengyi Li|Jingwen Leng|Zhouhan Lin|Minyi Guo
Transformer architecture has become the de-facto model for many machine learning tasks from natural language processing and computer vision. As such, improving its computational efficiency becomes paramount. One of major inefficiency based models is that they spend identical amount computation throughout all layers. Prior works have proposed to augment with capability skimming tokens improve efficiency. However, suffer not having effectual end-to-end optimization discrete predictor. To address above limitations, we propose Transkimmer architecture, which learns identify hidden state are required by each layer. The skimmed then forwarded directly final output, thus reducing successive key idea in add a parameterized predictor before layer make decision. We also adopt reparameterization trick skim loss training Transkimmer. achieves 10.97x average speedup on GLUE benchmark compared vanilla BERT-base baseline less than 1% accuracy degradation.
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)|arXiv (Cornell University)