Other transformer models (such as decoder models) which use the PyTorch MultiheadAttention module will benefit from the BetterTransformer fastpath. Planned future work is to expand the end-to-end BetterTransformer fastpath to models based on TransformerDecoder to support popular seq2seq and decoder-only (e.g., OPT) model architectures, and to training.
Speedups
The following graphs show the performance achieved for the BERT-base model with small and large-scale inputs:




Figure 1: PyTorch 1.12 Improvements with BetterTransformer fastpath execution

