
layout: blog_detail
title: "Accelerated PyTorch 2 Transformers"
author: Michael Gschwind, Driss Guessous, Christian Puhrsch

The PyTorch 2.0 release includes a new high-performance implementation of the PyTorch Transformer API with the goal of making training and deployment of state-of-the-art Transformer models affordable.  Following the successful release of “fastpath” inference execution (“Better Transformer”), this release introduces high-performance support for training and inference using a custom kernel architecture for scaled dot product attention (SPDA).