```
PyTorch 2. supports multiple different kernels optimized for specific use cases, with specific requirements. A kernel picker picks the best kernel for a particular combination of input parameters. If no optimized "custom kernel" for a particular combination of input parameters can be identified, the kernel picker selects a general kernel that can handle all input combinations. 
While future releases may extend this set of operators, PyTorch 2.0 launches with 3 implementations for the SDPA operator:

A generic kernel which implements the mathematical equation of SDPA in the function sdpa_math()
An optimized kernel based on the paper “Flash Attention”, which supports evaluation of SDPA with 16 bit floating point data types on compute architecture SM80 (A100).
