 
Figure 1: The Transformer model architecture based on “Attention is All You Need”. With the new PyTorch SDPA operator, Multi-Head Attention is efficiently implemented by a linear layer for the in-projection, the SDPA operator, and a linear layer for the out-projection.
With the new scaled_dot_product_attention operator, multihead attention can be implemented in just 3 steps: in projection with a linear layer, SDPA, and out projection with a linear layer.
```
In Projection
variable descriptions:
q,k,v = Query, Key, Value tensors
bsz = batch size
num_heads = Numner of heads for Multihead Attention
tgt_len = Target length
src_len = Source Length
head_dim: Head Dimension