class_docstr: Scaled Dot-Product Attention module (Attention is all you need by Vaswani et al., 2017) with optional residual attention from previous layer
  (Realformer: Transformer likes residual attention by He et al, 2020) and locality self sttention (Vision Transformer for Small-Size Datasets
  by Lee et al, 2021)
class_docstr_tok: ["Scaled","Dot-Product","Attention","module","(","Attention","is","all","you","need","by","Vaswani","et","al.",",","2017",")","with","optional","residual","attention","from","previous","layer","(","Realformer",":","Transformer","likes","residual","attention","by","He","et","al",",","2020",")","and","locality","self","sttention","(","Vision","Transformer","for","Small-Size","Datasets","by","Lee","et","al",",","2021",")"]
code: def __init__(
      self,
      hidden_size,
      n_heads,
      attn_dropout=0.0,
      res_attention=False,
      lsa=False,
  ):
      super().__init__()
      self.attn_dropout = nn.Dropout(attn_dropout)
      self.res_attention = res_attention
      head_dim = hidden_size // n_heads
      self.scale = nn.Parameter(
          torch.tensor(head_dim**-0.5), requires_grad=lsa
      )
      self.lsa = lsa
  
imports: import math
  import numpy
  from typing import Optional
  import torch
  import torch.nn
  import torch.nn.functional
  from ..common._base_windows import BaseWindows
  from ..losses.pytorch import MAE
label_desc: Use image node_id 1 to create a new _ScaledDotProductAttention object from inherited base classes: nn with example: obj = _ScaledDotProductAttention(hidden_size, n_heads, attn_dropout, res_attention, lsa)