class_docstr: Additive Attention with linear complexity to input sequence length.
  
  Additive attention was proposed and used in FastFormer.
  See Ref. [1] for details.
  This implementation is motivated by: https://github.com/jrzaurin/pytorch-widedeep.git
  
  References:
  ----------
  [1] Wu, Chuhan, et al. "Fastformer: Additive attention can be all you need." arXiv preprint arXiv:2108.09084 (2021).
class_docstr_tok: ["Additive","Attention","with","linear","complexity","to","input","sequence","length",".","Additive","attention","was","proposed","and","used","in","FastFormer",".","See","Ref",".","[","1","]","for","details",".","This","implementation","is","motivated","by",":","https",":","\/\/github.com\/jrzaurin\/pytorch-widedeep.git","References",":","--","--","--","--","--","[","1","]","Wu",",","Chuhan",",","et","al",".","``","Fastformer",":","Additive","attention","can","be","all","you","need",".","''","arXiv","preprint","arXiv:2108.09084","(","2021",")","."]
code: def forward(
      self,
      x_q: Tensor,
      x_kv: Tensor,
      *args,  # Not used. just to make the input consistent with MultiheadAttention.
  ) -> Tuple[Tensor, Dict[str, Tensor]]:
      batch_size, n_q_tokens, d_token = x_q.shape
      batch_size, n_k_tokens, d_token = x_kv.shape
  
      q = (
          self.qv_proj(x_q)
          if self.share_qv_weights
          else self.q_proj(x_q)
      )
      v = (
          self.qv_proj(x_kv)
          if self.share_qv_weights
          else self.v_proj(x_kv)
      )
      k = self.k_proj(x_kv)
  
      alphas = (self.W_q(q) / math.sqrt(self.head_dim)).softmax(dim=1)
      q_r = q.reshape(
          batch_size, n_q_tokens, self.n_heads, self.head_dim
      )
      global_query = torch.einsum(
          " b s h, b s h d -> b h d", alphas, q_r
      )
      global_query = global_query.reshape(
          batch_size, self.n_heads * self.head_dim
      ).unsqueeze(1)
  
      p = k * global_query
  
      betas = (self.W_k(p) / math.sqrt(self.head_dim)).softmax(dim=1)
      p_r = p.reshape(
          batch_size, n_k_tokens, self.n_heads, self.head_dim
      )
      global_key = torch.einsum(" b s h, b s h d -> b h d", betas, p_r)
      global_key = global_key.reshape(
          batch_size, self.n_heads * self.head_dim
      ).unsqueeze(1)
  
      u = v * global_key
      output = q + self.dropout(self.r_out(u))
  
      return output, {
          "query_weight": alphas,
          "key_weight": betas,
      }
  
imports: import enum
  import math
  import warnings
  from typing import Callable, Dict, List, Optional, Tuple, Union, cast
  import torch
  import torch.nn.functional
  from torch import Tensor, nn
  from .utils import init_weights
label_desc: Use image node_id 2 for calling the AdditiveAttention obj's underlying member method code with example usage: obj.forward(x_q, x_kv) and returns: output,  dict