class_docstr: Allows the model to jointly attend to information
  from different representation subspaces.
  See reference: Attention Is All You Need
  
  .. math::
      text{MultiHead}(Q, K, V) = text{Concat}(head_1,dots,head_h)W^O
      text{where} head_i = text{Attention}(QW_i^Q, KW_i^K, VW_i^V)
  
  Args:
      embed_dim: total dimension of the model.
      num_heads: parallel attention heads.
      dropout: a Dropout layer on attn_output_weights. Default: 0.0.
      bias: add bias as module parameter. Default: True.
      add_bias_kv: add bias to the key and value sequences at dim=0.
      add_zero_attn: add a new batch of zeros to the key and
                     value sequences at dim=1.
      kdim: total number of features in key. Default: None.
      vdim: total number of features in value. Default: None.
  
      Note: if kdim and vdim are None, they will be set to embed_dim such that
      query, key, and value have the same number of features.
  
  Examples::
  
      >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
      >>> attn_output, attn_output_weights = multihead_attn(query, key, value)
class_docstr_tok: ["Allows","the","model","to","jointly","attend","to","information","from","different","representation","subspaces",".","See","reference",":","Attention","Is","All","You","Need","..","math",":",":","text","{","MultiHead","}","(","Q",",","K",",","V",")","=","text","{","Concat","}","(","head_1",",","dots",",","head_h",")","W^O","text","{","where","}","head_i","=","text","{","Attention","}","(","QW_i^Q",",","KW_i^K",",","VW_i^V",")","Args",":","embed_dim",":","total","dimension","of","the","model",".","num_heads",":","parallel","attention","heads",".","dropout",":","a","Dropout","layer","on","attn_output_weights",".","Default",":","0.0.","bias",":","add","bias","as","module","parameter",".","Default",":","True",".","add_bias_kv",":","add","bias","to","the","key","and","value","sequences","at","dim=0",".","add_zero_attn",":","add","a","new","batch","of","zeros","to","the","key","and","value","sequences","at","dim=1",".","kdim",":","total","number","of","features","in","key",".","Default",":","None",".","vdim",":","total","number","of","features","in","value",".","Default",":","None",".","Note",":","if","kdim","and","vdim","are","None",",","they","will","be","set","to","embed_dim","such","that","query",",","key",",","and","value","have","the","same","number","of","features",".","Examples",":",":",">",">",">","multihead_attn","=","nn.MultiheadAttention","(","embed_dim",",","num_heads",")",">",">",">","attn_output",",","attn_output_weights","=","multihead_attn","(","query",",","key",",","value",")"]
docstr: Args:
      query, key, value: map a query and a set of key-value pairs to an output.
          See "Attention Is All You Need" for more details.
      key_padding_mask: if provided, specified padding elements in the key will
          be ignored by the attention. When given a binary mask and a value is True,
          the corresponding value on the attention layer will be ignored. When given
          a byte mask and a value is non-zero, the corresponding value on the attention
          layer will be ignored
      need_weights: output attn_output_weights.
      attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
          the batches while a 3D mask allows to specify a different mask for the entries of each batch.
  
  Shape:
      - Inputs:
      - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
        the embedding dimension.
      - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
        the embedding dimension.
      - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
        the embedding dimension.
      - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
        If a ByteTensor is provided, the non-zero positions will be ignored while the position
        with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the
        value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
      - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
        3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
        S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked
        positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
        while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
        is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
        is provided, it will be added to the attention weight.
  
      - Outputs:
      - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
        E is the embedding dimension.
      - attn_output_weights: :math:`(N, L, S)` where N is the batch size,
        L is the target sequence length, S is the source sequence length.
docstr_tok: ["Args",":","query",",","key",",","value",":","map","a","query","and","a","set","of","key-value","pairs","to","an","output",".","See","``","Attention","Is","All","You","Need","''","for","more","details",".","key_padding_mask",":","if","provided",",","specified","padding","elements","in","the","key","will","be","ignored","by","the","attention",".","When","given","a","binary","mask","and","a","value","is","True",",","the","corresponding","value","on","the","attention","layer","will","be","ignored",".","When","given","a","byte","mask","and","a","value","is","non-zero",",","the","corresponding","value","on","the","attention","layer","will","be","ignored","need_weights",":","output","attn_output_weights",".","attn_mask",":","2D","or","3D","mask","that","prevents","attention","to","certain","positions",".","A","2D","mask","will","be","broadcasted","for","all","the","batches","while","a","3D","mask","allows","to","specify","a","different","mask","for","the","entries","of","each","batch",".","Shape",":","-","Inputs",":","-","query",":",":","math",":","`","(","L",",","N",",","E",")","`","where","L","is","the","target","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".","-","key",":",":","math",":","`","(","S",",","N",",","E",")","`",",","where","S","is","the","source","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".","-","value",":",":","math",":","`","(","S",",","N",",","E",")","`","where","S","is","the","source","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".","-","key_padding_mask",":",":","math",":","`","(","N",",","S",")","`","where","N","is","the","batch","size",",","S","is","the","source","sequence","length",".","If","a","ByteTensor","is","provided",",","the","non-zero","positions","will","be","ignored","while","the","position","with","the","zero","positions","will","be","unchanged",".","If","a","BoolTensor","is","provided",",","the","positions","with","the","value","of","``","True","``","will","be","ignored","while","the","position","with","the","value","of","``","False","``","will","be","unchanged",".","-","attn_mask",":","2D","mask",":","math",":","`","(","L",",","S",")","`","where","L","is","the","target","sequence","length",",","S","is","the","source","sequence","length",".","3D","mask",":","math",":","`","(","N","*","num_heads",",","L",",","S",")","`","where","N","is","the","batch","size",",","L","is","the","target","sequence","length",",","S","is","the","source","sequence","length",".","attn_mask","ensure","that","position","i","is","allowed","to","attend","the","unmasked","positions",".","If","a","ByteTensor","is","provided",",","the","non-zero","positions","are","not","allowed","to","attend","while","the","zero","positions","will","be","unchanged",".","If","a","BoolTensor","is","provided",",","positions","with","``","True","``","is","not","allowed","to","attend","while","``","False","``","values","will","be","unchanged",".","If","a","FloatTensor","is","provided",",","it","will","be","added","to","the","attention","weight",".","-","Outputs",":","-","attn_output",":",":","math",":","`","(","L",",","N",",","E",")","`","where","L","is","the","target","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".","-","attn_output_weights",":",":","math",":","`","(","N",",","L",",","S",")","`","where","N","is","the","batch","size",",","L","is","the","target","sequence","length",",","S","is","the","source","sequence","length","."]
code: def forward(
      self,
      query,
      key,
      value,
      key_padding_mask=None,
      need_weights=True,
      attn_mask=None,
  ):
      # type: (Tensor, Tensor, Tensor, Optional[Tensor], bool, Optional[Tensor]) -> Tuple[Tensor, Optional[Tensor]]
      """
      Args:
          query, key, value: map a query and a set of key-value pairs to an output.
              See "Attention Is All You Need" for more details.
          key_padding_mask: if provided, specified padding elements in the key will
              be ignored by the attention. When given a binary mask and a value is True,
              the corresponding value on the attention layer will be ignored. When given
              a byte mask and a value is non-zero, the corresponding value on the attention
              layer will be ignored
          need_weights: output attn_output_weights.
          attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
              the batches while a 3D mask allows to specify a different mask for the entries of each batch.
  
      Shape:
          - Inputs:
          - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
            the embedding dimension.
          - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
            the embedding dimension.
          - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
            the embedding dimension.
          - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
            If a ByteTensor is provided, the non-zero positions will be ignored while the position
            with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the
            value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
          - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
            3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
            S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked
            positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
            while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
            is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
            is provided, it will be added to the attention weight.
  
          - Outputs:
          - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
            E is the embedding dimension.
          - attn_output_weights: :math:`(N, L, S)` where N is the batch size,
            L is the target sequence length, S is the source sequence length.
      """
  
      return multi_head_attention_forward(
          self,
          query=query,
          key=key,
          value=value,
          embed_dim_to_check=self.embed_dim,
          num_heads=self.num_heads,
          in_proj_weight=self.in_proj_weight,
          in_proj_bias=self.in_proj_bias,
          bias_k=self.bias_k,
          bias_v=self.bias_v,
          add_zero_attn=self.add_zero_attn,
          dropout_p=self.dropout,
          out_proj_weight=self.out_proj.weight,
          out_proj_bias=self.out_proj.bias,
          fixed_k=self.fixed_k,
          fixed_q=self.fixed_q,
          training=self.training,
          key_padding_mask=key_padding_mask,
          need_weights=need_weights,
          attn_mask=attn_mask,
      )
  
imports: import math
  from typing import Optional, Tuple
  import torch
  import torch.nn.functional
  from torch.nn import Module, init
  from torch.nn.functional import dropout, linear, softmax
  from torch.nn.init import constant_, xavier_normal_, xavier_uniform_
  from torch.nn.modules.dropout import Dropout
  from torch.nn.modules.normalization import LayerNorm
  from torch.nn.parameter import Parameter
label_desc: Use image node_id 4 for calling the MultiheadAttention obj's underlying member method code with example usage: obj.forward(query, key, value, key_padding_mask, need_weights, attn_mask) and returns: multi_head_attention_forward