docstr: Note::
      Please, refer to :func:`~torch.nn.MultiheadAttention.forward` for more
      information
  
  Args:
      query, key, value: map a query and a set of key-value pairs to an output.
          See "Attention Is All You Need" for more details.
      key_padding_mask: if provided, specified padding elements in the key will
          be ignored by the attention. When given a binary mask and a value is True,
          the corresponding value on the attention layer will be ignored.
      need_weights: output attn_output_weights.
      attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
          the batches while a 3D mask allows to specify a different mask for the entries of each batch.
  
  Shape:
      - Inputs:
      - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
        the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``.
      - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
        the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``.
      - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
        the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``.
      - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
        If a BoolTensor is provided, the positions with the
        value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
      - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
        3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
        S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked
        positions. If a BoolTensor is provided, positions with ``True``
        is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
        is provided, it will be added to the attention weight.
      - is_causal: If specified, applies a causal mask as attention mask. Mutually exclusive with providing attn_mask.
        Default: ``False``.
      - average_attn_weights: If true, indicates that the returned ``attn_weights`` should be averaged across
        heads. Otherwise, ``attn_weights`` are provided separately per head. Note that this flag only has an
        effect when ``need_weights=True.``. Default: True (i.e. average weights across heads)
  
      - Outputs:
      - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
        E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``.
      - attn_output_weights: If ``average_attn_weights=True``, returns attention weights averaged
        across heads of shape :math:`(N, L, S)`, where N is the batch size, L is the target sequence length,
        S is the source sequence length. If ``average_attn_weights=False``, returns attention weights per
        head of shape :math:`(N, num_heads, L, S)`.
      
docstr_tok: ["Note",":",":","Please",",","refer","to",":","func",":","`","~torch.nn.MultiheadAttention.forward","`","for","more","information","Args",":","query",",","key",",","value",":","map","a","query","and","a","set","of","key-value","pairs","to","an","output",".","See","``","Attention","Is","All","You","Need","''","for","more","details",".","key_padding_mask",":","if","provided",",","specified","padding","elements","in","the","key","will","be","ignored","by","the","attention",".","When","given","a","binary","mask","and","a","value","is","True",",","the","corresponding","value","on","the","attention","layer","will","be","ignored",".","need_weights",":","output","attn_output_weights",".","attn_mask",":","2D","or","3D","mask","that","prevents","attention","to","certain","positions",".","A","2D","mask","will","be","broadcasted","for","all","the","batches","while","a","3D","mask","allows","to","specify","a","different","mask","for","the","entries","of","each","batch",".","Shape",":","-","Inputs",":","-","query",":",":","math",":","`","(","L",",","N",",","E",")","`","where","L","is","the","target","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".",":","math",":","`","(","N",",","L",",","E",")","`","if","``","batch_first","``","is","``","True","``",".","-","key",":",":","math",":","`","(","S",",","N",",","E",")","`",",","where","S","is","the","source","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".",":","math",":","`","(","N",",","S",",","E",")","`","if","``","batch_first","``","is","``","True","``",".","-","value",":",":","math",":","`","(","S",",","N",",","E",")","`","where","S","is","the","source","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".",":","math",":","`","(","N",",","S",",","E",")","`","if","``","batch_first","``","is","``","True","``",".","-","key_padding_mask",":",":","math",":","`","(","N",",","S",")","`","where","N","is","the","batch","size",",","S","is","the","source","sequence","length",".","If","a","BoolTensor","is","provided",",","the","positions","with","the","value","of","``","True","``","will","be","ignored","while","the","position","with","the","value","of","``","False","``","will","be","unchanged",".","-","attn_mask",":","2D","mask",":","math",":","`","(","L",",","S",")","`","where","L","is","the","target","sequence","length",",","S","is","the","source","sequence","length",".","3D","mask",":","math",":","`","(","N","*","num_heads",",","L",",","S",")","`","where","N","is","the","batch","size",",","L","is","the","target","sequence","length",",","S","is","the","source","sequence","length",".","attn_mask","ensure","that","position","i","is","allowed","to","attend","the","unmasked","positions",".","If","a","BoolTensor","is","provided",",","positions","with","``","True","``","is","not","allowed","to","attend","while","``","False","``","values","will","be","unchanged",".","If","a","FloatTensor","is","provided",",","it","will","be","added","to","the","attention","weight",".","-","is_causal",":","If","specified",",","applies","a","causal","mask","as","attention","mask",".","Mutually","exclusive","with","providing","attn_mask",".","Default",":","``","False","``",".","-","average_attn_weights",":","If","true",",","indicates","that","the","returned","``","attn_weights","``","should","be","averaged","across","heads",".","Otherwise",",","``","attn_weights","``","are","provided","separately","per","head",".","Note","that","this","flag","only","has","an","effect","when","``","need_weights=True.","``",".","Default",":","True","(","i.e",".","average","weights","across","heads",")","-","Outputs",":","-","attn_output",":",":","math",":","`","(","L",",","N",",","E",")","`","where","L","is","the","target","sequence","length",",","N","is","the","batch","size",",","E","is","the","embedding","dimension",".",":","math",":","`","(","N",",","L",",","E",")","`","if","``","batch_first","``","is","``","True","``",".","-","attn_output_weights",":","If","``","average_attn_weights=True","``",",","returns","attention","weights","averaged","across","heads","of","shape",":","math",":","`","(","N",",","L",",","S",")","`",",","where","N","is","the","batch","size",",","L","is","the","target","sequence","length",",","S","is","the","source","sequence","length",".","If","``","average_attn_weights=False","``",",","returns","attention","weights","per","head","of","shape",":","math",":","`","(","N",",","num_heads",",","L",",","S",")","`","."]
code: def forward(
      self,
      query: Tensor,
      key: Tensor,
      value: Tensor,
      key_padding_mask: Optional[Tensor] = None,
      need_weights: bool = True,
      attn_mask: Optional[Tensor] = None,
      average_attn_weights: bool = True,
      is_causal: bool = False,
  ) -> Tuple[Tensor, Optional[Tensor]]:
      r"""
      Note::
          Please, refer to :func:`~torch.nn.MultiheadAttention.forward` for more
          information
  
      Args:
          query, key, value: map a query and a set of key-value pairs to an output.
              See "Attention Is All You Need" for more details.
          key_padding_mask: if provided, specified padding elements in the key will
              be ignored by the attention. When given a binary mask and a value is True,
              the corresponding value on the attention layer will be ignored.
          need_weights: output attn_output_weights.
          attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
              the batches while a 3D mask allows to specify a different mask for the entries of each batch.
  
      Shape:
          - Inputs:
          - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
            the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``.
          - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
            the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``.
          - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
            the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``.
          - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
            If a BoolTensor is provided, the positions with the
            value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
          - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
            3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
            S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked
            positions. If a BoolTensor is provided, positions with ``True``
            is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
            is provided, it will be added to the attention weight.
          - is_causal: If specified, applies a causal mask as attention mask. Mutually exclusive with providing attn_mask.
            Default: ``False``.
          - average_attn_weights: If true, indicates that the returned ``attn_weights`` should be averaged across
            heads. Otherwise, ``attn_weights`` are provided separately per head. Note that this flag only has an
            effect when ``need_weights=True.``. Default: True (i.e. average weights across heads)
  
          - Outputs:
          - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
            E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``.
          - attn_output_weights: If ``average_attn_weights=True``, returns attention weights averaged
            across heads of shape :math:`(N, L, S)`, where N is the batch size, L is the target sequence length,
            S is the source sequence length. If ``average_attn_weights=False``, returns attention weights per
            head of shape :math:`(N, num_heads, L, S)`.
      """
      return self._forward_impl(
          query,
          key,
          value,
          key_padding_mask,
          need_weights,
          attn_mask,
          average_attn_weights,
          is_causal,
      )
  
file_path: pytorch/torch/ao/nn/quantizable/modules/activation.py
imports: import torch
  import torch.jit
  from torch import nn
  import torch.nn.functional
  from torch import Tensor
  from typing import Optional, Tuple
  import warnings
label_desc: Use image node_id 6 for calling the MultiheadAttention obj's underlying member method code with example usage: obj.forward(query, key, value, key_padding_mask, need_weights, attn_mask, average_attn_weights, is_causal) and returns: self