class_docstr: TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network.
  
  This standard decoder layer is based on the paper "Attention Is All You Need".
  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
  Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in
  Neural Information Processing Systems, pages 6000-6010. Users may modify or implement
  in a different way during application.
  
  Args:
      d_model: the number of expected features in the input (required).
      nhead: the number of heads in the multiheadattention models (required).
      dim_feedforward: the dimension of the feedforward network model (default=2048).
      dropout: the dropout value (default=0.1).
      activation: the activation function of the intermediate layer, can be a string
          ("relu" or "gelu") or a unary callable. Default: relu
      layer_norm_eps: the eps value in layer normalization components (default=1e-5).
      batch_first: If ``True``, then the input and output tensors are provided
          as (batch, seq, feature). Default: ``False`` (seq, batch, feature).
      norm_first: if ``True``, layer norm is done prior to self attention, multihead
          attention and feedforward operations, respectively. Otherwise it's done after.
          Default: ``False`` (after).
      bias: If set to ``False``, ``Linear`` and ``LayerNorm`` layers will not learn an additive
          bias. Default: ``True``.
  
  Examples::
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
      >>> memory = torch.rand(10, 32, 512)
      >>> tgt = torch.rand(20, 32, 512)
      >>> out = decoder_layer(tgt, memory)
  
  Alternatively, when ``batch_first`` is ``True``:
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8, batch_first=True)
      >>> memory = torch.rand(32, 10, 512)
      >>> tgt = torch.rand(32, 20, 512)
      >>> out = decoder_layer(tgt, memory)
class_docstr_tok: ["TransformerDecoderLayer","is","made","up","of","self-attn",",","multi-head-attn","and","feedforward","network",".","This","standard","decoder","layer","is","based","on","the","paper","``","Attention","Is","All","You","Need","''",".","Ashish","Vaswani",",","Noam","Shazeer",",","Niki","Parmar",",","Jakob","Uszkoreit",",","Llion","Jones",",","Aidan","N","Gomez",",","Lukasz","Kaiser",",","and","Illia","Polosukhin",".","2017",".","Attention","is","all","you","need",".","In","Advances","in","Neural","Information","Processing","Systems",",","pages","6000-6010",".","Users","may","modify","or","implement","in","a","different","way","during","application",".","Args",":","d_model",":","the","number","of","expected","features","in","the","input","(","required",")",".","nhead",":","the","number","of","heads","in","the","multiheadattention","models","(","required",")",".","dim_feedforward",":","the","dimension","of","the","feedforward","network","model","(","default=2048",")",".","dropout",":","the","dropout","value","(","default=0.1",")",".","activation",":","the","activation","function","of","the","intermediate","layer",",","can","be","a","string","(","``","relu","''","or","``","gelu","''",")","or","a","unary","callable",".","Default",":","relu","layer_norm_eps",":","the","eps","value","in","layer","normalization","components","(","default=1e-5",")",".","batch_first",":","If","``","True","``",",","then","the","input","and","output","tensors","are","provided","as","(","batch",",","seq",",","feature",")",".","Default",":","``","False","``","(","seq",",","batch",",","feature",")",".","norm_first",":","if","``","True","``",",","layer","norm","is","done","prior","to","self","attention",",","multihead","attention","and","feedforward","operations",",","respectively",".","Otherwise","it","'s","done","after",".","Default",":","``","False","``","(","after",")",".","bias",":","If","set","to","``","False","``",",","``","Linear","``","and","``","LayerNorm","``","layers","will","not","learn","an","additive","bias",".","Default",":","``","True","``",".","Examples",":",":",">",">",">","decoder_layer","=","nn.TransformerDecoderLayer","(","d_model=512",",","nhead=8",")",">",">",">","memory","=","torch.rand","(","10",",","32",",","512",")",">",">",">","tgt","=","torch.rand","(","20",",","32",",","512",")",">",">",">","out","=","decoder_layer","(","tgt",",","memory",")","Alternatively",",","when","``","batch_first","``","is","``","True","``",":",">",">",">","decoder_layer","=","nn.TransformerDecoderLayer","(","d_model=512",",","nhead=8",",","batch_first=True",")",">",">",">","memory","=","torch.rand","(","32",",","10",",","512",")",">",">",">","tgt","=","torch.rand","(","32",",","20",",","512",")",">",">",">","out","=","decoder_layer","(","tgt",",","memory",")"]
docstr: Pass the inputs (and mask) through the decoder layer.
  
  Args:
      tgt: the sequence to the decoder layer (required).
      memory: the sequence from the last layer of the encoder (required).
      tgt_mask: the mask for the tgt sequence (optional).
      memory_mask: the mask for the memory sequence (optional).
      tgt_key_padding_mask: the mask for the tgt keys per batch (optional).
      memory_key_padding_mask: the mask for the memory keys per batch (optional).
      tgt_is_causal: If specified, applies a causal mask as ``tgt mask``.
          Default: ``False``.
          Warning:
          ``tgt_is_causal`` provides a hint that ``tgt_mask`` is
          the causal mask. Providing incorrect hints can result in
          incorrect execution, including forward and backward
          compatibility.
      memory_is_causal: If specified, applies a causal mask as
          ``memory mask``.
          Default: ``False``.
          Warning:
          ``memory_is_causal`` provides a hint that
          ``memory_mask`` is the causal mask. Providing incorrect
          hints can result in incorrect execution, including
          forward and backward compatibility.
  
  Shape:
      see the docs in Transformer class.
docstr_tok: ["Pass","the","inputs","(","and","mask",")","through","the","decoder","layer",".","Args",":","tgt",":","the","sequence","to","the","decoder","layer","(","required",")",".","memory",":","the","sequence","from","the","last","layer","of","the","encoder","(","required",")",".","tgt_mask",":","the","mask","for","the","tgt","sequence","(","optional",")",".","memory_mask",":","the","mask","for","the","memory","sequence","(","optional",")",".","tgt_key_padding_mask",":","the","mask","for","the","tgt","keys","per","batch","(","optional",")",".","memory_key_padding_mask",":","the","mask","for","the","memory","keys","per","batch","(","optional",")",".","tgt_is_causal",":","If","specified",",","applies","a","causal","mask","as","``","tgt","mask","``",".","Default",":","``","False","``",".","Warning",":","``","tgt_is_causal","``","provides","a","hint","that","``","tgt_mask","``","is","the","causal","mask",".","Providing","incorrect","hints","can","result","in","incorrect","execution",",","including","forward","and","backward","compatibility",".","memory_is_causal",":","If","specified",",","applies","a","causal","mask","as","``","memory","mask","``",".","Default",":","``","False","``",".","Warning",":","``","memory_is_causal","``","provides","a","hint","that","``","memory_mask","``","is","the","causal","mask",".","Providing","incorrect","hints","can","result","in","incorrect","execution",",","including","forward","and","backward","compatibility",".","Shape",":","see","the","docs","in","Transformer","class","."]
code: def forward(
      self,
      tgt: Tensor,
      memory: Tensor,
      tgt_mask: Optional[Tensor] = None,
      memory_mask: Optional[Tensor] = None,
      tgt_key_padding_mask: Optional[Tensor] = None,
      memory_key_padding_mask: Optional[Tensor] = None,
      tgt_is_causal: bool = False,
      memory_is_causal: bool = False,
  ) -> Tensor:
      r"""Pass the inputs (and mask) through the decoder layer.
  
      Args:
          tgt: the sequence to the decoder layer (required).
          memory: the sequence from the last layer of the encoder (required).
          tgt_mask: the mask for the tgt sequence (optional).
          memory_mask: the mask for the memory sequence (optional).
          tgt_key_padding_mask: the mask for the tgt keys per batch (optional).
          memory_key_padding_mask: the mask for the memory keys per batch (optional).
          tgt_is_causal: If specified, applies a causal mask as ``tgt mask``.
              Default: ``False``.
              Warning:
              ``tgt_is_causal`` provides a hint that ``tgt_mask`` is
              the causal mask. Providing incorrect hints can result in
              incorrect execution, including forward and backward
              compatibility.
          memory_is_causal: If specified, applies a causal mask as
              ``memory mask``.
              Default: ``False``.
              Warning:
              ``memory_is_causal`` provides a hint that
              ``memory_mask`` is the causal mask. Providing incorrect
              hints can result in incorrect execution, including
              forward and backward compatibility.
  
      Shape:
          see the docs in Transformer class.
      """
      # see Fig. 1 of https://arxiv.org/pdf/2002.04745v1.pdf
  
      x = tgt
      if self.norm_first:
          x = x + self._sa_block(
              self.norm1(x),
              tgt_mask,
              tgt_key_padding_mask,
              tgt_is_causal,
          )
          x = x + self._mha_block(
              self.norm2(x),
              memory,
              memory_mask,
              memory_key_padding_mask,
              memory_is_causal,
          )
          x = x + self._ff_block(self.norm3(x))
      else:
          x = self.norm1(
              x
              + self._sa_block(
                  x, tgt_mask, tgt_key_padding_mask, tgt_is_causal
              )
          )
          x = self.norm2(
              x
              + self._mha_block(
                  x,
                  memory,
                  memory_mask,
                  memory_key_padding_mask,
                  memory_is_causal,
              )
          )
          x = self.norm3(x + self._ff_block(x))
  
      return x
  
imports: import copy
  from typing import Optional, Any, Union, Callable
  import torch
  import warnings
  from torch import Tensor
  from ..None import functional
  from .module import Module
  from .activation import MultiheadAttention
  from .container import ModuleList
  from ..init import xavier_uniform_
  from .dropout import Dropout
  from .linear import Linear
  from .normalization import LayerNorm
label_desc: Use image node_id 3 for calling the TransformerDecoderLayer obj's underlying member method code with example usage: obj.forward(tgt, memory, tgt_mask, memory_mask, tgt_key_padding_mask, memory_key_padding_mask, tgt_is_causal, memory_is_causal) and returns: x