class_docstr: <a id="TransformerLayer"></a>
  
  ## Transformer Layer
  
  This can act as an encoder layer or a decoder layer.
  
  🗒 Some implementations, including the paper seem to have differences
  in where the layer-normalization is done.
  Here we do a layer normalization before attention and feed-forward networks,
  and add the original residual vectors.
  Alternative is to do a layer normalization after adding the residuals.
  But we found this to be less stable when training.
  We found a detailed discussion about this in the paper
   [On Layer Normalization in the Transformer Architecture](https://arxiv.org/abs/2002.04745).
docstr: * `d_model` is the token embedding size
  * `self_attn` is the self attention module
  * `src_attn` is the source attention module (when this is used in a decoder)
  * `feed_forward` is the feed forward module
  * `dropout_prob` is the probability of dropping out after self attention and FFN
docstr_tok: ["*","`","d_model","`","is","the","token","embedding","size","*","`","self_attn","`","is","the","self","attention","module","*","`","src_attn","`","is","the","source","attention","module","(","when","this","is","used","in","a","decoder",")","*","`","feed_forward","`","is","the","feed","forward","module","*","`","dropout_prob","`","is","the","probability","of","dropping","out","after","self","attention","and","FFN"]
code: def __init__(
      self,
      *,
      d_model: int,
      self_attn: MultiHeadAttention,
      src_attn: MultiHeadAttention = None,
      feed_forward: FeedForward,
      dropout_prob: float,
  ):
      """
      * `d_model` is the token embedding size
      * `self_attn` is the self attention module
      * `src_attn` is the source attention module (when this is used in a decoder)
      * `feed_forward` is the feed forward module
      * `dropout_prob` is the probability of dropping out after self attention and FFN
      """
      super().__init__()
      self.size = d_model
      self.self_attn = self_attn
      self.src_attn = src_attn
      self.feed_forward = feed_forward
      self.dropout = nn.Dropout(dropout_prob)
      self.norm_self_attn = nn.LayerNorm([d_model])
      if self.src_attn is not None:
          self.norm_src_attn = nn.LayerNorm([d_model])
      self.norm_ff = nn.LayerNorm([d_model])
      # Whether to save input to the feed forward layer
      self.is_save_ff_input = False
  
imports: import math
  import torch
  import torch.nn
  from labml_nn.utils import clone_module_list
  from .feed_forward import FeedForward
  from .mha import MultiHeadAttention
  from .positional_encoding import get_positional_encoding
label_desc: Use image node_id 1 to create a new TransformerLayer object from inherited base classes: nn with example: obj = TransformerLayer()