class_docstr: ## Layer Normalization
  
  Layer normalization $\text{LN}$ normalizes the input $X$ as follows:
  
  When input $X \in \mathbb{R}^{B \times C}$ is a batch of embeddings,
  where $B$ is the batch size and $C$ is the number of features.
  $\gamma \in \mathbb{R}^{C}$ and $\beta \in \mathbb{R}^{C}$.
  $$\text{LN}(X) = \gamma
  \frac{X - \underset{C}{\mathbb{E}}[X]}{\sqrt{\underset{C}{Var}[X] + \epsilon}}
  + \beta$$
  
  When input $X \in \mathbb{R}^{L \times B \times C}$ is a batch of a sequence of embeddings,
  where $B$ is the batch size, $C$ is the number of channels, $L$ is the length of the sequence.
  $\gamma \in \mathbb{R}^{C}$ and $\beta \in \mathbb{R}^{C}$.
  $$\text{LN}(X) = \gamma
  \frac{X - \underset{C}{\mathbb{E}}[X]}{\sqrt{\underset{C}{Var}[X] + \epsilon}}
  + \beta$$
  
  When input $X \in \mathbb{R}^{B \times C \times H \times W}$ is a batch of image representations,
  where $B$ is the batch size, $C$ is the number of channels, $H$ is the height and $W$ is the width.
  This is not a widely used scenario.
  $\gamma \in \mathbb{R}^{C \times H \times W}$ and $\beta \in \mathbb{R}^{C \times H \times W}$.
  $$\text{LN}(X) = \gamma
  \frac{X - \underset{C, H, W}{\mathbb{E}}[X]}{\sqrt{\underset{C, H, W}{Var}[X] + \epsilon}}
  + \beta$$
docstr: * `normalized_shape` $S$ is the shape of the elements (except the batch).
   The input should then be
   $X \in \mathbb{R}^{*   imes S[0]       imes S[1]       imes ...        imes S[n]}$
  * `eps` is $\epsilon$, used in $\sqrt{Var[X] + \epsilon}$ for numerical stability
  * `elementwise_affine` is whether to scale and shift the normalized value
  
  We've tried to use the same names for arguments as PyTorch `LayerNorm` implementation.
code: def __init__(
      self,
      normalized_shape: Union[int, List[int], Size],
      *,
      eps: float = 1e-5,
      elementwise_affine: bool = True,
  ):
      """
      * `normalized_shape` $S$ is the shape of the elements (except the batch).
       The input should then be
       $X \in \mathbb{R}^{* \times S[0] \times S[1] \times ... \times S[n]}$
      * `eps` is $\epsilon$, used in $\sqrt{Var[X] + \epsilon}$ for numerical stability
      * `elementwise_affine` is whether to scale and shift the normalized value
  
      We've tried to use the same names for arguments as PyTorch `LayerNorm` implementation.
      """
      super().__init__()
  
      # Convert `normalized_shape` to `torch.Size`
      if isinstance(normalized_shape, int):
          normalized_shape = torch.Size([normalized_shape])
      elif isinstance(normalized_shape, list):
          normalized_shape = torch.Size(normalized_shape)
      assert isinstance(normalized_shape, torch.Size)
  
      #
      self.normalized_shape = normalized_shape
      self.eps = eps
      self.elementwise_affine = elementwise_affine
      # Create parameters for $\gamma$ and $\beta$ for gain and bias
      if self.elementwise_affine:
          self.gain = nn.Parameter(torch.ones(normalized_shape))
          self.bias = nn.Parameter(torch.zeros(normalized_shape))
  
file_path: annotated_deep_learning_paper_implementations/labml_nn/normalization/layer_norm/__init__.py
imports: from typing import Union, List
  import torch
  from torch import nn, Size
  from labml_helpers.module import Module
label_desc: Use image node_id 1 to create a new LayerNorm object from inherited base classes: Module with example: obj = LayerNorm(normalized_shape)