class_docstr: ### AFT Local Operation
  
  $$Y_t = \sigma(Q_t) \odot
   rac{\sum_{t'=1}^T \exp(K_{t'} + w_{t,t'}) \odot V_{t'}}
   {\sum_{t'=1}^T \exp(K_{t'} + w_{t,t'})}$$
  
  where,
  
  egin{align}
  w'_{t,t'} =
  egin{cases}
  w_{t,t'},  & {      ext{for } \lvert t-t' 
  vert \lt s} \
  0, &        ext{otherwise}
  \end{cases}
  \end{align}
docstr: * `d_model` is the number of features in the `query`, `key` and `value` vectors.
  * `seq_len` is $T$
  * `local_window_size` is the local window size $s$
  * `bias` is whether to have a bias parameter for transformations for $Q$, $K$ and $V$.
code: def __init__(
      self,
      d_model: int,
      seq_len: int,
      local_window_size: int,
      bias: bool = True,
  ):
      """
      * `d_model` is the number of features in the `query`, `key` and `value` vectors.
      * `seq_len` is $T$
      * `local_window_size` is the local window size $s$
      * `bias` is whether to have a bias parameter for transformations for $Q$, $K$ and $V$.
      """
  
      super().__init__()
  
      # Local window size $s$
      self.local_window_size = local_window_size
      # These transform the `query`, `key` and `value` vectors.
      self.query = nn.Linear(d_model, d_model, bias=bias)
      self.key = nn.Linear(d_model, d_model, bias=bias)
      self.value = nn.Linear(d_model, d_model, bias=bias)
      # Pair-wise positional biases $w \in \mathbb{R}^{T \times T}$
      self.pos_bias = nn.Parameter(
          torch.zeros(seq_len, seq_len), requires_grad=True
      )
      # Mask for $w_{t,t'}$
      self.local_mask = nn.Parameter(
          self.create_local_mask(seq_len, local_window_size),
          requires_grad=False,
      )
      # Activation $\sigma$
      self.activation = nn.Sigmoid()
      # Output layer
      self.output = nn.Linear(d_model, d_model)
  
file_path: annotated_deep_learning_paper_implementations/labml_nn/transformers/aft/__init__.py
imports: from typing import Optional
  import torch
  from torch import nn
  from labml_helpers.module import Module
label_desc: Use image node_id 1 to create a new AFTLocal object from inherited base classes: Module with example: obj = AFTLocal(d_model, seq_len, local_window_size, bias)