class_docstr: Local state allows to have attention based only on data (no positional embedding),
  but while setting a constraint on the time window (e.g. decaying penalty term).
  Also a failed experiments with trying to provide some frequency based attention.
class_docstr_tok: ["Local","state","allows","to","have","attention","based","only","on","data","(","no","positional","embedding",")",",","but","while","setting","a","constraint","on","the","time","window","(","e.g",".","decaying","penalty","term",")",".","Also","a","failed","experiments","with","trying","to","provide","some","frequency","based","attention","."]
docstr: Args:
      channels (int): Size of Conv1d layers.
      heads (int, optional):  (default: 4)
      ndecay (int, optional): (default: 4)
docstr_tok: ["Args",":","channels","(","int",")",":","Size","of","Conv1d","layers",".","heads","(","int",",","optional",")",":","(","default",":","4",")","ndecay","(","int",",","optional",")",":","(","default",":","4",")"]
code: def __init__(self, channels: int, heads: int = 4, ndecay: int = 4):
      r"""
      Args:
          channels (int): Size of Conv1d layers.
          heads (int, optional):  (default: 4)
          ndecay (int, optional): (default: 4)
      """
      super(_LocalState, self).__init__()
      if channels % heads != 0:
          raise ValueError("Channels must be divisible by heads.")
      self.heads = heads
      self.ndecay = ndecay
      self.content = nn.Conv1d(channels, channels, 1)
      self.query = nn.Conv1d(channels, channels, 1)
      self.key = nn.Conv1d(channels, channels, 1)
  
      self.query_decay = nn.Conv1d(channels, heads * ndecay, 1)
      if ndecay:
          # Initialize decay close to zero (there is a sigmoid), for maximum initial window.
          self.query_decay.weight.data *= 0.01
          if self.query_decay.bias is None:
              raise ValueError("bias must not be None.")
          self.query_decay.bias.data[:] = -2
      self.proj = nn.Conv1d(channels + heads * 0, channels, 1)
  
imports: import math
  import typing
  from typing import Any, Dict, List, Optional
  import torch
  from torch import nn
  from torch.nn import functional
label_desc: Use image node_id 1 to create a new _LocalState object from inherited base classes: nn with example: obj = _LocalState(channels, heads, ndecay)