class_docstr: ALiBi (Attention with Linear Biases) relative position embedding for auto-regressive decoder
  and joint encoder (symmetric for forward and backward distance).
  Based on https://arxiv.org/bas/2108.12409
docstr: Args:
      bidirectional: Whether to use bidirectional relative position embedding
      num_attention_heads: Number of attention heads
      layer_type: Layer type. Can be one of [LayerType.encoder or LayerType.decoder]. Willdetermine the bias construction
      num_attention_heads_alibi: Number of attention heads for which alibi bias will be used
      max_seq_len: Maximum sequence length for precomputed relative positions. Larger sizes will result in more memory usage by computing alibi mask on-the-fly.
code: def __init__(
      self,
      bidirectional,
      num_attention_heads,
      layer_type,
      num_attention_heads_alibi=None,
      max_seq_len=512,
  ):
      """
      Args:
          bidirectional: Whether to use bidirectional relative position embedding
          num_attention_heads: Number of attention heads
          layer_type: Layer type. Can be one of [LayerType.encoder or LayerType.decoder]. Willdetermine the bias construction
          num_attention_heads_alibi: Number of attention heads for which alibi bias will be used
          max_seq_len: Maximum sequence length for precomputed relative positions. Larger sizes will result in more memory usage by computing alibi mask on-the-fly.
      """
      super().__init__()
  
      if (num_attention_heads_alibi is None) or (
          num_attention_heads_alibi <= 0
      ):
          num_attention_heads_alibi = num_attention_heads
  
      if num_attention_heads_alibi > num_attention_heads:
          raise ValueError(
              f"num_attention_heads_alibi ({num_attention_heads_alibi}) cannot be larger than num_attention_heads ({num_attention_heads})"
          )
  
      self.bidirectional = bidirectional
      self.num_attention_heads = num_attention_heads
      # LayerType.encoder or LayerType.decoder. Is only needed to determine the group for the all_reduce
      self.layer_type = layer_type
      # define the size of pre-computed relative position slopes.
      # define the number of attention heads for which alibi mask will be pre-computed (the rest are disabled).
      self.num_attention_heads_alibi = num_attention_heads_alibi
      # Larger sizes will result in more memory usage by computing alibi mask on-the-fly.
      self.max_seq_len = max_seq_len
  
      # cache the slopes
      self.slopes = build_slopes(
          num_attention_heads, num_attention_heads_alibi
      )
      # cache the relative position bias. shape (num_attention_heads, max_seq_len, max_seq_len)
      # if we use causal attention (not bidrectional), we can use singleton relative position
      self.relative_position = (
          build_relative_position(max_seq_len, full=bidirectional)
          .unsqueeze(0)
          .expand(num_attention_heads, -1, -1)
      )
  
file_path: NeMo/nemo/collections/nlp/modules/common/megatron/position_embedding/alibi_relative_position_embedding.py
imports: import math
  import torch
label_desc: Use image node_id 1 to create a new ALiBiRelativePositionEmbedding object from inherited base classes: torch.nn with example: obj = ALiBiRelativePositionEmbedding(bidirectional, num_attention_heads, layer_type, num_attention_heads_alibi, max_seq_len)