class_docstr: Relative Position Embedding implementation from the T5 paper : https://arxiv.org/abs/1910.10683
code: def __init__(
      self,
      init_method,
      bidirectional,
      num_attention_heads,
      layer_type,
      relative_position_num_buckets=32,
      relative_position_max_distance=128,
  ):
      super(T5RelativePositionEmbedding, self).__init__()
      self.relative_position_num_buckets = relative_position_num_buckets
      self.relative_position_max_distance = (
          relative_position_max_distance
      )
      self.self_attention_relative_position_bucket = None
      self.inter_attention_relative_position_bucket = None
      self.self_attention_relative_position_bias = None
      self.inter_attention_relative_position_bias = None
      self.bidirectional = bidirectional
  
      # LayerType.encoder or LayerType.decoder. Is only needed to determine the group for the all_reduce
      self.layer_type = layer_type
  
      # Relative position Embedding
      # Relative Position embedding (all attention layers).
      self.relative_position_embedding = torch.nn.Embedding(
          self.relative_position_num_buckets, num_attention_heads
      )
      self._relative_position_embedding_key = (
          "relative_position_embedding"
      )
      init_method(self.relative_position_embedding.weight)
  
file_path: NeMo/nemo/collections/nlp/modules/common/megatron/position_embedding/t5_relative_position_embedding.py
imports: import math
  import torch
label_desc: Use image node_id 1 to create a new T5RelativePositionEmbedding object from inherited base classes: torch.nn with example: obj = T5RelativePositionEmbedding(init_method, bidirectional, num_attention_heads, layer_type, relative_position_num_buckets, relative_position_max_distance)