class_docstr: This class is designed to support the `relative position <https://arxiv.org/pdf/1803.02155v2.pdf>`__ in attention.
  
  Different from the absolute position embedding,
  the relative position embedding encodes relative distance between input tokens and learn the pairwise relations of them.
  It is commonly calculated via a look-up table with learnable parameters,
  interacting with queries and keys in self-attention modules.
  
  This class is different from PyTorch's built-in ``nn.MultiheadAttention`` in:
  
  1. It supports relative position embedding.
  2. It only supports self attention.
  3. It uses fixed dimension for each head, rather than fixed total dimension.
code: def __init__(
      self,
      embed_dim: int | Categorical[int],
      num_heads: int | Categorical[int],
      head_dim: int | None = 64,
      attn_drop: float = 0.0,
      proj_drop: float = 0.0,
      qkv_bias: bool = False,
      qk_scale: float | None = None,
      rpe: bool = False,
      rpe_length: int = 14,
  ):
      super().__init__()
  
      # The self. attributes are only used for inspection.
      # The actual values are stored in the submodules.
      if current_model() is not None:
          self.embed_dim = ensure_frozen(embed_dim)
          self.num_heads = ensure_frozen(num_heads)
      else:
          self.embed_dim = embed_dim
          self.num_heads = num_heads
  
      # head_dim is fixed 64 in official AutoFormer. set head_dim = None to use flex head dim.
      self.head_dim = head_dim or (embed_dim // num_heads)
      self.scale = qk_scale or cast(int, head_dim) ** -0.5
      self.qkv_bias = qkv_bias
  
      if isinstance(head_dim, Mutable) and isinstance(
          num_heads, Mutable
      ):
          raise ValueError(
              "head_dim and num_heads can not be both mutable."
          )
  
      # Please refer to MixedMultiheadAttention for details.
      self.q = MutableLinear(
          cast(int, embed_dim),
          cast(int, head_dim) * num_heads,
          bias=qkv_bias,
      )
      self.k = MutableLinear(
          cast(int, embed_dim),
          cast(int, head_dim) * num_heads,
          bias=qkv_bias,
      )
      self.v = MutableLinear(
          cast(int, embed_dim),
          cast(int, head_dim) * num_heads,
          bias=qkv_bias,
      )
  
      self.attn_drop = nn.Dropout(attn_drop)
      self.proj = MutableLinear(
          cast(int, head_dim) * num_heads, cast(int, embed_dim)
      )
      self.proj_drop = nn.Dropout(proj_drop)
      self.rpe = rpe
  
      if self.rpe:
          if isinstance(head_dim, Mutable):
              raise ValueError(
                  "head_dim must be a fixed integer when rpe is True."
              )
          self.rel_pos_embed_k = RelativePosition2D(
              cast(int, head_dim), rpe_length
          )
          self.rel_pos_embed_v = RelativePosition2D(
              cast(int, head_dim), rpe_length
          )
  
imports: from __future__ import annotations
  from typing import Tuple, cast, Any, Dict
  import torch
  import torch.nn
  import torch.nn.functional
  import nni
  from nni.mutable import Categorical, MutableExpression, frozen_context, ensure_frozen, Mutable
  from nni.nas.nn.pytorch import ParametrizedModule, ModelSpace, MutableModule, Repeat, MutableConv2d, MutableLinear, MutableLayerNorm
  from nni.nas.space import current_model
  from nni.nas.oneshot.pytorch.supermodule.operation import MixedOperation
  from nni.nas.profiler.pytorch.flops import FlopsResult
  from nni.nas.profiler.pytorch.utils import MutableShape, ShapeTensor, profiler_leaf_module
  from .utils.pretrained import load_pretrained_weight
  from .utils.nn import DropPath
label_desc: Use image node_id 1 to create a new RelativePositionSelfAttention object from inherited base classes: MutableModule with example: obj = RelativePositionSelfAttention(embed_dim, num_heads, head_dim, attn_drop, proj_drop, qkv_bias, qk_scale, rpe, rpe_length)