class_docstr: Multi-headed self-attention for WavLM model :cite:`chen2022wavlm`.
  Wraps around ``torch.nn.MultiheadAttention``, creating relaive position embeddings and passing them to multi-headed
  attention as a mask.
  Source: https://github.com/microsoft/unilm/blob/2d8302f09c99bca2b82e6e868d81d4281cceebc8/wavlm/modules.py#L303-L763
  
  Args:
      embed_dim (int): Total dimension of the model.
      num_heads (int): The number of heads.
      dropout (float, optional): Dropout probability on attn_output_weights. (Default: to ``0.0``)
      bias (bool, optional): If ``True``, add bias to input / output projection layers. (Default: ``True``)
      has_relative_attention_bias (bool, optional): If ``True``, apply relative position embedding.
          Necessary in the first encoder layer, but not in the subsequent ones. (Default: ``False``)
      num_buckets (int, optional): Number of buckets for relative position embedding. (Default: ``32``)
      max_distance (int, optional): Naximum distance for relative position embedding. (Default: ``128``)
      gru_rel_pos (bool, optional): If ``True``, apply gated relative position embedding. (Default: ``False``)
code: def __init__(
      self,
      embed_dim: int,
      num_heads: int,
      dropout: float = 0.0,
      bias: bool = True,
      has_relative_attention_bias: bool = False,
      num_buckets: int = 32,
      max_distance: int = 128,
      gru_rel_pos: bool = True,
  ):
      super().__init__()
      self.embed_dim = embed_dim
      self.num_heads = num_heads
      self.has_relative_attention_bias = has_relative_attention_bias
      self.num_buckets = num_buckets
      self.max_distance = max_distance
  
      if has_relative_attention_bias:
          self.rel_attn_embed = nn.Embedding(num_buckets, num_heads)
      else:
          self.rel_attn_embed = None
  
      self.head_dim = embed_dim // num_heads
      assert (
          self.head_dim * num_heads == self.embed_dim
      ), "embed_dim must be divisible by num_heads"
  
      self.dropout = dropout
      self.attention = nn.MultiheadAttention(
          embed_dim,
          num_heads,
          dropout=dropout,
          bias=bias,
          batch_first=True,
      )
  
      self.gru_rel_pos = gru_rel_pos
      if self.gru_rel_pos:
          self.gru_rel_pos_linear = nn.Linear(self.head_dim, 8)
          self.gru_rel_pos_const = nn.Parameter(
              torch.ones(1, num_heads, 1, 1)
          )
      self.has_position_bias = True
  
file_path: audio/src/torchaudio/models/wav2vec2/wavlm_attention.py
imports: import math
  from typing import Optional, Tuple
  import torch
  from torch import nn, Tensor
label_desc: Use image node_id 1 to create a new WavLMSelfAttention object from inherited base classes: nn with example: obj = WavLMSelfAttention(embed_dim, num_heads, dropout, bias, has_relative_attention_bias, num_buckets, max_distance, gru_rel_pos)