class_docstr: Multi-head attention with Relative Positional embedding.
  https://arxiv.org/pdf/1809.04281.pdf
  
  It learns positional embeddings for a window of neighbours. For keys and values,
  it learns different set of embeddings. Key embeddings are agregated with the attention
  scores and value embeddings are aggregated with the output.
  
  Note:
      Example with relative attention window size 2
      input = [a, b, c, d, e]
      rel_attn_embeddings = [e(t-2), e(t-1), e(t+1), e(t+2)]
  
      So it learns 4 embedding vectors (in total 8) separately for key and value vectors.
  
      Considering the input c
          e(t-2) corresponds to c -> a
          e(t-2) corresponds to c -> b
          e(t-2) corresponds to c -> d
          e(t-2) corresponds to c -> e
  
      These embeddings are shared among different time steps. So input a, b, d and e also uses
      the same embeddings.
  
      Embeddings are ignored when the relative window is out of limit for the first and the last
      n items.
  
  Args:
      channels (int): input and inner layer channels.
      out_channels (int): output channels.
      num_heads (int): number of attention heads.
      rel_attn_window_size (int, optional): relation attention window size.
          If 4, for each time step next and previous 4 time steps are attended.
          If default, relative encoding is disabled and it is a regular transformer.
          Defaults to None.
      heads_share (bool, optional): [description]. Defaults to True.
      dropout_p (float, optional): dropout rate. Defaults to 0..
      input_length (int, optional): intput length for positional encoding. Defaults to None.
      proximal_bias (bool, optional): enable/disable proximal bias as in the paper. Defaults to False.
      proximal_init (bool, optional): enable/disable poximal init as in the paper.
          Init key and query layer weights the same. Defaults to False.
class_docstr_tok: ["Multi-head","attention","with","Relative","Positional","embedding",".","https",":","\/\/arxiv.org\/pdf\/1809.04281.pdf","It","learns","positional","embeddings","for","a","window","of","neighbours",".","For","keys","and","values",",","it","learns","different","set","of","embeddings",".","Key","embeddings","are","agregated","with","the","attention","scores","and","value","embeddings","are","aggregated","with","the","output",".","Note",":","Example","with","relative","attention","window","size","2","input","=","[","a",",","b",",","c",",","d",",","e","]","rel_attn_embeddings","=","[","e","(","t-2",")",",","e","(","t-1",")",",","e","(","t+1",")",",","e","(","t+2",")","]","So","it","learns","4","embedding","vectors","(","in","total","8",")","separately","for","key","and","value","vectors",".","Considering","the","input","c","e","(","t-2",")","corresponds","to","c","-",">","a","e","(","t-2",")","corresponds","to","c","-",">","b","e","(","t-2",")","corresponds","to","c","-",">","d","e","(","t-2",")","corresponds","to","c","-",">","e","These","embeddings","are","shared","among","different","time","steps",".","So","input","a",",","b",",","d","and","e","also","uses","the","same","embeddings",".","Embeddings","are","ignored","when","the","relative","window","is","out","of","limit","for","the","first","and","the","last","n","items",".","Args",":","channels","(","int",")",":","input","and","inner","layer","channels",".","out_channels","(","int",")",":","output","channels",".","num_heads","(","int",")",":","number","of","attention","heads",".","rel_attn_window_size","(","int",",","optional",")",":","relation","attention","window","size",".","If","4",",","for","each","time","step","next","and","previous","4","time","steps","are","attended",".","If","default",",","relative","encoding","is","disabled","and","it","is","a","regular","transformer",".","Defaults","to","None",".","heads_share","(","bool",",","optional",")",":","[","description","]",".","Defaults","to","True",".","dropout_p","(","float",",","optional",")",":","dropout","rate",".","Defaults","to","0","..","input_length","(","int",",","optional",")",":","intput","length","for","positional","encoding",".","Defaults","to","None",".","proximal_bias","(","bool",",","optional",")",":","enable\/disable","proximal","bias","as","in","the","paper",".","Defaults","to","False",".","proximal_init","(","bool",",","optional",")",":","enable\/disable","poximal","init","as","in","the","paper",".","Init","key","and","query","layer","weights","the","same",".","Defaults","to","False","."]
code: def __init__(
      self,
      channels,
      out_channels,
      num_heads,
      rel_attn_window_size=None,
      heads_share=True,
      dropout_p=0.0,
      input_length=None,
      proximal_bias=False,
      proximal_init=False,
  ):
      super().__init__()
      assert (
          channels % num_heads == 0
      ), " [!] channels should be divisible by num_heads."
      # class attributes
      self.channels = channels
      self.out_channels = out_channels
      self.num_heads = num_heads
      self.rel_attn_window_size = rel_attn_window_size
      self.heads_share = heads_share
      self.input_length = input_length
      self.proximal_bias = proximal_bias
      self.dropout_p = dropout_p
      self.attn = None
      # query, key, value layers
      self.k_channels = channels // num_heads
      self.conv_q = nn.Conv1d(channels, channels, 1)
      self.conv_k = nn.Conv1d(channels, channels, 1)
      self.conv_v = nn.Conv1d(channels, channels, 1)
      # output layers
      self.conv_o = nn.Conv1d(channels, out_channels, 1)
      self.dropout = nn.Dropout(dropout_p)
      # relative positional encoding layers
      if rel_attn_window_size is not None:
          n_heads_rel = 1 if heads_share else num_heads
          rel_stddev = self.k_channels**-0.5
          emb_rel_k = nn.Parameter(
              torch.randn(
                  n_heads_rel,
                  rel_attn_window_size * 2 + 1,
                  self.k_channels,
              )
              * rel_stddev
          )
          emb_rel_v = nn.Parameter(
              torch.randn(
                  n_heads_rel,
                  rel_attn_window_size * 2 + 1,
                  self.k_channels,
              )
              * rel_stddev
          )
          self.register_parameter("emb_rel_k", emb_rel_k)
          self.register_parameter("emb_rel_v", emb_rel_v)
  
      # init layers
      nn.init.xavier_uniform_(self.conv_q.weight)
      nn.init.xavier_uniform_(self.conv_k.weight)
      # proximal bias
      if proximal_init:
          self.conv_k.weight.data.copy_(self.conv_q.weight.data)
          self.conv_k.bias.data.copy_(self.conv_q.bias.data)
      nn.init.xavier_uniform_(self.conv_v.weight)
  
imports: import math
  import torch
  from torch import nn
  from torch.nn import functional
  from TTS.tts.layers.glow_tts.glow import LayerNorm
label_desc: Use image node_id 1 to create a new RelativePositionMultiHeadAttention object from inherited base classes: nn with example: obj = RelativePositionMultiHeadAttention(channels, out_channels, num_heads, rel_attn_window_size, heads_share, dropout_p, input_length, proximal_bias, proximal_init)