class_docstr: Compute relative multi-head attention for XLNet encoder.
  
  This module computes relative multi-head attention as explained in
  `Transformer-XL, (Zihang et. al)` and in `XLNet (Zhiling et. al)`.
  
  Args:
      r_r_bias: A tensor of shape `(num_heads, head_dim)`.
          The bias value added to query head while computing position based
          attention score.
  
      r_w_bias: A tensor of shape `(num_heads, head_dim)`.
          The bias value added to query head while computing content based
          attention score.
  
      r_s_bias (optional): A tensor of shape `(num_heads, head_dim)`.
          The bias value added to query head while computing segment based
          attention score.
  
      segment_embed (optional): A tensor of shape `(2, num_heads, head_dim)`
          if use_segments is True. Otherwise, this is set to None.
  
      hparams (dict or HParams, optional): Hyperparameters. Missing
          hyperparameter will be set to default values. See
          :meth:`default_hparams` for the hyperparameter sturcture
          and default values.
class_docstr_tok: ["Compute","relative","multi-head","attention","for","XLNet","encoder",".","This","module","computes","relative","multi-head","attention","as","explained","in","`","Transformer-XL",",","(","Zihang","et",".","al",")","`","and","in","`","XLNet","(","Zhiling","et",".","al",")","`",".","Args",":","r_r_bias",":","A","tensor","of","shape","`","(","num_heads",",","head_dim",")","`",".","The","bias","value","added","to","query","head","while","computing","position","based","attention","score",".","r_w_bias",":","A","tensor","of","shape","`","(","num_heads",",","head_dim",")","`",".","The","bias","value","added","to","query","head","while","computing","content","based","attention","score",".","r_s_bias","(","optional",")",":","A","tensor","of","shape","`","(","num_heads",",","head_dim",")","`",".","The","bias","value","added","to","query","head","while","computing","segment","based","attention","score",".","segment_embed","(","optional",")",":","A","tensor","of","shape","`","(","2",",","num_heads",",","head_dim",")","`","if","use_segments","is","True",".","Otherwise",",","this","is","set","to","None",".","hparams","(","dict","or","HParams",",","optional",")",":","Hyperparameters",".","Missing","hyperparameter","will","be","set","to","default","values",".","See",":","meth",":","`","default_hparams","`","for","the","hyperparameter","sturcture","and","default","values","."]
code: def __init__(
      self,
      r_r_bias,
      r_w_bias,
      r_s_bias=None,
      segment_embed=None,
      hparams=None,
  ):
      ModuleBase.__init__(self, hparams=hparams)
  
      self.num_heads = self._hparams.num_heads
      self.head_dim = self._hparams.head_dim
      hidden_dim = self._hparams.hidden_dim
  
      with tf.variable_scope(self.variable_scope):
          if self._hparams.initializer:
              tf.get_variable_scope().set_initializer(
                  layers.get_initializer(self._hparams.initializer)
              )
  
          # Official implementation creates these head variables.
          # If we create dense layers instead, there would be dimension
          # mismatch while loading the tensors
          # TODO(avinash) : Can we reshape tensors while loading the ckpt?
          self.q_head = tf.get_variable(
              "q/kernel", [hidden_dim, self.num_heads, self.head_dim]
          )
  
          self.k_head = tf.get_variable(
              "k/kernel", [hidden_dim, self.num_heads, self.head_dim]
          )
  
          self.v_head = tf.get_variable(
              "v/kernel", [hidden_dim, self.num_heads, self.head_dim]
          )
  
          self.k_head_r = tf.get_variable(
              "r/kernel", [hidden_dim, self.num_heads, self.head_dim]
          )
  
          self.dropout = layers.get_layer(
              hparams={
                  "type": "Dropout",
                  "kwargs": {"rate": self._hparams.dropout},
              }
          )
  
          self.dropout_attn = layers.get_layer(
              hparams={
                  "type": "Dropout",
                  "kwargs": {"rate": self._hparams.attention_dropout},
              }
          )
  
          self.output_projection = tf.get_variable(
              "o/kernel", [hidden_dim, self.num_heads, self.head_dim]
          )
  
          self.r_r_bias = r_r_bias
          self.r_w_bias = r_w_bias
  
          if self._hparams.use_segments:
              self.segment_embed = segment_embed
              self.r_s_bias = r_s_bias
  
          self.scale = 1 / (self.head_dim**0.5)
  
imports: import tensorflow
  from texar.tf.core import layers
  from texar.tf.module_base import ModuleBase
  from texar.tf.utils.mode import is_train_mode
label_desc: Use image node_id 1 to create a new RelativeMutiheadAttention object from inherited base classes: ModuleBase with example: obj = RelativeMutiheadAttention(r_r_bias, r_w_bias, r_s_bias, segment_embed, hparams)