class_docstr: Get the score based on the query and relative position index. This is used for implementing
   relative attention.
  
  For the multi-head attention with relative positional encoding, we have the formula
  
  .. math::
  
      out = \text{softmax}(\frac{Q K^T + R}{\sqrt{d}}) V
  
  
  Here, :math:`R` is the relative positional encoding matrix.
  
  Usually, :math:`R_{i, j}` is calculate based on the
  relative positional difference :math:`i - j`.
  
  This function aims at generating the R matrix given the query and the relative positions.
  We support the following methods:
  
  - method = 'transformer_xl'
      :math:`R_{i, j} = <Q, W S_{i - j}>`, in which :math:`S_{i, j}` is the sinusoidal embedding and
      :math:`W` is a Dense layer that maps :math:`S_{i - j}` to the same dimension as the query.
      This is proposed in paper:
  
          [ACL2019] Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
  
  - method = 'shaw'
      :math:`R_{i, j} = < Q, E_{i - j}>`, in which :math:`E_{i - j}` is the learned positional embedding
      This is proposed in paper:
  
          [NAACL2018] Self-Attention with Relative Position Representations
  
  - method = 't5'
      :math:`R_{i, j} = E_{i - j}`, in which :math:`E_{i - j}` is the bucket positional embedding.
      This is proposed in paper:
  
          [Arxiv2019] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  
  Like in MultiHeadAttentionCell, we support different layouts to cope with the query matrix.
  
  - layout="NKT"
      query: (B, K, L_q, C_k)
  - layout="NTK"
      query: (B, L_q, K, C_k)
  - layout="TNK"
      query: (L_q, B, K, C_k)
class_docstr_tok: ["Get","the","score","based","on","the","query","and","relative","position","index",".","This","is","used","for","implementing","relative","attention",".","For","the","multi-head","attention","with","relative","positional","encoding",",","we","have","the","formula","..","math",":",":","out","=","\\text","{","softmax","}","(","\\frac","{","Q","K^T","+","R","}","{","\\sqrt","{","d","}","}",")","V","Here",",",":","math",":","`","R","`","is","the","relative","positional","encoding","matrix",".","Usually",",",":","math",":","`","R_","{","i",",","j","}","`","is","calculate","based","on","the","relative","positional","difference",":","math",":","`","i","-","j","`",".","This","function","aims","at","generating","the","R","matrix","given","the","query","and","the","relative","positions",".","We","support","the","following","methods",":","-","method","=","'transformer_xl'",":","math",":","`","R_","{","i",",","j","}","=","<","Q",",","W","S_","{","i","-","j","}",">","`",",","in","which",":","math",":","`","S_","{","i",",","j","}","`","is","the","sinusoidal","embedding","and",":","math",":","`","W","`","is","a","Dense","layer","that","maps",":","math",":","`","S_","{","i","-","j","}","`","to","the","same","dimension","as","the","query",".","This","is","proposed","in","paper",":","[","ACL2019","]","Transformer-XL",":","Attentive","Language","Models","Beyond","a","Fixed-Length","Context","-","method","=","'shaw'",":","math",":","`","R_","{","i",",","j","}","=","<","Q",",","E_","{","i","-","j","}",">","`",",","in","which",":","math",":","`","E_","{","i","-","j","}","`","is","the","learned","positional","embedding","This","is","proposed","in","paper",":","[","NAACL2018","]","Self-Attention","with","Relative","Position","Representations","-","method","=","'t5'",":","math",":","`","R_","{","i",",","j","}","=","E_","{","i","-","j","}","`",",","in","which",":","math",":","`","E_","{","i","-","j","}","`","is","the","bucket","positional","embedding",".","This","is","proposed","in","paper",":","[","Arxiv2019","]","Exploring","the","Limits","of","Transfer","Learning","with","a","Unified","Text-to-Text","Transformer","Like","in","MultiHeadAttentionCell",",","we","support","different","layouts","to","cope","with","the","query","matrix",".","-","layout=","''","NKT","''","query",":","(","B",",","K",",","L_q",",","C_k",")","-","layout=","''","NTK","''","query",":","(","B",",","L_q",",","K",",","C_k",")","-","layout=","''","TNK","''","query",":","(","L_q",",","B",",","K",",","C_k",")"]
docstr: Layout of the cell
docstr_tok: ["Layout","of","the","cell"]
code: def layout(self) -> str:
      """Layout of the cell"""
      return self._layout
  
imports: import math
  import mxnet
  from mxnet import np, npx
  import numpy
  from mxnet.gluon.block import HybridBlock
  from mxnet.gluon import nn
  from .op import l2_normalize
  from .layers import SinusoidalPositionalEmbedding, BucketPositionalEmbedding, LearnedPositionalEmbedding
  from typing import Optional
label_desc: Use image node_id 2 for calling the RelAttentionScoreCell obj's underlying member method code with example usage: obj.layout() and returns: self