class_docstr: The rotary position embeddings from RoFormer_ (Su et. al).
  A crucial insight from the method is that the query and keys are
  transformed by rotation matrices which depend on the relative positions.
  
  Other implementations are available in the Rotary Transformer repo_ and in
  GPT-NeoX_, GPT-NeoX was an inspiration
  
  .. _RoFormer: https://arxiv.org/abs/2104.09864
  .. _repo: https://github.com/ZhuiyiTechnology/roformer
  .. _GPT-NeoX: https://github.com/EleutherAI/gpt-neox
  
  If scale_base is not None, this implements XPos (Sun et al., https://arxiv.org/abs/2212.10554).
  A recommended value for scale_base is 512: https://github.com/HazyResearch/flash-attention/issues/96
  Reference: https://github.com/sunyt32/torchscale/blob/main/torchscale/component/xpos_relative_position.py
class_docstr_tok: ["The","rotary","position","embeddings","from","RoFormer_","(","Su","et",".","al",")",".","A","crucial","insight","from","the","method","is","that","the","query","and","keys","are","transformed","by","rotation","matrices","which","depend","on","the","relative","positions",".","Other","implementations","are","available","in","the","Rotary","Transformer","repo_","and","in","GPT-NeoX_",",","GPT-NeoX","was","an","inspiration","..","_RoFormer",":","https",":","\/\/arxiv.org\/abs\/2104.09864","..","_repo",":","https",":","\/\/github.com\/ZhuiyiTechnology\/roformer","..","_GPT-NeoX",":","https",":","\/\/github.com\/EleutherAI\/gpt-neox","If","scale_base","is","not","None",",","this","implements","XPos","(","Sun","et","al.",",","https",":","\/\/arxiv.org\/abs\/2212.10554",")",".","A","recommended","value","for","scale_base","is","512",":","https",":","\/\/github.com\/HazyResearch\/flash-attention\/issues\/96","Reference",":","https",":","\/\/github.com\/sunyt32\/torchscale\/blob\/main\/torchscale\/component\/xpos_relative_position.py"]
code: def _update_cos_sin_cache(self, seqlen, device=None, dtype=None):
      # Reset the tables if the sequence length has changed,
      # if we're on a new device (possibly due to tracing for instance),
      # or if we're switching from inference mode to training
      if (
          seqlen > self._seq_len_cached
          or self._cos_cached is None
          or self._cos_cached.device != device
          or self._cos_cached.dtype != dtype
          or (self.training and self._cos_cached.is_inference())
      ):
          self._seq_len_cached = seqlen
          # We want fp32 here, not self.inv_freq.dtype, since the model could be loaded in bf16
          # And the output of arange can be quite large, so bf16 would lose a lot of precision.
          # However, for compatibility reason, we add an option to use the dtype of self.inv_freq.
          if self.pos_idx_in_fp32:
              t = torch.arange(
                  seqlen, device=device, dtype=torch.float32
              )
              # We want fp32 here as well since inv_freq will be multiplied with t, and the output
              # will be large. Having it in bf16 will lose a lot of precision and cause the
              # cos & sin output to change significantly.
              # We want to recompute self.inv_freq if it was not loaded in fp32
              if self.inv_freq.dtype != torch.float32:
                  inv_freq = self._compute_inv_freq(device=device)
              else:
                  inv_freq = self.inv_freq
          else:
              t = torch.arange(
                  seqlen, device=device, dtype=self.inv_freq.dtype
              )
              inv_freq = self.inv_freq
          # Don't do einsum, it converts fp32 to fp16 under AMP
          # freqs = torch.einsum("i,j->ij", t, self.inv_freq)
          freqs = torch.outer(t, inv_freq)
          if self.scale is None:
              self._cos_cached = torch.cos(freqs).to(dtype)
              self._sin_cached = torch.sin(freqs).to(dtype)
          else:
              power = (
                  torch.arange(
                      seqlen,
                      dtype=self.scale.dtype,
                      device=self.scale.device,
                  )
                  - seqlen // 2
              ) / self.scale_base
              scale = self.scale.to(device=power.device) ** rearrange(
                  power, "s -> s 1"
              )
              # We want the multiplication by scale to happen in fp32
              self._cos_cached = (torch.cos(freqs) * scale).to(dtype)
              self._sin_cached = (torch.sin(freqs) * scale).to(dtype)
              self._cos_k_cached = (torch.cos(freqs) / scale).to(dtype)
              self._sin_k_cached = (torch.sin(freqs) / scale).to(dtype)
  
imports: import math
  from typing import Optional, Tuple, Union
  import torch
  from einops import rearrange, repeat
  from flash_attn.ops.triton.rotary import apply_rotary
label_desc: Use image node_id 3 for calling the RotaryEmbedding obj's underlying member method code with example usage: obj._update_cos_sin_cache(seqlen, device, dtype) without return types