class_docstr: Rotary Embedding for language model.
  
  Args:
      kv_channels (int): Projection weights dimension in multi-head attention. Obtained from transformer config
      rotary_percent (float): Percent of rotary dimension to use for rotary position embeddings.
      seq_len_interpolation_factor (float, optional): scale of linearly interpolating RoPE for longer sequences. The value must be a float larger than 1.0. Defaults to None
      rotary_base (int, optional): Base period for rotary position embeddings. Defaults to 10000.
class_docstr_tok: ["Rotary","Embedding","for","language","model",".","Args",":","kv_channels","(","int",")",":","Projection","weights","dimension","in","multi-head","attention",".","Obtained","from","transformer","config","rotary_percent","(","float",")",":","Percent","of","rotary","dimension","to","use","for","rotary","position","embeddings",".","seq_len_interpolation_factor","(","float",",","optional",")",":","scale","of","linearly","interpolating","RoPE","for","longer","sequences",".","The","value","must","be","a","float","larger","than","1.0",".","Defaults","to","None","rotary_base","(","int",",","optional",")",":","Base","period","for","rotary","position","embeddings",".","Defaults","to","10000","."]
code: def __init__(
      self,
      kv_channels: int,
      rotary_percent: float,
      seq_len_interpolation_factor: float = None,
      rotary_base: int = 10000,
  ) -> None:
      super().__init__()
  
      dim = kv_channels
      if rotary_percent < 1.0:
          dim = int(dim * rotary_percent)
  
      self.seq_len_interpolation_factor = seq_len_interpolation_factor
      self.inv_freq = 1.0 / (
          rotary_base
          ** (
              torch.arange(
                  0,
                  dim,
                  2,
                  dtype=torch.float32,
                  device=torch.cuda.current_device(),
              )
              / dim
          )
      )
  
file_path: megatron-lm/megatron/core/models/common/embeddings/rotary_pos_embedding.py
imports: from __future__ import annotations
  from typing import TYPE_CHECKING
  import torch
  from torch import Tensor, nn
  from megatron.core import parallel_state
label_desc: Use image node_id 1 to create a new RotaryEmbedding object from inherited base classes: nn with example: obj = RotaryEmbedding(kv_channels, rotary_percent, seq_len_interpolation_factor, rotary_base)