class_docstr: Multi-headed attention from 'Attention Is All You Need' paper
code: def __init__(
      self,
      hidden_size=128,
      num_heads=4,
      max_position_embeddings=1024,
      bias=False,
  ):
      super().__init__()
      self.hidden_size = hidden_size
      self.num_heads = num_heads
      self.head_dim = self.hidden_size // self.num_heads
      self.max_position_embeddings = max_position_embeddings
      self.rope_theta = 10000.0
      self.q_proj = nn.Linear(
          self.hidden_size, self.num_heads * self.head_dim, bias=bias
      )
      self.k_proj = nn.Linear(
          self.hidden_size, self.num_heads * self.head_dim, bias=bias
      )
      self.v_proj = nn.Linear(
          self.hidden_size, self.num_heads * self.head_dim, bias=bias
      )
      self.o_proj = nn.Linear(
          self.num_heads * self.head_dim, self.hidden_size, bias=bias
      )
      self.rotary_emb = RotaryEmbedding(
          self.head_dim,
          max_position_embeddings=self.max_position_embeddings,
          base=self.rope_theta,
      )
  
imports: import math
  from typing import Optional
  import pytest
  import torch
  import torch.utils.checkpoint
  from helpers import assert_similar, random_tensor
  from torch import nn
  from quanto import Calibration, quantize
label_desc: Use image node_id 1 to create a new Attention object from inherited base classes: nn with example: obj = Attention(hidden_size, num_heads, max_position_embeddings, bias)