class_docstr: Lunar Causal attention.
  See "Linformer: Self-Attention with Linear Complexity" for more details.
code: def __init__(
      self,
      embed_dim,
      num_heads,
      dropout=0.0,
      bias=True,
      tie_kv=True,
      parallel=True,
  ):
      super().__init__()
      self.embed_dim = embed_dim
      self.num_heads = num_heads
      self.parallel = parallel
      self.dropout_module = FairseqDropout(
          dropout, module_name=self.__class__.__name__
      )
  
      self.head_dim = embed_dim // num_heads
      assert (
          self.head_dim * num_heads == self.embed_dim
      ), "embed_dim must be divisible by num_heads"
      self.scaling = self.head_dim**-0.5
  
      self.pq_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
      self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
      self.pc_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
      if tie_kv:
          self.c_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
          self.k_proj = self.v_proj = None
      else:
          self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
          self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
          self.c_proj = None
  
      self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
      self.reset_parameters()
  
      self.onnx_trace = False
      self.tpu = False
  
imports: import math
  from typing import Dict, Optional, Tuple, Union
  import torch
  import torch.nn.functional
  from fairseq import utils
  from fairseq.incremental_decoding_utils import with_incremental_state
  from fairseq.modules.fairseq_dropout import FairseqDropout
  from torch import nn, Tensor
label_desc: Use image node_id 1 to create a new LunarCausalAttention object from inherited base classes: nn with example: obj = LunarCausalAttention(embed_dim, num_heads, dropout, bias, tie_kv, parallel)