class_docstr: Parallel attention and feedforward with residual discovered by Wang et al + EleutherAI from GPT-J fame.
  
  Args:
      dim (`int`):
          Dimension of features.
      dim_head (`int`):
          Head of fused dimensions.
      heads (`int`):
          Number of heads.
      ff_mult (`int`):
          Multiple number for feedforward.
class_docstr_tok: ["Parallel","attention","and","feedforward","with","residual","discovered","by","Wang","et","al","+","EleutherAI","from","GPT-J","fame",".","Args",":","dim","(","`","int","`",")",":","Dimension","of","features",".","dim_head","(","`","int","`",")",":","Head","of","fused","dimensions",".","heads","(","`","int","`",")",":","Number","of","heads",".","ff_mult","(","`","int","`",")",":","Multiple","number","for","feedforward","."]
code: def __init__(self, dim, dim_head=64, heads=8, ff_mult=4):
      super().__init__()
      self.norm = LayerNorm(dim)
  
      attn_inner_dim = dim_head * heads
      ff_inner_dim = dim * ff_mult
      self.fused_dims = (
          attn_inner_dim,
          dim_head,
          dim_head,
          (ff_inner_dim * 2),
      )
  
      self.heads = heads
      self.scale = dim_head**-0.5
      self.rotary_emb = RotaryEmbedding(dim_head)
  
      self.fused_attn_ff_proj = nn.Linear(
          dim, sum(self.fused_dims), bias=False
      )
      self.attn_out = nn.Linear(attn_inner_dim, dim, bias=False)
  
      self.ff_out = nn.Sequential(
          SwiGLU(), nn.Linear(ff_inner_dim, dim, bias=False)
      )
  
      # for caching causal mask and rotary embeddings
  
      self.register_buffer("mask", None, persistent=False)
      self.register_buffer("pos_emb", None, persistent=False)
  
file_path: towhee/towhee/models/coca/coca.py
imports: import torch
  from torch import einsum, nn
  import torch.nn.functional
  from einops import rearrange, repeat
  from towhee.models.layers.cross_attention import LayerNorm, Residual, RotaryEmbedding, apply_rotary_pos_emb, CrossAttention
  from towhee.models.layers.activations.swiglu import SwiGLU
label_desc: Use image node_id 1 to create a new ParallelTransformerBlock object from inherited base classes: nn with example: obj = ParallelTransformerBlock(dim, dim_head, heads, ff_mult)