class_docstr: Cross-Covariance Attention (XCA).
  
  This is a redistirbution of cross-covariance implementation originally
  released at https://github.com/facebookresearch/xcit.
  
  Cross-covariance attention was introduced in the following paper:
  
  XCiT: Cross-Covariance Image Transformers
  A El-Nouby, H Touvron, M Caron, P Bojanowski, M Douze, A Joulin, I Laptev,
  N Neverova, G Synnaeve, J Verbeek, H Jegou
  
  Operation where the channels are updated using a weighted sum. The weights
  are obtained from the (softmax normalized) Cross-covariance matrix
  (Q^T \cdot K \in d_h \times d_h)/
  
  Args:
      dim: Dimension of input.
      num_heads: Number of attention heads.
      qkv_bias: Whether to use bias in qkv term.
      attn_drop: Attention dropout probability.
      proj_drop: Projection dropout probability.
code: def __init__(
      self,
      dim: int,
      num_heads: int = 8,
      qkv_bias: bool = False,
      attn_drop: float = 0.0,
      proj_drop: float = 0.0,
  ):
      super().__init__()
      self.num_heads = num_heads
      self.temperature = nn.Parameter(torch.ones(num_heads, 1, 1))
      self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
      self.attn_drop = nn.Dropout(attn_drop)
      self.proj = nn.Linear(dim, dim)
      self.proj_drop = nn.Dropout(proj_drop)
  
file_path: NeuralCompression/neuralcompression/layers/_xca.py
imports: import torch
  import torch.nn
  import torch.nn.functional
  from torch import Tensor
label_desc: Use image node_id 1 to create a new XCA object from inherited base classes: nn with example: obj = XCA(dim, num_heads, qkv_bias, attn_drop, proj_drop)