class_docstr: Cross-Covariance Attention (XCA).
  
  This is a redistirbution of cross-covariance implementation originally
  released at https://github.com/facebookresearch/xcit.
  
  Cross-covariance attention was introduced in the following paper:
  
  XCiT: Cross-Covariance Image Transformers
  A El-Nouby, H Touvron, M Caron, P Bojanowski, M Douze, A Joulin, I Laptev,
  N Neverova, G Synnaeve, J Verbeek, H Jegou
  
  Operation where the channels are updated using a weighted sum. The weights
  are obtained from the (softmax normalized) Cross-covariance matrix
  (Q^T \cdot K \in d_h \times d_h)/
  
  Args:
      dim: Dimension of input.
      num_heads: Number of attention heads.
      qkv_bias: Whether to use bias in qkv term.
      attn_drop: Attention dropout probability.
      proj_drop: Projection dropout probability.
code: def forward(self, x: Tensor) -> Tensor:
      B, N, C = x.shape
      # Result of next line is (qkv, B, num (H)eads,  (C')hannels per head, N)
      qkv: Tensor = (
          self.qkv(x)
          .reshape(B, N, 3, self.num_heads, C // self.num_heads)
          .permute(2, 0, 3, 4, 1)
      )
      q, k, v = qkv.unbind(
          0
      )  # make torchscript happy (cannot use tensor as tuple)
  
      # Paper section 3.2 l2-Normalization and temperature scaling
      q = F.normalize(q, dim=-1)
      k = F.normalize(k, dim=-1)
      attn = (q @ k.transpose(-2, -1)) * self.temperature
      attn = attn.softmax(dim=-1)
      attn = self.attn_drop(attn)
  
      # (B, H, C', N), permute -> (B, N, H, C')
      x = (attn @ v).permute(0, 3, 1, 2).reshape(B, N, C)
      x = self.proj(x)
      x = self.proj_drop(x)
  
      return x
  
file_path: NeuralCompression/neuralcompression/layers/_xca.py
imports: import torch
  import torch.nn
  import torch.nn.functional
  from torch import Tensor
label_desc: Use image node_id 2 for calling the XCA obj's underlying member method code with example usage: obj.forward(x) and returns: x