docstr: Large kernel attention mechanism, as proposed in `Visual Attention Network`_, Guo et al (2022).
  The original notation is tentatively kept as is. See https://github.com/Visual-Attention-Network
  for the reference implementation
  
  .. Note: compared to the paper, this block contains the LKA (Large Kernel Attention)
      and the prior and posterior transformations (Conv2d and activation)
  
  .. _`Visual Attention Network` : https://arxiv.org/pdf/2202.09741.pdf
docstr_tok: ["Large","kernel","attention","mechanism",",","as","proposed","in","`","Visual","Attention","Network","`","_",",","Guo","et","al","(","2022",")",".","The","original","notation","is","tentatively","kept","as","is",".","See","https",":","\/\/github.com\/Visual-Attention-Network","for","the","reference","implementation","..","Note",":","compared","to","the","paper",",","this","block","contains","the","LKA","(","Large","Kernel","Attention",")","and","the","prior","and","posterior","transformations","(","Conv2d","and","activation",")","..","_","`","Visual","Attention","Network","`",":","https",":","\/\/arxiv.org\/pdf\/2202.09741.pdf"]
code: def __init__(
      self,
      dim_model: int,
      *_,
      **__,
  ):
      """
      Large kernel attention mechanism, as proposed in `Visual Attention Network`_, Guo et al (2022).
      The original notation is tentatively kept as is. See https://github.com/Visual-Attention-Network
      for the reference implementation
  
      .. Note: compared to the paper, this block contains the LKA (Large Kernel Attention)
          and the prior and posterior transformations (Conv2d and activation)
  
      .. _`Visual Attention Network` : https://arxiv.org/pdf/2202.09741.pdf
      """
      super().__init__()
  
      self.block = nn.Sequential(
          nn.Conv2d(dim_model, dim_model, 1),
          nn.GELU(),
          LKA(dim_model),
          nn.Conv2d(dim_model, dim_model, 1),
      )
  
      # MHA related flags:
      self.requires_same_k_q_dimensions = (
          True  # This mechanism only really supports self attention
      )
      self.supports_attention_mask = False
      self.requires_skip_multi_head = True  # This mechanism skips the multihead attention altogether
      self.requires_squared_context = True  # Recovering the 2D structure from context assumes squared content
  
      self.requires_input_projection = False  # This mechanism does not require that the MHA projects inputs
  
file_path: xformers/xformers/components/attention/visual.py
imports: import math
  from dataclasses import dataclass
  import torch
  import torch.nn
  from xformers.components.attention import Attention, AttentionConfig, register_attention
label_desc: Use image node_id 1 to create a new Visual object from inherited base classes: Attention with example: obj = Visual(dim_model)