docstr: Linformer attention mechanism,
  from `Linformer: Self-Attention with Linear Complexity`_, Wang et al (2020).
  The original notation is kept as is.
  
  .. _`Linformer: Self-Attention with Linear Complexity` : https://arxiv.org/abs/2006.04768v2
docstr_tok: ["Linformer","attention","mechanism",",","from","`","Linformer",":","Self-Attention","with","Linear","Complexity","`","_",",","Wang","et","al","(","2020",")",".","The","original","notation","is","kept","as","is",".","..","_","`","Linformer",":","Self-Attention","with","Linear","Complexity","`",":","https",":","\/\/arxiv.org\/abs\/2006.04768v2"]
code: def __init__(
      self,
      dropout: float,
      seq_len: int,
      k: Optional[int] = None,
      *args,
      **kwargs,
  ):
      """
      Linformer attention mechanism,
      from `Linformer: Self-Attention with Linear Complexity`_, Wang et al (2020).
      The original notation is kept as is.
  
      .. _`Linformer: Self-Attention with Linear Complexity` : https://arxiv.org/abs/2006.04768v2
      """
      super().__init__()
  
      if k is None:
          k = seq_len // 4
  
      self.k = k
      self.E = nn.Linear(seq_len, k, bias=False)
      self.F = nn.Linear(seq_len, k, bias=False)
      self.attn_drop = nn.Dropout(dropout, inplace=False)
      self.seq_len = seq_len
  
      # MHA related flags:
      # kq need to have the same dimension
      self.requires_same_k_q_dimensions = True
  
      # This attention does not support attention masks
      self.supports_attention_mask = False
  
file_path: xformers/xformers/components/attention/linformer.py
imports: from dataclasses import dataclass
  from typing import Optional
  import torch
  import torch.nn
  from xformers.components.attention import Attention, AttentionConfig, register_attention
  from xformers.components.attention.core import scaled_dot_product_attention
label_desc: Use image node_id 1 to create a new LinformerAttention object from inherited base classes: Attention with example: obj = LinformerAttention(dropout, seq_len, k)