class_docstr: A multi-head masked self-attention dispatch mechanism, with a projection at the end,
  following the architecture proposed in `Attention is all you need`_, Vaswani et al.
  
  The actual attention mechanism can vary, as well as the projections.
  This can be used to wrap the proposed attention mechanisms and make them multi-head aware,
  but it is optional.
  
  Args:
      dim_model: The model/embedding dimension
      num_heads: The number of heads being used
      attention: The attention mechanism (needs to be registered to the xformers library)
      bias: Whether to use bias for the projections : (Q, K, V, Output)
      residual_dropout: Amount of dropout on the residual path
      use_separate_proj_weight: Use different weights for the Q, K, V projections
      dim_key: Optionally use a different dimension for the key
      dim_value:  Optionally use a different dimension for the value
      in_proj_container: Optionally provide the input projection module
      use_rotary_embeddings: Use rotary embeddings
      out_proj: Optionally provide the output projection module
  
  
  .. _`Attention is all you need`: https://arxiv.org/abs/1706.03762v5
class_docstr_tok: ["A","multi-head","masked","self-attention","dispatch","mechanism",",","with","a","projection","at","the","end",",","following","the","architecture","proposed","in","`","Attention","is","all","you","need","`","_",",","Vaswani","et","al",".","The","actual","attention","mechanism","can","vary",",","as","well","as","the","projections",".","This","can","be","used","to","wrap","the","proposed","attention","mechanisms","and","make","them","multi-head","aware",",","but","it","is","optional",".","Args",":","dim_model",":","The","model\/embedding","dimension","num_heads",":","The","number","of","heads","being","used","attention",":","The","attention","mechanism","(","needs","to","be","registered","to","the","xformers","library",")","bias",":","Whether","to","use","bias","for","the","projections",":","(","Q",",","K",",","V",",","Output",")","residual_dropout",":","Amount","of","dropout","on","the","residual","path","use_separate_proj_weight",":","Use","different","weights","for","the","Q",",","K",",","V","projections","dim_key",":","Optionally","use","a","different","dimension","for","the","key","dim_value",":","Optionally","use","a","different","dimension","for","the","value","in_proj_container",":","Optionally","provide","the","input","projection","module","use_rotary_embeddings",":","Use","rotary","embeddings","out_proj",":","Optionally","provide","the","output","projection","module","..","_","`","Attention","is","all","you","need","`",":","https",":","\/\/arxiv.org\/abs\/1706.03762v5"]
code: def from_config(cls, config: MultiHeadDispatchConfig):
      # Generate the class inputs from the config
      fields = asdict(config)
  
      # Skip all Nones so that default values are used
      fields = {k: v for k, v in fields.items() if v is not None}
  
      return cls(**fields)
  
file_path: xformers/xformers/components/multi_head_dispatch.py
imports: import logging
  from dataclasses import asdict, dataclass
  from typing import Optional, Tuple
  import torch
  import torch.nn
  from torch.nn.init import constant_
  from xformers.components.attention import Attention
  from xformers.components.input_projection import InputProjection, InputProjectionConfig
  from xformers.components.positional_embedding import RotaryEmbedding
label_desc: Use image node_id 3 for calling the MultiHeadDispatch obj's underlying member method code with example usage: obj.from_config(cls, config) and returns: cls