class_docstr: FT-Transformer for categorical tabular features.
  The input dimension is automatically computed based on
  the number of categories in each categorical column.
class_docstr_tok: ["FT-Transformer","for","categorical","tabular","features",".","The","input","dimension","is","automatically","computed","based","on","the","number","of","categories","in","each","categorical","column","."]
docstr: Parameters
  ----------
  prefix
      The model prefix.
  num_categories
      A list of integers. Each one is the number of categories in one categorical column.
  token_dim
      The size of one token after categorical/numerical tokenizers.
  hidden_size
      The embedding dimension of the transformer backbone.
  out_features
      Dimension of output features.
  num_classes
      Number of classes. 1 for a regression task.
  token_bias
      If `True`, for each feature, an additional trainable vector will be added in `_CategoricalFeatureTokenizer`
      to the embedding regardless of feature value. Notably, the bias are not shared between features.
  token_initialization
      Initialization policy for parameters in `_CategoricalFeatureTokenizer` and `_CLSToke`.
      Must be one of `['uniform', 'normal']`.
  num_blocks
      Number of the `FT_Transformer` blocks, which should be non-negative.
  attention_n_heads
      Number of attention heads in each `FT_Transformer` block, which should be positive.
  attention_initialization
      Weights initialization scheme for Multi Headed Attention module.
  attention_dropout
      Dropout ratio for the Multi Headed Attention module.
  residual_dropout
      Dropout ratio for the linear layers in FT_Transformer block.
  ffn_activation
      Activation function type for the Feed-Forward Network module.
  ffn_normalization
      Normalization scheme of the Feed-Forward Network module.
  ffn_hidden_size
      Number of the hidden nodes of the linear layers in the Feed-Forward Network module.
  ffn_dropout
      Dropout ratio of the hidden nodes of the linear layers in the Feed-Forward Network module.
  prenormalization, first_prenormalization
      Prenormalization to stabilize the training.
  kv_compression_ratio
      The compression ration to reduce the input sequence length.
  kv_compression_sharing
      If `true` the projections will share weights.
  head_activation
      Activation function type of the MLP layer.
  head_normalization
      Normalization scheme of the MLP layer.
  additive_attention
      If 'true' the transformer will use additive attention with linear complexity to sequence length.
  share_qv_weights
      if 'true', then value and query transformation parameters are shared in additive attention.
  pooling_mode
      The pooling mode for the Transformer. Can be "cls", or "mean"
  
  References
  ----------
  1. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, Artem Babenko,
  "Revisiting Deep Learning Models for Tabular Data", 2021
  https://arxiv.org/pdf/2106.11959.pdf
  2. Code: https://github.com/Yura52/tabular-dl-revisiting-models
code: def __init__(
      self,
      prefix: str,
      num_numerical_columns: int,
      num_categories: List[int],
      embedding_arch: List[str],
      token_dim: int,
      hidden_size: Optional[int] = 192,
      hidden_features: Optional[int] = 192,
      num_classes: Optional[int] = 0,
      token_bias: Optional[bool] = True,
      token_initialization: Optional[str] = "normal",
      num_blocks: Optional[int] = 0,
      attention_n_heads: Optional[int] = 8,
      attention_initialization: Optional[str] = "kaiming",
      attention_normalization: Optional[str] = "layer_norm",
      attention_dropout: Optional[str] = 0.2,
      residual_dropout: Optional[str] = 0.0,
      ffn_activation: Optional[str] = "reglu",
      ffn_normalization: Optional[str] = "layer_norm",
      ffn_hidden_size: Optional[str] = 6,
      ffn_dropout: Optional[str] = 0.0,
      prenormalization: Optional[bool] = True,
      first_prenormalization: Optional[bool] = False,
      kv_compression_ratio: Optional[float] = None,
      kv_compression_sharing: Optional[str] = None,
      head_activation: Optional[str] = "relu",
      head_normalization: Optional[str] = "layer_norm",
      additive_attention: Optional[bool] = False,
      share_qv_weights: Optional[bool] = False,
      pooling_mode: Optional[str] = "cls",
  ) -> None:
      """
      Parameters
      ----------
      prefix
          The model prefix.
      num_categories
          A list of integers. Each one is the number of categories in one categorical column.
      token_dim
          The size of one token after categorical/numerical tokenizers.
      hidden_size
          The embedding dimension of the transformer backbone.
      out_features
          Dimension of output features.
      num_classes
          Number of classes. 1 for a regression task.
      token_bias
          If `True`, for each feature, an additional trainable vector will be added in `_CategoricalFeatureTokenizer`
          to the embedding regardless of feature value. Notably, the bias are not shared between features.
      token_initialization
          Initialization policy for parameters in `_CategoricalFeatureTokenizer` and `_CLSToke`.
          Must be one of `['uniform', 'normal']`.
      num_blocks
          Number of the `FT_Transformer` blocks, which should be non-negative.
      attention_n_heads
          Number of attention heads in each `FT_Transformer` block, which should be positive.
      attention_initialization
          Weights initialization scheme for Multi Headed Attention module.
      attention_dropout
          Dropout ratio for the Multi Headed Attention module.
      residual_dropout
          Dropout ratio for the linear layers in FT_Transformer block.
      ffn_activation
          Activation function type for the Feed-Forward Network module.
      ffn_normalization
          Normalization scheme of the Feed-Forward Network module.
      ffn_hidden_size
          Number of the hidden nodes of the linear layers in the Feed-Forward Network module.
      ffn_dropout
          Dropout ratio of the hidden nodes of the linear layers in the Feed-Forward Network module.
      prenormalization, first_prenormalization
          Prenormalization to stabilize the training.
      kv_compression_ratio
          The compression ration to reduce the input sequence length.
      kv_compression_sharing
          If `true` the projections will share weights.
      head_activation
          Activation function type of the MLP layer.
      head_normalization
          Normalization scheme of the MLP layer.
      additive_attention
          If 'true' the transformer will use additive attention with linear complexity to sequence length.
      share_qv_weights
          if 'true', then value and query transformation parameters are shared in additive attention.
      pooling_mode
          The pooling mode for the Transformer. Can be "cls", or "mean"
  
      References
      ----------
      1. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, Artem Babenko,
      "Revisiting Deep Learning Models for Tabular Data", 2021
      https://arxiv.org/pdf/2106.11959.pdf
      2. Code: https://github.com/Yura52/tabular-dl-revisiting-models
      """
  
      super().__init__()
  
      assert (
          num_categories or num_numerical_columns > 0
      ), "there must be categorical columns or numerical columns"
      assert token_dim > 0, "d_token must be positive"
      assert num_blocks >= 0, "n_blocks must be non-negative"
      assert attention_n_heads > 0, "attention_n_heads must be positive"
      assert token_initialization in [
          "uniform",
          "normal",
      ], "initialization must be uniform or normal"
  
      self.prefix = prefix
      self.out_features = hidden_size
      self.pooling_mode = pooling_mode
  
      self.categorical_feature_tokenizer = None
      self.numerical_feature_tokenizer = None
  
      if num_categories:
          self.num_categories = num_categories
          self.categorical_feature_tokenizer = (
              CategoricalFeatureTokenizer(
                  num_categories=num_categories,
                  d_token=token_dim,
                  bias=token_bias,
                  initialization=token_initialization,
              )
          )
          self.categorical_adapter = nn.Linear(token_dim, hidden_size)
  
      if num_numerical_columns > 0:
          self.numerical_feature_tokenizer = NumEmbeddings(
              in_features=num_numerical_columns,
              d_embedding=token_dim,
              embedding_arch=embedding_arch,
          )
          self.numerical_adapter = nn.Linear(token_dim, hidden_size)
  
      self.transformer = Custom_Transformer(
          d_token=hidden_size,
          n_blocks=num_blocks,
          attention_n_heads=attention_n_heads,
          attention_dropout=attention_dropout,
          attention_initialization=attention_initialization,
          attention_normalization=attention_normalization,
          ffn_d_hidden=ffn_hidden_size,
          ffn_dropout=ffn_dropout,
          ffn_activation=ffn_activation,
          ffn_normalization=ffn_normalization,
          residual_dropout=residual_dropout,
          prenormalization=prenormalization,
          first_prenormalization=first_prenormalization,
          last_layer_query_idx=None,
          n_tokens=None,
          kv_compression_ratio=kv_compression_ratio,
          kv_compression_sharing=kv_compression_sharing,
          head_activation=head_activation,
          head_normalization=head_normalization,
          d_out=hidden_features,
          projection=False,
          additive_attention=additive_attention,
          share_qv_weights=share_qv_weights,
      )
  
      self.head = Custom_Transformer.Head(
          d_in=hidden_size,
          d_out=num_classes,
          bias=True,
          activation=head_activation,
          normalization=head_normalization,
      )
  
      self.cls_token = CLSToken(
          d_token=hidden_size,
          initialization="uniform",
      )
  
      # init weights
      if self.numerical_feature_tokenizer:
          self.numerical_adapter.apply(init_weights)
      if self.categorical_feature_tokenizer:
          self.categorical_adapter.apply(init_weights)
      self.head.apply(init_weights)
      self.name_to_id = self.get_layer_ids()
  
imports: from typing import List, Optional
  import torch
  from torch import Tensor, nn
  from ..constants import CATEGORICAL, FEATURES, LABEL, LOGITS, NUMERICAL
  from .custom_transformer import CLSToken, Custom_Transformer, _TokenInitialization
  from .utils import init_weights
label_desc: Use image node_id 1 to create a new FT_Transformer object from inherited base classes: nn with example: obj = FT_Transformer(prefix, num_numerical_columns, num_categories, embedding_arch, token_dim, hidden_size, hidden_features, num_classes, token_bias, token_initialization, num_blocks, attention_n_heads, attention_initialization, attention_normalization, attention_dropout, residual_dropout, ffn_activation, ffn_normalization, ffn_hidden_size, ffn_dropout, prenormalization, first_prenormalization, kv_compression_ratio, kv_compression_sharing, head_activation, head_normalization, additive_attention, share_qv_weights, pooling_mode)