docstr: Temporal Fusion Transformers (TFT) for Interpretable Time Series Forecasting.
  
  This is an implementation of the TFT architecture, as outlined in [1]_.
  
  The internal sub models are adopted from `pytorch-forecasting's TemporalFusionTransformer
  <https://pytorch-forecasting.readthedocs.io/en/latest/models.html>`_ implementation.
  
  This model supports past covariates (known for `input_chunk_length` points before prediction time),
  future covariates (known for `output_chunk_length` points after prediction time), static covariates,
  as well as probabilistic forecasting.
  
  The TFT applies multi-head attention queries on future inputs from mandatory ``future_covariates``.
  Specifying future encoders with ``add_encoders`` (read below) can automatically generate future covariates
  and allows to use the model without having to pass any ``future_covariates`` to :func:`fit()` and
  :func:`predict()`.
  
  By default, this model uses the ``QuantileRegression`` likelihood, which means that its forecasts are
  probabilistic; it is recommended to call :func`predict()` with ``num_samples >> 1`` to get meaningful results.
  
  Parameters
  ----------
  input_chunk_length
      Number of time steps in the past to take as a model input (per chunk). Applies to the target
      series, and past and/or future covariates (if the model supports it).
      Also called: Encoder length
  output_chunk_length
      Number of time steps predicted at once (per chunk) by the internal model. Also, the number of future values
      from future covariates to use as a model input (if the model supports future covariates). It is not the same
      as forecast horizon `n` used in `predict()`, which is the desired number of prediction points generated
      using either a one-shot- or auto-regressive forecast. Setting `n <= output_chunk_length` prevents
      auto-regression. This is useful when the covariates don't extend far enough into the future, or to prohibit
      the model from using future values of past and / or future covariates for prediction (depending on the
      model's covariate support).
      Also called: Decoder length
  hidden_size
      Hidden state size of the TFT. It is the main hyper-parameter and common across the internal TFT
      architecture.
  lstm_layers
      Number of layers for the Long Short Term Memory (LSTM) Encoder and Decoder (1 is a good default).
  num_attention_heads
      Number of attention heads (4 is a good default)
  full_attention
      If ``False``, only attends to previous time steps in the decoder. If ``True`` attends to previous,
      current, and future time steps. Defaults to ``False``.
  feed_forward
      A feedforward network is a fully-connected layer with an activation. TFT Can be one of the glu variant's
      FeedForward Network (FFN)[2]. The glu variant's FeedForward Network are a series of FFNs designed to work
      better with Transformer based models. Defaults to ``"GatedResidualNetwork"``. ["GLU", "Bilinear", "ReGLU",
      "GEGLU", "SwiGLU", "ReLU", "GELU"] or the TFT original FeedForward Network ["GatedResidualNetwork"].
  dropout
      Fraction of neurons affected by dropout. This is compatible with Monte Carlo dropout
      at inference time for model uncertainty estimation (enabled with ``mc_dropout=True`` at
      prediction time).
  hidden_continuous_size
      Default for hidden size for processing continuous variables
  categorical_embedding_sizes
      A dictionary used to construct embeddings for categorical static covariates. The keys are the column names
      of the categorical static covariates. Each value is either a single integer or a tuple of integers.
      For a single integer give the number of unique categories (n) of the corresponding variable. For example
      ``{"some_column": 64}``. The embedding size will be automatically determined by
      ``min(round(1.6 * n**0.56), 100)``.
      For a tuple of integers, give (number of unique categories, embedding size). For example
      ``{"some_column": (64, 8)}``.
      Note that ``TorchForecastingModels`` only support numeric data. Consider transforming/encoding your data
      with `darts.dataprocessing.transformers.static_covariates_transformer.StaticCovariatesTransformer`.
  add_relative_index
      Whether to add positional values to future covariates. Defaults to ``False``.
      This allows to use the TFTModel without having to pass future_covariates to :func:`fit()` and
      :func:`train()`. It gives a value to the position of each step from input and output chunk relative
      to the prediction point. The values are normalized with ``input_chunk_length``.
  loss_fn: nn.Module
      PyTorch loss function used for training. By default, the TFT model is probabilistic and uses a
      ``likelihood`` instead (``QuantileRegression``). To make the model deterministic, you can set the `
      `likelihood`` to None and give a ``loss_fn`` argument.
  likelihood
      The likelihood model to be used for probabilistic forecasts. By default, the TFT uses
      a ``QuantileRegression`` likelihood.
  norm_type: str | nn.Module
      The type of LayerNorm variant to use.  Default: ``LayerNorm``. Available options are
      ["LayerNorm", "RMSNorm", "LayerNormNoBias"], or provide a custom nn.Module.
  use_static_covariates
      Whether the model should use static covariate information in case the input `series` passed to ``fit()``
      contain static covariates. If ``True``, and static covariates are available at fitting time, will enforce
      that all target `series` have the same static covariate dimensionality in ``fit()`` and ``predict()``.
  **kwargs
      Optional arguments to initialize the pytorch_lightning.Module, pytorch_lightning.Trainer, and
      Darts' :class:`TorchForecastingModel`.
  
  torch_metrics
      A torch metric or a ``MetricCollection`` used for evaluation. A full list of available metrics can be found
      at https://torchmetrics.readthedocs.io/en/latest/. Default: ``None``.
  optimizer_cls
      The PyTorch optimizer class to be used. Default: ``torch.optim.Adam``.
  optimizer_kwargs
      Optionally, some keyword arguments for the PyTorch optimizer (e.g., ``{'lr': 1e-3}``
      for specifying a learning rate). Otherwise, the default values of the selected ``optimizer_cls``
      will be used. Default: ``None``.
  lr_scheduler_cls
      Optionally, the PyTorch learning rate scheduler class to be used. Specifying ``None`` corresponds
      to using a constant learning rate. Default: ``None``.
  lr_scheduler_kwargs
      Optionally, some keyword arguments for the PyTorch learning rate scheduler. Default: ``None``.
  use_reversible_instance_norm
      Whether to use reversible instance normalization `RINorm` against distribution shift as shown in [3]_.
      It is only applied to the features of the target series and not the covariates.
  batch_size
      Number of time series (input and output sequences) used in each training pass. Default: ``32``.
  n_epochs
      Number of epochs over which to train the model. Default: ``100``.
  model_name
      Name of the model. Used for creating checkpoints and saving tensorboard data. If not specified,
      defaults to the following string ``"YYYY-mm-dd_HH_MM_SS_torch_model_run_PID"``, where the initial part
      of the name is formatted with the local date and time, while PID is the processed ID (preventing models
      spawned at the same time by different processes to share the same model_name). E.g.,
      ``"2021-06-14_09_53_32_torch_model_run_44607"``.
  work_dir
      Path of the working directory, where to save checkpoints and Tensorboard summaries.
      Default: current working directory.
  log_tensorboard
      If set, use Tensorboard to log the different parameters. The logs will be located in:
      ``"{work_dir}/darts_logs/{model_name}/logs/"``. Default: ``False``.
  nr_epochs_val_period
      Number of epochs to wait before evaluating the validation loss (if a validation
      ``TimeSeries`` is passed to the :func:`fit()` method). Default: ``1``.
  force_reset
      If set to ``True``, any previously-existing model with the same name will be reset (all checkpoints will
      be discarded). Default: ``False``.
  save_checkpoints
      Whether to automatically save the untrained model and checkpoints from training.
      To load the model from checkpoint, call :func:`MyModelClass.load_from_checkpoint()`, where
      :class:`MyModelClass` is the :class:`TorchForecastingModel` class that was used (such as :class:`TFTModel`,
      :class:`NBEATSModel`, etc.). If set to ``False``, the model can still be manually saved using
      :func:`save()` and loaded using :func:`load()`. Default: ``False``.
  add_encoders
      A large number of past and future covariates can be automatically generated with `add_encoders`.
      This can be done by adding multiple pre-defined index encoders and/or custom user-made functions that
      will be used as index encoders. Additionally, a transformer such as Darts' :class:`Scaler` can be added to
      transform the generated covariates. This happens all under one hood and only needs to be specified at
      model creation.
      Read :meth:`SequentialEncoder <darts.dataprocessing.encoders.SequentialEncoder>` to find out more about
      ``add_encoders``. Default: ``None``. An example showing some of ``add_encoders`` features:
  
      .. highlight:: python
      .. code-block:: python
  
          def encode_year(idx):
              return (idx.year - 1950) / 50
  
          add_encoders={
              'cyclic': {'future': ['month']},
              'datetime_attribute': {'future': ['hour', 'dayofweek']},
              'position': {'past': ['relative'], 'future': ['relative']},
              'custom': {'past': [encode_year]},
              'transformer': Scaler(),
              'tz': 'CET'
          }
      ..
  random_state
      Control the randomness of the weight's initialization. Check this
      `link <https://scikit-learn.org/stable/glossary.html#term-random_state>`_ for more details.
      Default: ``None``.
  pl_trainer_kwargs
      By default :class:`TorchForecastingModel` creates a PyTorch Lightning Trainer with several useful presets
      that performs the training, validation and prediction processes. These presets include automatic
      checkpointing, tensorboard logging, setting the torch device and more.
      With ``pl_trainer_kwargs`` you can add additional kwargs to instantiate the PyTorch Lightning trainer
      object. Check the `PL Trainer documentation
      <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html>`_ for more information about the
      supported kwargs. Default: ``None``.
      Running on GPU(s) is also possible using ``pl_trainer_kwargs`` by specifying keys ``"accelerator",
      "devices", and "auto_select_gpus"``. Some examples for setting the devices inside the ``pl_trainer_kwargs``
      dict:
  
      - ``{"accelerator": "cpu"}`` for CPU,
      - ``{"accelerator": "gpu", "devices": [i]}`` to use only GPU ``i`` (``i`` must be an integer),
      - ``{"accelerator": "gpu", "devices": -1, "auto_select_gpus": True}`` to use all available GPUS.
  
      For more info, see here:
      https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-flags , and
      https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu_basic.html#train-on-multiple-gpus
  
      With parameter ``"callbacks"`` you can add custom or PyTorch-Lightning built-in callbacks to Darts'
      :class:`TorchForecastingModel`. Below is an example for adding EarlyStopping to the training process.
      The model will stop training early if the validation loss `val_loss` does not improve beyond
      specifications. For more information on callbacks, visit:
      `PyTorch Lightning Callbacks
      <https://pytorch-lightning.readthedocs.io/en/stable/extensions/callbacks.html>`_
  
      .. highlight:: python
      .. code-block:: python
  
          from pytorch_lightning.callbacks.early_stopping import EarlyStopping
  
          # stop training when validation loss does not decrease more than 0.05 (`min_delta`) over
          # a period of 5 epochs (`patience`)
          my_stopper = EarlyStopping(
              monitor="val_loss",
              patience=5,
              min_delta=0.05,
              mode='min',
          )
  
          pl_trainer_kwargs={"callbacks": [my_stopper]}
      ..
  
      Note that you can also use a custom PyTorch Lightning Trainer for training and prediction with optional
      parameter ``trainer`` in :func:`fit()` and :func:`predict()`.
  show_warnings
      whether to show warnings raised from PyTorch Lightning. Useful to detect potential issues of
      your forecasting use case. Default: ``False``.
  
  References
  ----------
  .. [1] https://arxiv.org/pdf/1912.09363.pdf
  .. [2] Shazeer, Noam, "GLU Variants Improve Transformer", 2020. arVix https://arxiv.org/abs/2002.05202.
  .. [3] T. Kim et al. "Reversible Instance Normalization for Accurate Time-Series Forecasting against
          Distribution Shift", https://openreview.net/forum?id=cGDAkQo1C0p
  
  Examples
  --------
  >>> from darts.datasets import WeatherDataset
  >>> from darts.models import TFTModel
  >>> series = WeatherDataset().load()
  >>> # predicting atmospheric pressure
  >>> target = series['p (mbar)'][:100]
  >>> # optionally, past observed rainfall (pretending to be unknown beyond index 100)
  >>> past_cov = series['rain (mm)'][:100]
  >>> # future temperatures (pretending this component is a forecast)
  >>> future_cov = series['T (degC)'][:106]
  >>> # by default, TFTModel is trained using a `QuantileRegression` making it a probabilistic forecasting model
  >>> model = TFTModel(
  >>>     input_chunk_length=6,
  >>>     output_chunk_length=6,
  >>>     n_epochs=5,
  >>> )
  >>> # future_covariates are mandatory for `TFTModel`
  >>> model.fit(target, past_covariates=past_cov, future_covariates=future_cov)
  >>> # TFTModel is probabilistic by definition; using `num_samples >> 1` to generate probabilistic forecasts
  >>> pred = model.predict(6, num_samples=100)
  >>> # shape : (forecast horizon, components, num_samples)
  >>> pred.all_values().shape
  (6, 1, 100)
  >>> # showing the first 3 samples for each timestamp
  >>> pred.all_values()[:,:,:3]
  array([[[-0.06414202, -0.7188093 ,  0.52541292]],
         [[ 0.02928407, -0.40867163,  1.19650033]],
         [[ 0.77252372, -0.50859694,  0.360166  ]],
         [[ 0.9586113 ,  1.24147138, -0.01625545]],
         [[ 1.06863863,  0.2987822 , -0.69213369]],
         [[-0.83076568, -0.25780816, -0.28318784]]])
  
  .. note::
      `TFT example notebook <https://unit8co.github.io/darts/examples/13-TFT-examples.html>`_ presents
      techniques that can be used to improve the forecasts quality compared to this simple usage example.
docstr_tok: ["Temporal","Fusion","Transformers","(","TFT",")","for","Interpretable","Time","Series","Forecasting",".","This","is","an","implementation","of","the","TFT","architecture",",","as","outlined","in","[","1","]","_",".","The","internal","sub","models","are","adopted","from","`","pytorch-forecasting","'s","TemporalFusionTransformer","<","https",":","\/\/pytorch-forecasting.readthedocs.io\/en\/latest\/models.html",">","`","_","implementation",".","This","model","supports","past","covariates","(","known","for","`","input_chunk_length","`","points","before","prediction","time",")",",","future","covariates","(","known","for","`","output_chunk_length","`","points","after","prediction","time",")",",","static","covariates",",","as","well","as","probabilistic","forecasting",".","The","TFT","applies","multi-head","attention","queries","on","future","inputs","from","mandatory","``","future_covariates","``",".","Specifying","future","encoders","with","``","add_encoders","``","(","read","below",")","can","automatically","generate","future","covariates","and","allows","to","use","the","model","without","having","to","pass","any","``","future_covariates","``","to",":","func",":","`","fit","(",")","`","and",":","func",":","`","predict","(",")","`",".","By","default",",","this","model","uses","the","``","QuantileRegression","``","likelihood",",","which","means","that","its","forecasts","are","probabilistic",";","it","is","recommended","to","call",":","func","`","predict","(",")","`","with","``","num_samples",">",">","1","``","to","get","meaningful","results",".","Parameters","--","--","--","--","--","input_chunk_length","Number","of","time","steps","in","the","past","to","take","as","a","model","input","(","per","chunk",")",".","Applies","to","the","target","series",",","and","past","and\/or","future","covariates","(","if","the","model","supports","it",")",".","Also","called",":","Encoder","length","output_chunk_length","Number","of","time","steps","predicted","at","once","(","per","chunk",")","by","the","internal","model",".","Also",",","the","number","of","future","values","from","future","covariates","to","use","as","a","model","input","(","if","the","model","supports","future","covariates",")",".","It","is","not","the","same","as","forecast","horizon","`","n","`","used","in","`","predict","(",")","`",",","which","is","the","desired","number","of","prediction","points","generated","using","either","a","one-shot-","or","auto-regressive","forecast",".","Setting","`","n","<","=","output_chunk_length","`","prevents","auto-regression",".","This","is","useful","when","the","covariates","do","n't","extend","far","enough","into","the","future",",","or","to","prohibit","the","model","from","using","future","values","of","past","and","\/","or","future","covariates","for","prediction","(","depending","on","the","model","'s","covariate","support",")",".","Also","called",":","Decoder","length","hidden_size","Hidden","state","size","of","the","TFT",".","It","is","the","main","hyper-parameter","and","common","across","the","internal","TFT","architecture",".","lstm_layers","Number","of","layers","for","the","Long","Short","Term","Memory","(","LSTM",")","Encoder","and","Decoder","(","1","is","a","good","default",")",".","num_attention_heads","Number","of","attention","heads","(","4","is","a","good","default",")","full_attention","If","``","False","``",",","only","attends","to","previous","time","steps","in","the","decoder",".","If","``","True","``","attends","to","previous",",","current",",","and","future","time","steps",".","Defaults","to","``","False","``",".","feed_forward","A","feedforward","network","is","a","fully-connected","layer","with","an","activation",".","TFT","Can","be","one","of","the","glu","variant's","FeedForward","Network","(","FFN",")","[","2","]",".","The","glu","variant","'s","FeedForward","Network","are","a","series","of","FFNs","designed","to","work","better","with","Transformer","based","models",".","Defaults","to","``","``","GatedResidualNetwork","''","``",".","[","``","GLU","''",",","``","Bilinear","''",",","``","ReGLU","''",",","``","GEGLU","''",",","``","SwiGLU","''",",","``","ReLU","''",",","``","GELU","''","]","or","the","TFT","original","FeedForward","Network","[","``","GatedResidualNetwork","''","]",".","dropout","Fraction","of","neurons","affected","by","dropout",".","This","is","compatible","with","Monte","Carlo","dropout","at","inference","time","for","model","uncertainty","estimation","(","enabled","with","``","mc_dropout=True","``","at","prediction","time",")",".","hidden_continuous_size","Default","for","hidden","size","for","processing","continuous","variables","categorical_embedding_sizes","A","dictionary","used","to","construct","embeddings","for","categorical","static","covariates",".","The","keys","are","the","column","names","of","the","categorical","static","covariates",".","Each","value","is","either","a","single","integer","or","a","tuple","of","integers",".","For","a","single","integer","give","the","number","of","unique","categories","(","n",")","of","the","corresponding","variable",".","For","example","``","{","``","some_column","''",":","64","}","``",".","The","embedding","size","will","be","automatically","determined","by","``","min","(","round","(","1.6","*","n","*","*","0.56",")",",","100",")","``",".","For","a","tuple","of","integers",",","give","(","number","of","unique","categories",",","embedding","size",")",".","For","example","``","{","``","some_column","''",":","(","64",",","8",")","}","``",".","Note","that","``","TorchForecastingModels","``","only","support","numeric","data",".","Consider","transforming\/encoding","your","data","with","`","darts.dataprocessing.transformers.static_covariates_transformer.StaticCovariatesTransformer","`",".","add_relative_index","Whether","to","add","positional","values","to","future","covariates",".","Defaults","to","``","False","``",".","This","allows","to","use","the","TFTModel","without","having","to","pass","future_covariates","to",":","func",":","`","fit","(",")","`","and",":","func",":","`","train","(",")","`",".","It","gives","a","value","to","the","position","of","each","step","from","input","and","output","chunk","relative","to","the","prediction","point",".","The","values","are","normalized","with","``","input_chunk_length","``",".","loss_fn",":","nn.Module","PyTorch","loss","function","used","for","training",".","By","default",",","the","TFT","model","is","probabilistic","and","uses","a","``","likelihood","``","instead","(","``","QuantileRegression","``",")",".","To","make","the","model","deterministic",",","you","can","set","the","`","`","likelihood","``","to","None","and","give","a","``","loss_fn","``","argument",".","likelihood","The","likelihood","model","to","be","used","for","probabilistic","forecasts",".","By","default",",","the","TFT","uses","a","``","QuantileRegression","``","likelihood",".","norm_type",":","str","|","nn.Module","The","type","of","LayerNorm","variant","to","use",".","Default",":","``","LayerNorm","``",".","Available","options","are","[","``","LayerNorm","''",",","``","RMSNorm","''",",","``","LayerNormNoBias","''","]",",","or","provide","a","custom","nn.Module",".","use_static_covariates","Whether","the","model","should","use","static","covariate","information","in","case","the","input","`","series","`","passed","to","``","fit","(",")","``","contain","static","covariates",".","If","``","True","``",",","and","static","covariates","are","available","at","fitting","time",",","will","enforce","that","all","target","`","series","`","have","the","same","static","covariate","dimensionality","in","``","fit","(",")","``","and","``","predict","(",")","``",".","*","*","kwargs","Optional","arguments","to","initialize","the","pytorch_lightning.Module",",","pytorch_lightning.Trainer",",","and","Darts","'",":","class",":","`","TorchForecastingModel","`",".","torch_metrics","A","torch","metric","or","a","``","MetricCollection","``","used","for","evaluation",".","A","full","list","of","available","metrics","can","be","found","at","https",":","\/\/torchmetrics.readthedocs.io\/en\/latest\/",".","Default",":","``","None","``",".","optimizer_cls","The","PyTorch","optimizer","class","to","be","used",".","Default",":","``","torch.optim.Adam","``",".","optimizer_kwargs","Optionally",",","some","keyword","arguments","for","the","PyTorch","optimizer","(","e.g.",",","``","{","'lr","'",":","1e-3","}","``","for","specifying","a","learning","rate",")",".","Otherwise",",","the","default","values","of","the","selected","``","optimizer_cls","``","will","be","used",".","Default",":","``","None","``",".","lr_scheduler_cls","Optionally",",","the","PyTorch","learning","rate","scheduler","class","to","be","used",".","Specifying","``","None","``","corresponds","to","using","a","constant","learning","rate",".","Default",":","``","None","``",".","lr_scheduler_kwargs","Optionally",",","some","keyword","arguments","for","the","PyTorch","learning","rate","scheduler",".","Default",":","``","None","``",".","use_reversible_instance_norm","Whether","to","use","reversible","instance","normalization","`","RINorm","`","against","distribution","shift","as","shown","in","[","3","]","_",".","It","is","only","applied","to","the","features","of","the","target","series","and","not","the","covariates",".","batch_size","Number","of","time","series","(","input","and","output","sequences",")","used","in","each","training","pass",".","Default",":","``","32","``",".","n_epochs","Number","of","epochs","over","which","to","train","the","model",".","Default",":","``","100","``",".","model_name","Name","of","the","model",".","Used","for","creating","checkpoints","and","saving","tensorboard","data",".","If","not","specified",",","defaults","to","the","following","string","``","``","YYYY-mm-dd_HH_MM_SS_torch_model_run_PID","''","``",",","where","the","initial","part","of","the","name","is","formatted","with","the","local","date","and","time",",","while","PID","is","the","processed","ID","(","preventing","models","spawned","at","the","same","time","by","different","processes","to","share","the","same","model_name",")",".","E.g.",",","``","``","2021-06-14_09_53_32_torch_model_run_44607","''","``",".","work_dir","Path","of","the","working","directory",",","where","to","save","checkpoints","and","Tensorboard","summaries",".","Default",":","current","working","directory",".","log_tensorboard","If","set",",","use","Tensorboard","to","log","the","different","parameters",".","The","logs","will","be","located","in",":","``","``","{","work_dir","}","\/darts_logs\/","{","model_name","}","\/logs\/","''","``",".","Default",":","``","False","``",".","nr_epochs_val_period","Number","of","epochs","to","wait","before","evaluating","the","validation","loss","(","if","a","validation","``","TimeSeries","``","is","passed","to","the",":","func",":","`","fit","(",")","`","method",")",".","Default",":","``","1","``",".","force_reset","If","set","to","``","True","``",",","any","previously-existing","model","with","the","same","name","will","be","reset","(","all","checkpoints","will","be","discarded",")",".","Default",":","``","False","``",".","save_checkpoints","Whether","to","automatically","save","the","untrained","model","and","checkpoints","from","training",".","To","load","the","model","from","checkpoint",",","call",":","func",":","`","MyModelClass.load_from_checkpoint","(",")","`",",","where",":","class",":","`","MyModelClass","`","is","the",":","class",":","`","TorchForecastingModel","`","class","that","was","used","(","such","as",":","class",":","`","TFTModel","`",",",":","class",":","`","NBEATSModel","`",",","etc.",")",".","If","set","to","``","False","``",",","the","model","can","still","be","manually","saved","using",":","func",":","`","save","(",")","`","and","loaded","using",":","func",":","`","load","(",")","`",".","Default",":","``","False","``",".","add_encoders","A","large","number","of","past","and","future","covariates","can","be","automatically","generated","with","`","add_encoders","`",".","This","can","be","done","by","adding","multiple","pre-defined","index","encoders","and\/or","custom","user-made","functions","that","will","be","used","as","index","encoders",".","Additionally",",","a","transformer","such","as","Darts","'",":","class",":","`","Scaler","`","can","be","added","to","transform","the","generated","covariates",".","This","happens","all","under","one","hood","and","only","needs","to","be","specified","at","model","creation",".","Read",":","meth",":","`","SequentialEncoder","<","darts.dataprocessing.encoders.SequentialEncoder",">","`","to","find","out","more","about","``","add_encoders","``",".","Default",":","``","None","``",".","An","example","showing","some","of","``","add_encoders","``","features",":","..","highlight",":",":","python","..","code-block",":",":","python","def","encode_year","(","idx",")",":","return","(","idx.year","-","1950",")","\/","50","add_encoders=","{","'cyclic","'",":","{","'future","'",":","[","'month","'","]","}",",","'datetime_attribute","'",":","{","'future","'",":","[","'hour","'",",","'dayofweek","'","]","}",",","'position","'",":","{","'past","'",":","[","'relative","'","]",",","'future","'",":","[","'relative","'","]","}",",","'custom","'",":","{","'past","'",":","[","encode_year","]","}",",","'transformer","'",":","Scaler","(",")",",","'tz","'",":","'CET'","}","..","random_state","Control","the","randomness","of","the","weight","'s","initialization",".","Check","this","`","link","<","https",":","\/\/scikit-learn.org\/stable\/glossary.html","#","term-random_state",">","`","_","for","more","details",".","Default",":","``","None","``",".","pl_trainer_kwargs","By","default",":","class",":","`","TorchForecastingModel","`","creates","a","PyTorch","Lightning","Trainer","with","several","useful","presets","that","performs","the","training",",","validation","and","prediction","processes",".","These","presets","include","automatic","checkpointing",",","tensorboard","logging",",","setting","the","torch","device","and","more",".","With","``","pl_trainer_kwargs","``","you","can","add","additional","kwargs","to","instantiate","the","PyTorch","Lightning","trainer","object",".","Check","the","`","PL","Trainer","documentation","<","https",":","\/\/pytorch-lightning.readthedocs.io\/en\/stable\/common\/trainer.html",">","`","_","for","more","information","about","the","supported","kwargs",".","Default",":","``","None","``",".","Running","on","GPU","(","s",")","is","also","possible","using","``","pl_trainer_kwargs","``","by","specifying","keys","``","``","accelerator","''",",","``","devices","''",",","and","``","auto_select_gpus","''","``",".","Some","examples","for","setting","the","devices","inside","the","``","pl_trainer_kwargs","``","dict",":","-","``","{","``","accelerator","''",":","``","cpu","''","}","``","for","CPU",",","-","``","{","``","accelerator","''",":","``","gpu","''",",","``","devices","''",":","[","i","]","}","``","to","use","only","GPU","``","i","``","(","``","i","``","must","be","an","integer",")",",","-","``","{","``","accelerator","''",":","``","gpu","''",",","``","devices","''",":","-1",",","``","auto_select_gpus","''",":","True","}","``","to","use","all","available","GPUS",".","For","more","info",",","see","here",":","https",":","\/\/pytorch-lightning.readthedocs.io\/en\/stable\/common\/trainer.html","#","trainer-flags",",","and","https",":","\/\/pytorch-lightning.readthedocs.io\/en\/stable\/accelerators\/gpu_basic.html","#","train-on-multiple-gpus","With","parameter","``","``","callbacks","''","``","you","can","add","custom","or","PyTorch-Lightning","built-in","callbacks","to","Darts'",":","class",":","`","TorchForecastingModel","`",".","Below","is","an","example","for","adding","EarlyStopping","to","the","training","process",".","The","model","will","stop","training","early","if","the","validation","loss","`","val_loss","`","does","not","improve","beyond","specifications",".","For","more","information","on","callbacks",",","visit",":","`","PyTorch","Lightning","Callbacks","<","https",":","\/\/pytorch-lightning.readthedocs.io\/en\/stable\/extensions\/callbacks.html",">","`","_","..","highlight",":",":","python","..","code-block",":",":","python","from","pytorch_lightning.callbacks.early_stopping","import","EarlyStopping","#","stop","training","when","validation","loss","does","not","decrease","more","than","0.05","(","`","min_delta","`",")","over","#","a","period","of","5","epochs","(","`","patience","`",")","my_stopper","=","EarlyStopping","(","monitor=","''","val_loss","''",",","patience=5",",","min_delta=0.05",",","mode='min","'",",",")","pl_trainer_kwargs=","{","``","callbacks","''",":","[","my_stopper","]","}","..","Note","that","you","can","also","use","a","custom","PyTorch","Lightning","Trainer","for","training","and","prediction","with","optional","parameter","``","trainer","``","in",":","func",":","`","fit","(",")","`","and",":","func",":","`","predict","(",")","`",".","show_warnings","whether","to","show","warnings","raised","from","PyTorch","Lightning",".","Useful","to","detect","potential","issues","of","your","forecasting","use","case",".","Default",":","``","False","``",".","References","--","--","--","--","--","..","[","1","]","https",":","\/\/arxiv.org\/pdf\/1912.09363.pdf","..","[","2","]","Shazeer",",","Noam",",","``","GLU","Variants","Improve","Transformer","''",",","2020.","arVix","https",":","\/\/arxiv.org\/abs\/2002.05202",".","..","[","3","]","T.","Kim","et","al",".","``","Reversible","Instance","Normalization","for","Accurate","Time-Series","Forecasting","against","Distribution","Shift","''",",","https",":","\/\/openreview.net\/forum","?","id=cGDAkQo1C0p","Examples","--","--","--","--",">",">",">","from","darts.datasets","import","WeatherDataset",">",">",">","from","darts.models","import","TFTModel",">",">",">","series","=","WeatherDataset","(",")",".load","(",")",">",">",">","#","predicting","atmospheric","pressure",">",">",">","target","=","series","[","'","p","(","mbar",")","'","]","[",":100","]",">",">",">","#","optionally",",","past","observed","rainfall","(","pretending","to","be","unknown","beyond","index","100",")",">",">",">","past_cov","=","series","[","'rain","(","mm",")","'","]","[",":100","]",">",">",">","#","future","temperatures","(","pretending","this","component","is","a","forecast",")",">",">",">","future_cov","=","series","[","'T","(","degC",")","'","]","[",":106","]",">",">",">","#","by","default",",","TFTModel","is","trained","using","a","`","QuantileRegression","`","making","it","a","probabilistic","forecasting","model",">",">",">","model","=","TFTModel","(",">",">",">","input_chunk_length=6",",",">",">",">","output_chunk_length=6",",",">",">",">","n_epochs=5",",",">",">",">",")",">",">",">","#","future_covariates","are","mandatory","for","`","TFTModel","`",">",">",">","model.fit","(","target",",","past_covariates=past_cov",",","future_covariates=future_cov",")",">",">",">","#","TFTModel","is","probabilistic","by","definition",";","using","`","num_samples",">",">","1","`","to","generate","probabilistic","forecasts",">",">",">","pred","=","model.predict","(","6",",","num_samples=100",")",">",">",">","#","shape",":","(","forecast","horizon",",","components",",","num_samples",")",">",">",">","pred.all_values","(",")",".shape","(","6",",","1",",","100",")",">",">",">","#","showing","the","first","3","samples","for","each","timestamp",">",">",">","pred.all_values","(",")","[",":",",",":",",:3","]","array","(","[","[","[","-0.06414202",",","-0.7188093",",","0.52541292","]","]",",","[","[","0.02928407",",","-0.40867163",",","1.19650033","]","]",",","[","[","0.77252372",",","-0.50859694",",","0.360166","]","]",",","[","[","0.9586113",",","1.24147138",",","-0.01625545","]","]",",","[","[","1.06863863",",","0.2987822",",","-0.69213369","]","]",",","[","[","-0.83076568",",","-0.25780816",",","-0.28318784","]","]","]",")","..","note",":",":","`","TFT","example","notebook","<","https",":","\/\/unit8co.github.io\/darts\/examples\/13-TFT-examples.html",">","`","_","presents","techniques","that","can","be","used","to","improve","the","forecasts","quality","compared","to","this","simple","usage","example","."]
code: def __init__(
      self,
      input_chunk_length: int,
      output_chunk_length: int,
      hidden_size: Union[int, List[int]] = 16,
      lstm_layers: int = 1,
      num_attention_heads: int = 4,
      full_attention: bool = False,
      feed_forward: str = "GatedResidualNetwork",
      dropout: float = 0.1,
      hidden_continuous_size: int = 8,
      categorical_embedding_sizes: Optional[
          Dict[str, Union[int, Tuple[int, int]]]
      ] = None,
      add_relative_index: bool = False,
      loss_fn: Optional[nn.Module] = None,
      likelihood: Optional[Likelihood] = None,
      norm_type: Union[str, nn.Module] = "LayerNorm",
      use_static_covariates: bool = True,
      **kwargs,
  ):
      """Temporal Fusion Transformers (TFT) for Interpretable Time Series Forecasting.
  
      This is an implementation of the TFT architecture, as outlined in [1]_.
  
      The internal sub models are adopted from `pytorch-forecasting's TemporalFusionTransformer
      <https://pytorch-forecasting.readthedocs.io/en/latest/models.html>`_ implementation.
  
      This model supports past covariates (known for `input_chunk_length` points before prediction time),
      future covariates (known for `output_chunk_length` points after prediction time), static covariates,
      as well as probabilistic forecasting.
  
      The TFT applies multi-head attention queries on future inputs from mandatory ``future_covariates``.
      Specifying future encoders with ``add_encoders`` (read below) can automatically generate future covariates
      and allows to use the model without having to pass any ``future_covariates`` to :func:`fit()` and
      :func:`predict()`.
  
      By default, this model uses the ``QuantileRegression`` likelihood, which means that its forecasts are
      probabilistic; it is recommended to call :func`predict()` with ``num_samples >> 1`` to get meaningful results.
  
      Parameters
      ----------
      input_chunk_length
          Number of time steps in the past to take as a model input (per chunk). Applies to the target
          series, and past and/or future covariates (if the model supports it).
          Also called: Encoder length
      output_chunk_length
          Number of time steps predicted at once (per chunk) by the internal model. Also, the number of future values
          from future covariates to use as a model input (if the model supports future covariates). It is not the same
          as forecast horizon `n` used in `predict()`, which is the desired number of prediction points generated
          using either a one-shot- or auto-regressive forecast. Setting `n <= output_chunk_length` prevents
          auto-regression. This is useful when the covariates don't extend far enough into the future, or to prohibit
          the model from using future values of past and / or future covariates for prediction (depending on the
          model's covariate support).
          Also called: Decoder length
      hidden_size
          Hidden state size of the TFT. It is the main hyper-parameter and common across the internal TFT
          architecture.
      lstm_layers
          Number of layers for the Long Short Term Memory (LSTM) Encoder and Decoder (1 is a good default).
      num_attention_heads
          Number of attention heads (4 is a good default)
      full_attention
          If ``False``, only attends to previous time steps in the decoder. If ``True`` attends to previous,
          current, and future time steps. Defaults to ``False``.
      feed_forward
          A feedforward network is a fully-connected layer with an activation. TFT Can be one of the glu variant's
          FeedForward Network (FFN)[2]. The glu variant's FeedForward Network are a series of FFNs designed to work
          better with Transformer based models. Defaults to ``"GatedResidualNetwork"``. ["GLU", "Bilinear", "ReGLU",
          "GEGLU", "SwiGLU", "ReLU", "GELU"] or the TFT original FeedForward Network ["GatedResidualNetwork"].
      dropout
          Fraction of neurons affected by dropout. This is compatible with Monte Carlo dropout
          at inference time for model uncertainty estimation (enabled with ``mc_dropout=True`` at
          prediction time).
      hidden_continuous_size
          Default for hidden size for processing continuous variables
      categorical_embedding_sizes
          A dictionary used to construct embeddings for categorical static covariates. The keys are the column names
          of the categorical static covariates. Each value is either a single integer or a tuple of integers.
          For a single integer give the number of unique categories (n) of the corresponding variable. For example
          ``{"some_column": 64}``. The embedding size will be automatically determined by
          ``min(round(1.6 * n**0.56), 100)``.
          For a tuple of integers, give (number of unique categories, embedding size). For example
          ``{"some_column": (64, 8)}``.
          Note that ``TorchForecastingModels`` only support numeric data. Consider transforming/encoding your data
          with `darts.dataprocessing.transformers.static_covariates_transformer.StaticCovariatesTransformer`.
      add_relative_index
          Whether to add positional values to future covariates. Defaults to ``False``.
          This allows to use the TFTModel without having to pass future_covariates to :func:`fit()` and
          :func:`train()`. It gives a value to the position of each step from input and output chunk relative
          to the prediction point. The values are normalized with ``input_chunk_length``.
      loss_fn: nn.Module
          PyTorch loss function used for training. By default, the TFT model is probabilistic and uses a
          ``likelihood`` instead (``QuantileRegression``). To make the model deterministic, you can set the `
          `likelihood`` to None and give a ``loss_fn`` argument.
      likelihood
          The likelihood model to be used for probabilistic forecasts. By default, the TFT uses
          a ``QuantileRegression`` likelihood.
      norm_type: str | nn.Module
          The type of LayerNorm variant to use.  Default: ``LayerNorm``. Available options are
          ["LayerNorm", "RMSNorm", "LayerNormNoBias"], or provide a custom nn.Module.
      use_static_covariates
          Whether the model should use static covariate information in case the input `series` passed to ``fit()``
          contain static covariates. If ``True``, and static covariates are available at fitting time, will enforce
          that all target `series` have the same static covariate dimensionality in ``fit()`` and ``predict()``.
      **kwargs
          Optional arguments to initialize the pytorch_lightning.Module, pytorch_lightning.Trainer, and
          Darts' :class:`TorchForecastingModel`.
  
      torch_metrics
          A torch metric or a ``MetricCollection`` used for evaluation. A full list of available metrics can be found
          at https://torchmetrics.readthedocs.io/en/latest/. Default: ``None``.
      optimizer_cls
          The PyTorch optimizer class to be used. Default: ``torch.optim.Adam``.
      optimizer_kwargs
          Optionally, some keyword arguments for the PyTorch optimizer (e.g., ``{'lr': 1e-3}``
          for specifying a learning rate). Otherwise, the default values of the selected ``optimizer_cls``
          will be used. Default: ``None``.
      lr_scheduler_cls
          Optionally, the PyTorch learning rate scheduler class to be used. Specifying ``None`` corresponds
          to using a constant learning rate. Default: ``None``.
      lr_scheduler_kwargs
          Optionally, some keyword arguments for the PyTorch learning rate scheduler. Default: ``None``.
      use_reversible_instance_norm
          Whether to use reversible instance normalization `RINorm` against distribution shift as shown in [3]_.
          It is only applied to the features of the target series and not the covariates.
      batch_size
          Number of time series (input and output sequences) used in each training pass. Default: ``32``.
      n_epochs
          Number of epochs over which to train the model. Default: ``100``.
      model_name
          Name of the model. Used for creating checkpoints and saving tensorboard data. If not specified,
          defaults to the following string ``"YYYY-mm-dd_HH_MM_SS_torch_model_run_PID"``, where the initial part
          of the name is formatted with the local date and time, while PID is the processed ID (preventing models
          spawned at the same time by different processes to share the same model_name). E.g.,
          ``"2021-06-14_09_53_32_torch_model_run_44607"``.
      work_dir
          Path of the working directory, where to save checkpoints and Tensorboard summaries.
          Default: current working directory.
      log_tensorboard
          If set, use Tensorboard to log the different parameters. The logs will be located in:
          ``"{work_dir}/darts_logs/{model_name}/logs/"``. Default: ``False``.
      nr_epochs_val_period
          Number of epochs to wait before evaluating the validation loss (if a validation
          ``TimeSeries`` is passed to the :func:`fit()` method). Default: ``1``.
      force_reset
          If set to ``True``, any previously-existing model with the same name will be reset (all checkpoints will
          be discarded). Default: ``False``.
      save_checkpoints
          Whether to automatically save the untrained model and checkpoints from training.
          To load the model from checkpoint, call :func:`MyModelClass.load_from_checkpoint()`, where
          :class:`MyModelClass` is the :class:`TorchForecastingModel` class that was used (such as :class:`TFTModel`,
          :class:`NBEATSModel`, etc.). If set to ``False``, the model can still be manually saved using
          :func:`save()` and loaded using :func:`load()`. Default: ``False``.
      add_encoders
          A large number of past and future covariates can be automatically generated with `add_encoders`.
          This can be done by adding multiple pre-defined index encoders and/or custom user-made functions that
          will be used as index encoders. Additionally, a transformer such as Darts' :class:`Scaler` can be added to
          transform the generated covariates. This happens all under one hood and only needs to be specified at
          model creation.
          Read :meth:`SequentialEncoder <darts.dataprocessing.encoders.SequentialEncoder>` to find out more about
          ``add_encoders``. Default: ``None``. An example showing some of ``add_encoders`` features:
  
          .. highlight:: python
          .. code-block:: python
  
              def encode_year(idx):
                  return (idx.year - 1950) / 50
  
              add_encoders={
                  'cyclic': {'future': ['month']},
                  'datetime_attribute': {'future': ['hour', 'dayofweek']},
                  'position': {'past': ['relative'], 'future': ['relative']},
                  'custom': {'past': [encode_year]},
                  'transformer': Scaler(),
                  'tz': 'CET'
              }
          ..
      random_state
          Control the randomness of the weight's initialization. Check this
          `link <https://scikit-learn.org/stable/glossary.html#term-random_state>`_ for more details.
          Default: ``None``.
      pl_trainer_kwargs
          By default :class:`TorchForecastingModel` creates a PyTorch Lightning Trainer with several useful presets
          that performs the training, validation and prediction processes. These presets include automatic
          checkpointing, tensorboard logging, setting the torch device and more.
          With ``pl_trainer_kwargs`` you can add additional kwargs to instantiate the PyTorch Lightning trainer
          object. Check the `PL Trainer documentation
          <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html>`_ for more information about the
          supported kwargs. Default: ``None``.
          Running on GPU(s) is also possible using ``pl_trainer_kwargs`` by specifying keys ``"accelerator",
          "devices", and "auto_select_gpus"``. Some examples for setting the devices inside the ``pl_trainer_kwargs``
          dict:
  
          - ``{"accelerator": "cpu"}`` for CPU,
          - ``{"accelerator": "gpu", "devices": [i]}`` to use only GPU ``i`` (``i`` must be an integer),
          - ``{"accelerator": "gpu", "devices": -1, "auto_select_gpus": True}`` to use all available GPUS.
  
          For more info, see here:
          https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-flags , and
          https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu_basic.html#train-on-multiple-gpus
  
          With parameter ``"callbacks"`` you can add custom or PyTorch-Lightning built-in callbacks to Darts'
          :class:`TorchForecastingModel`. Below is an example for adding EarlyStopping to the training process.
          The model will stop training early if the validation loss `val_loss` does not improve beyond
          specifications. For more information on callbacks, visit:
          `PyTorch Lightning Callbacks
          <https://pytorch-lightning.readthedocs.io/en/stable/extensions/callbacks.html>`_
  
          .. highlight:: python
          .. code-block:: python
  
              from pytorch_lightning.callbacks.early_stopping import EarlyStopping
  
              # stop training when validation loss does not decrease more than 0.05 (`min_delta`) over
              # a period of 5 epochs (`patience`)
              my_stopper = EarlyStopping(
                  monitor="val_loss",
                  patience=5,
                  min_delta=0.05,
                  mode='min',
              )
  
              pl_trainer_kwargs={"callbacks": [my_stopper]}
          ..
  
          Note that you can also use a custom PyTorch Lightning Trainer for training and prediction with optional
          parameter ``trainer`` in :func:`fit()` and :func:`predict()`.
      show_warnings
          whether to show warnings raised from PyTorch Lightning. Useful to detect potential issues of
          your forecasting use case. Default: ``False``.
  
      References
      ----------
      .. [1] https://arxiv.org/pdf/1912.09363.pdf
      .. [2] Shazeer, Noam, "GLU Variants Improve Transformer", 2020. arVix https://arxiv.org/abs/2002.05202.
      .. [3] T. Kim et al. "Reversible Instance Normalization for Accurate Time-Series Forecasting against
              Distribution Shift", https://openreview.net/forum?id=cGDAkQo1C0p
  
      Examples
      --------
      >>> from darts.datasets import WeatherDataset
      >>> from darts.models import TFTModel
      >>> series = WeatherDataset().load()
      >>> # predicting atmospheric pressure
      >>> target = series['p (mbar)'][:100]
      >>> # optionally, past observed rainfall (pretending to be unknown beyond index 100)
      >>> past_cov = series['rain (mm)'][:100]
      >>> # future temperatures (pretending this component is a forecast)
      >>> future_cov = series['T (degC)'][:106]
      >>> # by default, TFTModel is trained using a `QuantileRegression` making it a probabilistic forecasting model
      >>> model = TFTModel(
      >>>     input_chunk_length=6,
      >>>     output_chunk_length=6,
      >>>     n_epochs=5,
      >>> )
      >>> # future_covariates are mandatory for `TFTModel`
      >>> model.fit(target, past_covariates=past_cov, future_covariates=future_cov)
      >>> # TFTModel is probabilistic by definition; using `num_samples >> 1` to generate probabilistic forecasts
      >>> pred = model.predict(6, num_samples=100)
      >>> # shape : (forecast horizon, components, num_samples)
      >>> pred.all_values().shape
      (6, 1, 100)
      >>> # showing the first 3 samples for each timestamp
      >>> pred.all_values()[:,:,:3]
      array([[[-0.06414202, -0.7188093 ,  0.52541292]],
             [[ 0.02928407, -0.40867163,  1.19650033]],
             [[ 0.77252372, -0.50859694,  0.360166  ]],
             [[ 0.9586113 ,  1.24147138, -0.01625545]],
             [[ 1.06863863,  0.2987822 , -0.69213369]],
             [[-0.83076568, -0.25780816, -0.28318784]]])
  
      .. note::
          `TFT example notebook <https://unit8co.github.io/darts/examples/13-TFT-examples.html>`_ presents
          techniques that can be used to improve the forecasts quality compared to this simple usage example.
      """
      model_kwargs = {
          key: val for key, val in self.model_params.items()
      }
      if likelihood is None and loss_fn is None:
          # This is the default if no loss information is provided
          model_kwargs["loss_fn"] = None
          model_kwargs["likelihood"] = QuantileRegression()
  
      super().__init__(
          **self._extract_torch_model_params(**model_kwargs)
      )
  
      # extract pytorch lightning module kwargs
      self.pl_module_params = self._extract_pl_module_params(
          **model_kwargs
      )
  
      self.hidden_size = hidden_size
      self.lstm_layers = lstm_layers
      self.num_attention_heads = num_attention_heads
      self.full_attention = full_attention
      self.feed_forward = feed_forward
      self.dropout = dropout
      self.hidden_continuous_size = hidden_continuous_size
      self.categorical_embedding_sizes = (
          categorical_embedding_sizes
          if categorical_embedding_sizes is not None
          else {}
      )
      self.add_relative_index = add_relative_index
      self.output_dim: Optional[Tuple[int, int]] = None
      self.norm_type = norm_type
      self._considers_static_covariates = use_static_covariates
  
imports: from typing import Dict, List, Optional, Sequence, Tuple, Union
  import numpy
  import pandas
  import torch
  from torch import nn
  from torch.nn import LSTM
  from darts import TimeSeries
  from darts.logging import get_logger, raise_if, raise_if_not, raise_log
  from darts.models.components import glu_variants, layer_norm_variants
  from darts.models.components.glu_variants import GLU_FFN
  from darts.models.forecasting.pl_forecasting_module import PLMixedCovariatesModule, io_processor
  from darts.models.forecasting.tft_submodels import _GateAddNorm, _GatedResidualNetwork, _InterpretableMultiHeadAttention, _MultiEmbedding, _VariableSelectionNetwork, get_embedding_size
  from darts.models.forecasting.torch_forecasting_model import MixedCovariatesTorchModel
  from darts.utils.data import MixedCovariatesSequentialDataset, MixedCovariatesTrainingDataset, TrainingDataset
  from darts.utils.likelihood_models import Likelihood, QuantileRegression
label_desc: Use image node_id 1 to create a new TFTModel object from inherited base classes: MixedCovariatesTorchModel with example: obj = TFTModel(input_chunk_length, output_chunk_length, hidden_size, lstm_layers, num_attention_heads, full_attention, feed_forward, dropout, hidden_continuous_size, categorical_embedding_sizes, add_relative_index, loss_fn, likelihood, norm_type, use_static_covariates)