class_docstr: Class for benchmarking multiple ForecastConfig on a rolling window basis.
  
  Attributes
  ----------
  df : `pandas.DataFrame`
      Timeseries data to forecast.
      Contains columns [`time_col`, `value_col`], and optional regressor columns.
      Regressor columns should include future values for prediction.
  
  configs : `Dict` [`str`,  `ForecastConfig`]
      Dictionary of model configurations.
      A model configuration is a ``ForecastConfig``.
      See :class:`~greykite.framework.templates.autogen.forecast_config.ForecastConfig` for details on
      valid ``ForecastConfig``.
      Validity of the ``configs`` for benchmarking is checked via the ``validate`` method.
  
  tscv : `~greykite.sklearn.cross_validation.RollingTimeSeriesSplit`
      Cross-validation object that determines the rolling window evaluation.
      See :class:`~greykite.sklearn.cross_validation.RollingTimeSeriesSplit` for details.
      The ``forecast_horizon`` and ``periods_between_train_test`` parameters of ``configs`` are
      matched against that of ``tscv``. A ValueError is raised if there is a mismatch.
  
  forecaster : `~greykite.framework.templates.forecaster.Forecaster`
      Forecaster used to create the forecasts.
  
  is_run : bool, default False
      Indicator of whether the `run` method is executed.
      After executing `run`, this indicator is set to True.
      Some class methods like ``get_forecast`` requires ``is_run`` to be True
      to be executed.
  
  result : `dict`
      Stores the benchmarking results. Has the same keys as ``configs``.
  
  forecasts : `pandas.DataFrame`, default None
      Merged DataFrame of forecasts, upper and lower confidence interval for all
      input ``configs``. Also stores train end date and forecast step for each prediction.
docstr: Returns rolling train and test evaluation metric values.
  
  Parameters
  ----------
  metric_dict : `dict` [`str`, `callable`]
      Evaluation metrics to compute.
  
          - key: evaluation metric name, used to create column name in output.
          - value: metric function to apply to forecast df in each split to generate the column value.
                  Signature (y_true: `str`, y_pred: `str`) -> transformed value: `float`.
  
      For example::
  
          metric_dict = {
              "median_residual": lambda y_true, y_pred: np.median(y_true - y_pred),
              "mean_squared_error": lambda y_true, y_pred: np.mean((y_true - y_pred)**2)
          }
  
      Some predefined functions are available in
      `~greykite.common.evaluation`. For example::
  
          metric_dict = {
              "correlation": lambda y_true, y_pred: correlation(y_true, y_pred),
              "RMSE": lambda y_true, y_pred: root_mean_squared_error(y_true, y_pred),
              "Q_95": lambda y_true, y_pred: partial(quantile_loss(y_true, y_pred, q=0.95))
          }
  
      As shorthand, it is sufficient to provide the corresponding ``EvaluationMetricEnum``
      member.  These are auto-expanded into the appropriate function.
      So the following is equivalent::
  
          metric_dict = {
              "correlation": EvaluationMetricEnum.Correlation,
              "RMSE": EvaluationMetricEnum.RootMeanSquaredError,
              "Q_95": EvaluationMetricEnum.Quantile95
          }
  
  config_names : `list` [`str`], default None
      Which config results to plot. A list of config names.
      If None, uses all the available config keys.
  
  Returns
  -------
  evaluation_metrics_df : pd.DataFrame
      A DataFrame containing splitwise train and test evaluation metrics for ``metric_dict``
      and ``config_names``.
  
      For example. Let's assume::
  
          metric_dict = {
              "RMSE": EvaluationMetricEnum.RootMeanSquaredError,
              "Q_95": EvaluationMetricEnum.Quantile95
          }
  
          config_names = ["default_prophet", "custom_silverkite"]
          These are valid ``config_names`` and there are 2 splits for each.
  
          Then evaluation_metrics_df =
  
          config_name     split_num   train_RMSE  test_RMSE   train_Q_95  test_Q_95
          default_prophet      0          *           *           *           *
          default_prophet      1          *           *           *           *
          custom_silverkite    0          *           *           *           *
          custom_silverkite    1          *           *           *           *
  
          where * represents computed values.
docstr_tok: ["Returns","rolling","train","and","test","evaluation","metric","values",".","Parameters","--","--","--","--","--","metric_dict",":","`","dict","`","[","`","str","`",",","`","callable","`","]","Evaluation","metrics","to","compute",".","-","key",":","evaluation","metric","name",",","used","to","create","column","name","in","output",".","-","value",":","metric","function","to","apply","to","forecast","df","in","each","split","to","generate","the","column","value",".","Signature","(","y_true",":","`","str","`",",","y_pred",":","`","str","`",")","-",">","transformed","value",":","`","float","`",".","For","example",":",":","metric_dict","=","{","``","median_residual","''",":","lambda","y_true",",","y_pred",":","np.median","(","y_true","-","y_pred",")",",","``","mean_squared_error","''",":","lambda","y_true",",","y_pred",":","np.mean","(","(","y_true","-","y_pred",")","*","*","2",")","}","Some","predefined","functions","are","available","in","`","~greykite.common.evaluation","`",".","For","example",":",":","metric_dict","=","{","``","correlation","''",":","lambda","y_true",",","y_pred",":","correlation","(","y_true",",","y_pred",")",",","``","RMSE","''",":","lambda","y_true",",","y_pred",":","root_mean_squared_error","(","y_true",",","y_pred",")",",","``","Q_95","''",":","lambda","y_true",",","y_pred",":","partial","(","quantile_loss","(","y_true",",","y_pred",",","q=0.95",")",")","}","As","shorthand",",","it","is","sufficient","to","provide","the","corresponding","``","EvaluationMetricEnum","``","member",".","These","are","auto-expanded","into","the","appropriate","function",".","So","the","following","is","equivalent",":",":","metric_dict","=","{","``","correlation","''",":","EvaluationMetricEnum.Correlation",",","``","RMSE","''",":","EvaluationMetricEnum.RootMeanSquaredError",",","``","Q_95","''",":","EvaluationMetricEnum.Quantile95","}","config_names",":","`","list","`","[","`","str","`","]",",","default","None","Which","config","results","to","plot",".","A","list","of","config","names",".","If","None",",","uses","all","the","available","config","keys",".","Returns","--","--","--","-","evaluation_metrics_df",":","pd.DataFrame","A","DataFrame","containing","splitwise","train","and","test","evaluation","metrics","for","``","metric_dict","``","and","``","config_names","``",".","For","example",".","Let","'s","assume",":",":","metric_dict","=","{","``","RMSE","''",":","EvaluationMetricEnum.RootMeanSquaredError",",","``","Q_95","''",":","EvaluationMetricEnum.Quantile95","}","config_names","=","[","``","default_prophet","''",",","``","custom_silverkite","''","]","These","are","valid","``","config_names","``","and","there","are","2","splits","for","each",".","Then","evaluation_metrics_df","=","config_name","split_num","train_RMSE","test_RMSE","train_Q_95","test_Q_95","default_prophet","0","*","*","*","*","default_prophet","1","*","*","*","*","custom_silverkite","0","*","*","*","*","custom_silverkite","1","*","*","*","*","where","*","represents","computed","values","."]
code: def get_evaluation_metrics(
      self, metric_dict: Dict, config_names: List = None
  ):
      """Returns rolling train and test evaluation metric values.
  
      Parameters
      ----------
      metric_dict : `dict` [`str`, `callable`]
          Evaluation metrics to compute.
  
              - key: evaluation metric name, used to create column name in output.
              - value: metric function to apply to forecast df in each split to generate the column value.
                      Signature (y_true: `str`, y_pred: `str`) -> transformed value: `float`.
  
          For example::
  
              metric_dict = {
                  "median_residual": lambda y_true, y_pred: np.median(y_true - y_pred),
                  "mean_squared_error": lambda y_true, y_pred: np.mean((y_true - y_pred)**2)
              }
  
          Some predefined functions are available in
          `~greykite.common.evaluation`. For example::
  
              metric_dict = {
                  "correlation": lambda y_true, y_pred: correlation(y_true, y_pred),
                  "RMSE": lambda y_true, y_pred: root_mean_squared_error(y_true, y_pred),
                  "Q_95": lambda y_true, y_pred: partial(quantile_loss(y_true, y_pred, q=0.95))
              }
  
          As shorthand, it is sufficient to provide the corresponding ``EvaluationMetricEnum``
          member.  These are auto-expanded into the appropriate function.
          So the following is equivalent::
  
              metric_dict = {
                  "correlation": EvaluationMetricEnum.Correlation,
                  "RMSE": EvaluationMetricEnum.RootMeanSquaredError,
                  "Q_95": EvaluationMetricEnum.Quantile95
              }
  
      config_names : `list` [`str`], default None
          Which config results to plot. A list of config names.
          If None, uses all the available config keys.
  
      Returns
      -------
      evaluation_metrics_df : pd.DataFrame
          A DataFrame containing splitwise train and test evaluation metrics for ``metric_dict``
          and ``config_names``.
  
          For example. Let's assume::
  
              metric_dict = {
                  "RMSE": EvaluationMetricEnum.RootMeanSquaredError,
                  "Q_95": EvaluationMetricEnum.Quantile95
              }
  
              config_names = ["default_prophet", "custom_silverkite"]
              These are valid ``config_names`` and there are 2 splits for each.
  
              Then evaluation_metrics_df =
  
              config_name     split_num   train_RMSE  test_RMSE   train_Q_95  test_Q_95
              default_prophet      0          *           *           *           *
              default_prophet      1          *           *           *           *
              custom_silverkite    0          *           *           *           *
              custom_silverkite    1          *           *           *           *
  
              where * represents computed values.
      """
      if not self.is_run:
          raise ValueError(
              "Please execute the 'run' method before computing evaluation metrics."
          )
  
      metric_dict = self.autocomplete_metric_dict(
          metric_dict=metric_dict, enum_class=EvaluationMetricEnum
      )
  
      config_names = self.get_valid_config_names(
          config_names=config_names
      )
  
      evaluation_metrics_df = pd.DataFrame()
      for config_name in config_names:
          rolling_evaluation = self.result[config_name][
              "rolling_evaluation"
          ]
          for num, (split_key, split_value) in enumerate(
              rolling_evaluation.items()
          ):
              forecast = split_value["pipeline_result"].forecast
              split_metrics = {
                  "config_name": config_name,
                  "split_num": num,
              }
              # Updates train metrics
              df_train = forecast.df_train
              split_metrics.update(
                  {
                      f"train_{metric_name}": metric_func(
                          df_train[forecast.actual_col],
                          df_train[forecast.predicted_col],
                      )
                      for metric_name, metric_func in metric_dict.items()
                  }
              )
              # Updates test metrics
              df_test = forecast.df_test
              if (
                  df_test.shape[0] > 0
                  and forecast.test_na_count < df_test.shape[0]
              ):
                  split_metrics.update(
                      {
                          f"test_{metric_name}": metric_func(
                              df_test[forecast.actual_col],
                              df_test[forecast.predicted_col],
                          )
                          for metric_name, metric_func in metric_dict.items()
                      }
                  )
              else:
                  split_metrics.update(
                      {
                          f"test_{metric_name}": np.nan
                          for metric_name, metric_func in metric_dict.items()
                      }
                  )
  
              split_metrics_df = pd.DataFrame(
                  split_metrics, index=[num]
              )
              evaluation_metrics_df = pd.concat(
                  [evaluation_metrics_df, split_metrics_df]
              )
      # Resets index and fills missing values (e.g. when correlation is not defined) with np.nan
      evaluation_metrics_df = evaluation_metrics_df.reset_index(
          drop=True
      ).fillna(value=np.nan)
      temp_df = evaluation_metrics_df.copy()
      # Rearranges columns so that train and test error of a config are side by side
      evaluation_metrics_df = pd.DataFrame()
      evaluation_metrics_df["config_name"] = temp_df["config_name"]
      evaluation_metrics_df["split_num"] = temp_df["split_num"]
      for metric_name in metric_dict.keys():
          evaluation_metrics_df[f"train_{metric_name}"] = temp_df[
              f"train_{metric_name}"
          ]
          evaluation_metrics_df[f"test_{metric_name}"] = temp_df[
              f"test_{metric_name}"
          ]
  
      return evaluation_metrics_df
  
imports: from typing import Dict
  from typing import List
  import numpy
  import pandas
  import plotly.graph_objects
  from plotly.colors import DEFAULT_PLOTLY_COLORS
  from tqdm.autonotebook import tqdm
  from greykite.common.constants import ACTUAL_COL
  from greykite.common.constants import PREDICTED_COL
  from greykite.common.constants import PREDICTED_LOWER_COL
  from greykite.common.constants import PREDICTED_UPPER_COL
  from greykite.common.constants import TIME_COL
  from greykite.common.constants import VALUE_COL
  from greykite.common.evaluation import EvaluationMetricEnum
  from greykite.common.evaluation import add_finite_filter_to_scorer
  from greykite.common.logging import LoggingLevelEnum
  from greykite.common.logging import log_message
  from greykite.common.python_utils import get_pattern_cols
  from greykite.common.viz.timeseries_plotting import plot_multivariate
  from greykite.common.viz.timeseries_plotting import plot_multivariate_grouped
  from greykite.framework.benchmark.benchmark_class_helper import forecast_pipeline_rolling_evaluation
  from greykite.framework.constants import FORECAST_STEP_COL
  from greykite.framework.templates.autogen.forecast_config import ForecastConfig
  from greykite.framework.templates.forecaster import Forecaster
  from greykite.sklearn.cross_validation import RollingTimeSeriesSplit
label_desc: Use image node_id 7 for calling the BenchmarkForecastConfig obj's underlying member method code with example usage: obj.get_evaluation_metrics(metric_dict, config_names) and returns: evaluation_metrics_df