class_docstr: Stores predicted and actual values.
  Provides functionality to evaluate a forecast:
  
      - plots true against actual with prediction bands.
      - evaluates model performance.
  
  Input should be one of two kinds of forecast results:
  
      - model fit to train data, forecast on test set (actuals available).
      - model fit to all data, forecast on future dates (actuals not available).
  
  The input ``df`` is a concatenation of fitted and forecasted values.
  
  Attributes
  ----------
  df : `pandas.DataFrame`
      Timestamp, predicted, and actual values.
  time_col : `str`
      Column in ``df`` with timestamp (default "ts").
  actual_col : `str`
      Column in ``df`` with actual values (default "y").
  predicted_col : `str`
      Column in ``df`` with predicted values (default "forecast").
  predicted_lower_col : `str` or None
      Column in ``df`` with predicted lower bound (default "forecast_lower", optional).
  predicted_upper_col : `str` or None
      Column in ``df`` with predicted upper bound (default "forecast_upper", optional).
  null_model_predicted_col : `str` or None
      Column in ``df`` with predicted value of null model (default "forecast_null", optional).
  ylabel : `str`
      Unit of measurement (default "y")
  train_end_date : `str` or `datetime` or None, default None
      End date for train period. If `None`, assumes all data were used for training.
  test_start_date : `str` or `datetime` or None, default None
      Start date of test period. If `None`, set to the ``time_col`` value immediately after
      ``train_end_date``. This assumes that all data not used in training were used for testing.
  forecast_horizon : `int` or None, default None
      Number of periods forecasted into the future. Must be > 0.
  coverage : `float` or None
      Intended coverage of the prediction bands (0.0 to 1.0).
  r2_loss_function : `function`
      Loss function to calculate ``cst.R2_null_model_score``, with signature
      ``loss_func(y_true, y_pred)`` (default mean_squared_error)
  estimator : An instance of an estimator that implements `greykite.models.base_forecast_estimator.BaseForecastEstimator`.
      The fitted estimator, the last step in the forecast pipeline.
  relative_error_tolerance : `float` or None, default None
      Threshold to compute the ``Outside Tolerance`` metric,
      defined as the fraction of forecasted values whose relative
      error is strictly greater than ``relative_error_tolerance``.
      For example, 0.05 allows for 5% relative error.
      If `None`, the metric is not computed.
  df_train : `pandas.DataFrame`
      Subset of ``df`` where ``df[time_col]`` <= ``train_end_date``.
  df_test : `pandas.DataFrame`
      Subset of ``df`` where ``df[time_col]`` > ``train_end_date``.
  train_evaluation : `dict` [`str`, `float`]
      Evaluation metrics on training set.
  test_evaluation : `dict` [`str`, `float`]
      Evaluation metrics on test set (if actual values provided after train_end_date).
  test_na_count : `int`
      Count of NA values in test data.
docstr: Group-wise computation of evaluation metrics. Whereas ``self.get_grouping_evaluation``
  computes one metric, this allows computation of any number of custom metrics.
  
  For example:
  
      * Mean and quantiles of squared error by group.
      * Mean and quantiles of residuals by group.
      * Mean and quantiles of actual and forecast by group.
      * % of actuals outside prediction intervals by group
      * any combination of the above metrics by the same group
  
  First adds a groupby column by passing ``groupby_`` parameters to
  `~greykite.common.viz.timeseries_plotting.add_groupby_column`.
  Then computes grouped evaluation metrics by passing ``map_func_dict``,
  ``agg_kwargs`` and ``extend_col_names`` to
  `~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation`.
  
  Exactly one of: ``groupby_time_feature``, ``groupby_sliding_window_size``,
  ``groupby_custom_column`` must be provided.
  
  which: `str`
      "train" or "test". Which dataset to evaluate.
  groupby_time_feature : `str` or None, optional
      If provided, groups by a column generated by
      `~greykite.common.features.timeseries_features.build_time_features_df`.
      See that function for valid values.
  groupby_sliding_window_size : `int` or None, optional
      If provided, sequentially partitions data into groups of size
      ``groupby_sliding_window_size``.
  groupby_custom_column : `pandas.Series` or None, optional
      If provided, groups by this column value. Should be same length as the DataFrame.
  map_func_dict : `dict` [`str`, `callable`] or None, default None
      Row-wise transformation functions to create new columns.
      If None, no new columns are added.
  
          - key: new column name
          - value: row-wise function to apply to ``df`` to generate the column value.
                   Signature (row: `pandas.DataFrame`) -> transformed value: `float`.
  
      For example::
  
          map_func_dict = {
              "residual": lambda row: row["actual"] - row["forecast"],
              "squared_error": lambda row: (row["actual"] - row["forecast"])**2
          }
  
      Some predefined functions are available in
      `~greykite.common.evaluation.ElementwiseEvaluationMetricEnum`. For example::
  
          map_func_dict = {
              "residual": lambda row: ElementwiseEvaluationMetricEnum.Residual.get_metric_func()(
                  row["actual"],
                  row["forecast"]),
              "squared_error": lambda row: ElementwiseEvaluationMetricEnum.SquaredError.get_metric_func()(
                  row["actual"],
                  row["forecast"]),
              "q90_loss": lambda row: ElementwiseEvaluationMetricEnum.Quantile90.get_metric_func()(
                  row["actual"],
                  row["forecast"]),
              "abs_percent_error": lambda row: ElementwiseEvaluationMetricEnum.AbsolutePercentError.get_metric_func()(
                  row["actual"],
                  row["forecast"]),
              "coverage": lambda row: ElementwiseEvaluationMetricEnum.Coverage.get_metric_func()(
                  row["actual"],
                  row["forecast_lower"],
                  row["forecast_upper"]),
          }
  
      As shorthand, it is sufficient to provide the enum member name.  These are
      auto-expanded into the appropriate function.
      So the following is equivalent::
  
          map_func_dict = {
              "residual": ElementwiseEvaluationMetricEnum.Residual.name,
              "squared_error": ElementwiseEvaluationMetricEnum.SquaredError.name,
              "q90_loss": ElementwiseEvaluationMetricEnum.Quantile90.name,
              "abs_percent_error": ElementwiseEvaluationMetricEnum.AbsolutePercentError.name,
              "coverage": ElementwiseEvaluationMetricEnum.Coverage.name,
          }
  
  agg_kwargs : `dict` or None, default None
      Passed as keyword args to `pandas.core.groupby.DataFrameGroupBy.aggregate` after creating
      new columns and grouping by ``groupby_col``.
  
      See `pandas.core.groupby.DataFrameGroupBy.aggregate` or
      `~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation`
      for details.
  
  extend_col_names : `bool` or None, default False
      How to flatten index after aggregation.
      In some cases, the column index after aggregation is a multi-index.
      This parameter controls how to flatten an index with 2 levels to 1 level.
  
          - If None, the index is not flattened.
          - If True, column name is a composite: ``{index0}_{index1}``
            Use this option if index1 is not unique.
          - If False, column name is simply ``{index1}``
  
      Ignored if the ColumnIndex after aggregation has only one level (e.g.
      if named aggregation is used in ``agg_kwargs``).
  
  Returns
  -------
  df_transformed : `pandas.DataFrame`
      ``df`` after transformation and optional aggregation.
  
      If ``groupby_col`` is None, returns ``df`` with additional columns as the keys in ``map_func_dict``.
      Otherwise, ``df`` is grouped by ``groupby_col`` and this becomes the index. Columns
      are determined by ``agg_kwargs`` and ``extend_col_names``.
  
  See Also
  --------
  `~greykite.common.viz.timeseries_plotting.add_groupby_column` : called by this function
  `~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation` : called by this function
docstr_tok: ["Group-wise","computation","of","evaluation","metrics",".","Whereas","``","self.get_grouping_evaluation","``","computes","one","metric",",","this","allows","computation","of","any","number","of","custom","metrics",".","For","example",":","*","Mean","and","quantiles","of","squared","error","by","group",".","*","Mean","and","quantiles","of","residuals","by","group",".","*","Mean","and","quantiles","of","actual","and","forecast","by","group",".","*","%","of","actuals","outside","prediction","intervals","by","group","*","any","combination","of","the","above","metrics","by","the","same","group","First","adds","a","groupby","column","by","passing","``","groupby_","``","parameters","to","`","~greykite.common.viz.timeseries_plotting.add_groupby_column","`",".","Then","computes","grouped","evaluation","metrics","by","passing","``","map_func_dict","``",",","``","agg_kwargs","``","and","``","extend_col_names","``","to","`","~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation","`",".","Exactly","one","of",":","``","groupby_time_feature","``",",","``","groupby_sliding_window_size","``",",","``","groupby_custom_column","``","must","be","provided",".","which",":","`","str","`","``","train","''","or","``","test","''",".","Which","dataset","to","evaluate",".","groupby_time_feature",":","`","str","`","or","None",",","optional","If","provided",",","groups","by","a","column","generated","by","`","~greykite.common.features.timeseries_features.build_time_features_df","`",".","See","that","function","for","valid","values",".","groupby_sliding_window_size",":","`","int","`","or","None",",","optional","If","provided",",","sequentially","partitions","data","into","groups","of","size","``","groupby_sliding_window_size","``",".","groupby_custom_column",":","`","pandas.Series","`","or","None",",","optional","If","provided",",","groups","by","this","column","value",".","Should","be","same","length","as","the","DataFrame",".","map_func_dict",":","`","dict","`","[","`","str","`",",","`","callable","`","]","or","None",",","default","None","Row-wise","transformation","functions","to","create","new","columns",".","If","None",",","no","new","columns","are","added",".","-","key",":","new","column","name","-","value",":","row-wise","function","to","apply","to","``","df","``","to","generate","the","column","value",".","Signature","(","row",":","`","pandas.DataFrame","`",")","-",">","transformed","value",":","`","float","`",".","For","example",":",":","map_func_dict","=","{","``","residual","''",":","lambda","row",":","row","[","``","actual","''","]","-","row","[","``","forecast","''","]",",","``","squared_error","''",":","lambda","row",":","(","row","[","``","actual","''","]","-","row","[","``","forecast","''","]",")","*","*","2","}","Some","predefined","functions","are","available","in","`","~greykite.common.evaluation.ElementwiseEvaluationMetricEnum","`",".","For","example",":",":","map_func_dict","=","{","``","residual","''",":","lambda","row",":","ElementwiseEvaluationMetricEnum.Residual.get_metric_func","(",")","(","row","[","``","actual","''","]",",","row","[","``","forecast","''","]",")",",","``","squared_error","''",":","lambda","row",":","ElementwiseEvaluationMetricEnum.SquaredError.get_metric_func","(",")","(","row","[","``","actual","''","]",",","row","[","``","forecast","''","]",")",",","``","q90_loss","''",":","lambda","row",":","ElementwiseEvaluationMetricEnum.Quantile90.get_metric_func","(",")","(","row","[","``","actual","''","]",",","row","[","``","forecast","''","]",")",",","``","abs_percent_error","''",":","lambda","row",":","ElementwiseEvaluationMetricEnum.AbsolutePercentError.get_metric_func","(",")","(","row","[","``","actual","''","]",",","row","[","``","forecast","''","]",")",",","``","coverage","''",":","lambda","row",":","ElementwiseEvaluationMetricEnum.Coverage.get_metric_func","(",")","(","row","[","``","actual","''","]",",","row","[","``","forecast_lower","''","]",",","row","[","``","forecast_upper","''","]",")",",","}","As","shorthand",",","it","is","sufficient","to","provide","the","enum","member","name",".","These","are","auto-expanded","into","the","appropriate","function",".","So","the","following","is","equivalent",":",":","map_func_dict","=","{","``","residual","''",":","ElementwiseEvaluationMetricEnum.Residual.name",",","``","squared_error","''",":","ElementwiseEvaluationMetricEnum.SquaredError.name",",","``","q90_loss","''",":","ElementwiseEvaluationMetricEnum.Quantile90.name",",","``","abs_percent_error","''",":","ElementwiseEvaluationMetricEnum.AbsolutePercentError.name",",","``","coverage","''",":","ElementwiseEvaluationMetricEnum.Coverage.name",",","}","agg_kwargs",":","`","dict","`","or","None",",","default","None","Passed","as","keyword","args","to","`","pandas.core.groupby.DataFrameGroupBy.aggregate","`","after","creating","new","columns","and","grouping","by","``","groupby_col","``",".","See","`","pandas.core.groupby.DataFrameGroupBy.aggregate","`","or","`","~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation","`","for","details",".","extend_col_names",":","`","bool","`","or","None",",","default","False","How","to","flatten","index","after","aggregation",".","In","some","cases",",","the","column","index","after","aggregation","is","a","multi-index",".","This","parameter","controls","how","to","flatten","an","index","with","2","levels","to","1","level",".","-","If","None",",","the","index","is","not","flattened",".","-","If","True",",","column","name","is","a","composite",":","``","{","index0","}","_","{","index1","}","``","Use","this","option","if","index1","is","not","unique",".","-","If","False",",","column","name","is","simply","``","{","index1","}","``","Ignored","if","the","ColumnIndex","after","aggregation","has","only","one","level","(","e.g",".","if","named","aggregation","is","used","in","``","agg_kwargs","``",")",".","Returns","--","--","--","-","df_transformed",":","`","pandas.DataFrame","`","``","df","``","after","transformation","and","optional","aggregation",".","If","``","groupby_col","``","is","None",",","returns","``","df","``","with","additional","columns","as","the","keys","in","``","map_func_dict","``",".","Otherwise",",","``","df","``","is","grouped","by","``","groupby_col","``","and","this","becomes","the","index",".","Columns","are","determined","by","``","agg_kwargs","``","and","``","extend_col_names","``",".","See","Also","--","--","--","--","`","~greykite.common.viz.timeseries_plotting.add_groupby_column","`",":","called","by","this","function","`","~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation","`",":","called","by","this","function"]
code: def get_flexible_grouping_evaluation(
      self,
      which="train",
      groupby_time_feature=None,
      groupby_sliding_window_size=None,
      groupby_custom_column=None,
      map_func_dict=None,
      agg_kwargs=None,
      extend_col_names=False,
  ):
      """Group-wise computation of evaluation metrics. Whereas ``self.get_grouping_evaluation``
      computes one metric, this allows computation of any number of custom metrics.
  
      For example:
  
          * Mean and quantiles of squared error by group.
          * Mean and quantiles of residuals by group.
          * Mean and quantiles of actual and forecast by group.
          * % of actuals outside prediction intervals by group
          * any combination of the above metrics by the same group
  
      First adds a groupby column by passing ``groupby_`` parameters to
      `~greykite.common.viz.timeseries_plotting.add_groupby_column`.
      Then computes grouped evaluation metrics by passing ``map_func_dict``,
      ``agg_kwargs`` and ``extend_col_names`` to
      `~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation`.
  
      Exactly one of: ``groupby_time_feature``, ``groupby_sliding_window_size``,
      ``groupby_custom_column`` must be provided.
  
      which: `str`
          "train" or "test". Which dataset to evaluate.
      groupby_time_feature : `str` or None, optional
          If provided, groups by a column generated by
          `~greykite.common.features.timeseries_features.build_time_features_df`.
          See that function for valid values.
      groupby_sliding_window_size : `int` or None, optional
          If provided, sequentially partitions data into groups of size
          ``groupby_sliding_window_size``.
      groupby_custom_column : `pandas.Series` or None, optional
          If provided, groups by this column value. Should be same length as the DataFrame.
      map_func_dict : `dict` [`str`, `callable`] or None, default None
          Row-wise transformation functions to create new columns.
          If None, no new columns are added.
  
              - key: new column name
              - value: row-wise function to apply to ``df`` to generate the column value.
                       Signature (row: `pandas.DataFrame`) -> transformed value: `float`.
  
          For example::
  
              map_func_dict = {
                  "residual": lambda row: row["actual"] - row["forecast"],
                  "squared_error": lambda row: (row["actual"] - row["forecast"])**2
              }
  
          Some predefined functions are available in
          `~greykite.common.evaluation.ElementwiseEvaluationMetricEnum`. For example::
  
              map_func_dict = {
                  "residual": lambda row: ElementwiseEvaluationMetricEnum.Residual.get_metric_func()(
                      row["actual"],
                      row["forecast"]),
                  "squared_error": lambda row: ElementwiseEvaluationMetricEnum.SquaredError.get_metric_func()(
                      row["actual"],
                      row["forecast"]),
                  "q90_loss": lambda row: ElementwiseEvaluationMetricEnum.Quantile90.get_metric_func()(
                      row["actual"],
                      row["forecast"]),
                  "abs_percent_error": lambda row: ElementwiseEvaluationMetricEnum.AbsolutePercentError.get_metric_func()(
                      row["actual"],
                      row["forecast"]),
                  "coverage": lambda row: ElementwiseEvaluationMetricEnum.Coverage.get_metric_func()(
                      row["actual"],
                      row["forecast_lower"],
                      row["forecast_upper"]),
              }
  
          As shorthand, it is sufficient to provide the enum member name.  These are
          auto-expanded into the appropriate function.
          So the following is equivalent::
  
              map_func_dict = {
                  "residual": ElementwiseEvaluationMetricEnum.Residual.name,
                  "squared_error": ElementwiseEvaluationMetricEnum.SquaredError.name,
                  "q90_loss": ElementwiseEvaluationMetricEnum.Quantile90.name,
                  "abs_percent_error": ElementwiseEvaluationMetricEnum.AbsolutePercentError.name,
                  "coverage": ElementwiseEvaluationMetricEnum.Coverage.name,
              }
  
      agg_kwargs : `dict` or None, default None
          Passed as keyword args to `pandas.core.groupby.DataFrameGroupBy.aggregate` after creating
          new columns and grouping by ``groupby_col``.
  
          See `pandas.core.groupby.DataFrameGroupBy.aggregate` or
          `~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation`
          for details.
  
      extend_col_names : `bool` or None, default False
          How to flatten index after aggregation.
          In some cases, the column index after aggregation is a multi-index.
          This parameter controls how to flatten an index with 2 levels to 1 level.
  
              - If None, the index is not flattened.
              - If True, column name is a composite: ``{index0}_{index1}``
                Use this option if index1 is not unique.
              - If False, column name is simply ``{index1}``
  
          Ignored if the ColumnIndex after aggregation has only one level (e.g.
          if named aggregation is used in ``agg_kwargs``).
  
      Returns
      -------
      df_transformed : `pandas.DataFrame`
          ``df`` after transformation and optional aggregation.
  
          If ``groupby_col`` is None, returns ``df`` with additional columns as the keys in ``map_func_dict``.
          Otherwise, ``df`` is grouped by ``groupby_col`` and this becomes the index. Columns
          are determined by ``agg_kwargs`` and ``extend_col_names``.
  
      See Also
      --------
      `~greykite.common.viz.timeseries_plotting.add_groupby_column` : called by this function
      `~greykite.common.viz.timeseries_plotting.flexible_grouping_evaluation` : called by this function
      """
      df = self.df_train if which.lower() == "train" else self.df_test
      df = df.copy()
      result = add_groupby_column(
          df=df,
          time_col=self.time_col,
          groupby_time_feature=groupby_time_feature,
          groupby_sliding_window_size=groupby_sliding_window_size,
          groupby_custom_column=groupby_custom_column,
      )
      df = result["df"]
  
      map_func_dict = self.autocomplete_map_func_dict(map_func_dict)
      grouped_df = flexible_grouping_evaluation(
          df,
          map_func_dict=map_func_dict,
          groupby_col=result["groupby_col"],
          agg_kwargs=agg_kwargs,
          extend_col_names=extend_col_names,
      )
  
      return grouped_df
  
imports: from functools import partial
  import pandas
  from sklearn.metrics import mean_squared_error
  from greykite.common import constants
  from greykite.common.evaluation import ElementwiseEvaluationMetricEnum
  from greykite.common.evaluation import EvaluationMetricEnum
  from greykite.common.evaluation import add_finite_filter_to_scorer
  from greykite.common.evaluation import calc_pred_coverage
  from greykite.common.evaluation import calc_pred_err
  from greykite.common.evaluation import fraction_outside_tolerance
  from greykite.common.evaluation import r2_null_model_score
  from greykite.common.python_utils import apply_func_to_columns
  from greykite.common.time_properties import infer_freq
  from greykite.common.viz.timeseries_plotting import add_groupby_column
  from greykite.common.viz.timeseries_plotting import flexible_grouping_evaluation
  from greykite.common.viz.timeseries_plotting import grouping_evaluation
  from greykite.common.viz.timeseries_plotting import plot_forecast_vs_actual
  from greykite.common.viz.timeseries_plotting import plot_multivariate
  from greykite.common.viz.timeseries_plotting import plot_univariate
  from greykite.framework.input.univariate_time_series import UnivariateTimeSeries
label_desc: Use image node_id 8 for calling the UnivariateForecast obj's underlying member method code with example usage: obj.get_flexible_grouping_evaluation(which, groupby_time_feature, groupby_sliding_window_size, groupby_custom_column, map_func_dict, agg_kwargs, extend_col_names) and returns: grouped_df