class_docstr: Stores predicted and actual values.
  Provides functionality to evaluate a forecast:
  
      - plots true against actual with prediction bands.
      - evaluates model performance.
  
  Input should be one of two kinds of forecast results:
  
      - model fit to train data, forecast on test set (actuals available).
      - model fit to all data, forecast on future dates (actuals not available).
  
  The input ``df`` is a concatenation of fitted and forecasted values.
  
  Attributes
  ----------
  df : `pandas.DataFrame`
      Timestamp, predicted, and actual values.
  time_col : `str`
      Column in ``df`` with timestamp (default "ts").
  actual_col : `str`
      Column in ``df`` with actual values (default "y").
  predicted_col : `str`
      Column in ``df`` with predicted values (default "forecast").
  predicted_lower_col : `str` or None
      Column in ``df`` with predicted lower bound (default "forecast_lower", optional).
  predicted_upper_col : `str` or None
      Column in ``df`` with predicted upper bound (default "forecast_upper", optional).
  null_model_predicted_col : `str` or None
      Column in ``df`` with predicted value of null model (default "forecast_null", optional).
  ylabel : `str`
      Unit of measurement (default "y")
  train_end_date : `str` or `datetime` or None, default None
      End date for train period. If `None`, assumes all data were used for training.
  test_start_date : `str` or `datetime` or None, default None
      Start date of test period. If `None`, set to the ``time_col`` value immediately after
      ``train_end_date``. This assumes that all data not used in training were used for testing.
  forecast_horizon : `int` or None, default None
      Number of periods forecasted into the future. Must be > 0.
  coverage : `float` or None
      Intended coverage of the prediction bands (0.0 to 1.0).
  r2_loss_function : `function`
      Loss function to calculate ``cst.R2_null_model_score``, with signature
      ``loss_func(y_true, y_pred)`` (default mean_squared_error)
  estimator : An instance of an estimator that implements `greykite.models.base_forecast_estimator.BaseForecastEstimator`.
      The fitted estimator, the last step in the forecast pipeline.
  relative_error_tolerance : `float` or None, default None
      Threshold to compute the ``Outside Tolerance`` metric,
      defined as the fraction of forecasted values whose relative
      error is strictly greater than ``relative_error_tolerance``.
      For example, 0.05 allows for 5% relative error.
      If `None`, the metric is not computed.
  df_train : `pandas.DataFrame`
      Subset of ``df`` where ``df[time_col]`` <= ``train_end_date``.
  df_test : `pandas.DataFrame`
      Subset of ``df`` where ``df[time_col]`` > ``train_end_date``.
  train_evaluation : `dict` [`str`, `float`]
      Evaluation metrics on training set.
  test_evaluation : `dict` [`str`, `float`]
      Evaluation metrics on test set (if actual values provided after train_end_date).
  test_na_count : `int`
      Count of NA values in test data.
docstr: Plots group-wise evaluation metrics. Whereas
  `~greykite.framework.output.univariate_forecast.UnivariateForecast.plot_grouping_evaluation`
  shows one metric, this can show any number of custom metrics.
  
  For example:
  
      * Mean and quantiles of squared error by group.
      * Mean and quantiles of residuals by group.
      * Mean and quantiles of actual and forecast by group.
      * % of actuals outside prediction intervals by group
      * any combination of the above metrics by the same group
  
  See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`
  for details.
  
  which: `str`
      "train" or "test". Which dataset to evaluate.
  groupby_time_feature : `str` or None, optional
      If provided, groups by a column generated by
      `~greykite.common.features.timeseries_features.build_time_features_df`.
      See that function for valid values.
  groupby_sliding_window_size : `int` or None, optional
      If provided, sequentially partitions data into groups of size
      ``groupby_sliding_window_size``.
  groupby_custom_column : `pandas.Series` or None, optional
      If provided, groups by this column value. Should be same length as the DataFrame.
  map_func_dict : `dict` [`str`, `callable`] or None, default None
      Grouping evaluation metric specification, along with ``agg_kwargs``.
      See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`.
  agg_kwargs : `dict` or None, default None
      Grouping evaluation metric specification, along with ``map_func_dict``.
      See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`.
  extend_col_names : `bool` or None, default False
      How to name the grouping metrics.
      See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`.
  y_col_style_dict: `dict` [`str`, `dict` or None] or "plotly" or "auto" or "auto-fill", default "auto-fill"
      The column(s) to plot on the y-axis, and how to style them. The names should match
      those generated by ``agg_kwargs`` and ``extend_col_names``.
      The function
      `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`
      can be used to check the column names.
  
      For convenience, start with "auto-fill" or "plotly", then adjust styling as needed.
  
      See `~greykite.common.viz.timeseries_plotting.plot_multivariate` for details.
  
  default_color: `str`, default "rgba(0, 145, 202, 1.0)" (blue)
      Default line color when ``y_col_style_dict`` is one of "auto", "auto-fill".
  xlabel : `str` or None, default None
      x-axis label. If None, default is ``x_col``.
  ylabel : `str` or None, default None
      y-axis label. If None, y-axis is not labeled.
  title : `str` or None, default None
      Plot title. If None and ``ylabel`` is provided, a default title is used.
  showlegend : `bool`, default True
      Whether to show the legend.
  
  Returns
  -------
  fig : `plotly.graph_objects.Figure`
      Interactive plotly graph showing the evaluation metrics.
  
      See `~greykite.common.viz.timeseries_plotting.plot_forecast_vs_actual`
      return value for how to plot the figure and add customization.
  
  See Also
  --------
  `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation` : called by this function
  `~greykite.common.viz.timeseries_plotting.plot_multivariate` : called by this function
docstr_tok: ["Plots","group-wise","evaluation","metrics",".","Whereas","`","~greykite.framework.output.univariate_forecast.UnivariateForecast.plot_grouping_evaluation","`","shows","one","metric",",","this","can","show","any","number","of","custom","metrics",".","For","example",":","*","Mean","and","quantiles","of","squared","error","by","group",".","*","Mean","and","quantiles","of","residuals","by","group",".","*","Mean","and","quantiles","of","actual","and","forecast","by","group",".","*","%","of","actuals","outside","prediction","intervals","by","group","*","any","combination","of","the","above","metrics","by","the","same","group","See","`","~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation","`","for","details",".","which",":","`","str","`","``","train","''","or","``","test","''",".","Which","dataset","to","evaluate",".","groupby_time_feature",":","`","str","`","or","None",",","optional","If","provided",",","groups","by","a","column","generated","by","`","~greykite.common.features.timeseries_features.build_time_features_df","`",".","See","that","function","for","valid","values",".","groupby_sliding_window_size",":","`","int","`","or","None",",","optional","If","provided",",","sequentially","partitions","data","into","groups","of","size","``","groupby_sliding_window_size","``",".","groupby_custom_column",":","`","pandas.Series","`","or","None",",","optional","If","provided",",","groups","by","this","column","value",".","Should","be","same","length","as","the","DataFrame",".","map_func_dict",":","`","dict","`","[","`","str","`",",","`","callable","`","]","or","None",",","default","None","Grouping","evaluation","metric","specification",",","along","with","``","agg_kwargs","``",".","See","`","~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation","`",".","agg_kwargs",":","`","dict","`","or","None",",","default","None","Grouping","evaluation","metric","specification",",","along","with","``","map_func_dict","``",".","See","`","~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation","`",".","extend_col_names",":","`","bool","`","or","None",",","default","False","How","to","name","the","grouping","metrics",".","See","`","~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation","`",".","y_col_style_dict",":","`","dict","`","[","`","str","`",",","`","dict","`","or","None","]","or","``","plotly","''","or","``","auto","''","or","``","auto-fill","''",",","default","``","auto-fill","''","The","column","(","s",")","to","plot","on","the","y-axis",",","and","how","to","style","them",".","The","names","should","match","those","generated","by","``","agg_kwargs","``","and","``","extend_col_names","``",".","The","function","`","~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation","`","can","be","used","to","check","the","column","names",".","For","convenience",",","start","with","``","auto-fill","''","or","``","plotly","''",",","then","adjust","styling","as","needed",".","See","`","~greykite.common.viz.timeseries_plotting.plot_multivariate","`","for","details",".","default_color",":","`","str","`",",","default","``","rgba","(","0",",","145",",","202",",","1.0",")","''","(","blue",")","Default","line","color","when","``","y_col_style_dict","``","is","one","of","``","auto","''",",","``","auto-fill","''",".","xlabel",":","`","str","`","or","None",",","default","None","x-axis","label",".","If","None",",","default","is","``","x_col","``",".","ylabel",":","`","str","`","or","None",",","default","None","y-axis","label",".","If","None",",","y-axis","is","not","labeled",".","title",":","`","str","`","or","None",",","default","None","Plot","title",".","If","None","and","``","ylabel","``","is","provided",",","a","default","title","is","used",".","showlegend",":","`","bool","`",",","default","True","Whether","to","show","the","legend",".","Returns","--","--","--","-","fig",":","`","plotly.graph_objects.Figure","`","Interactive","plotly","graph","showing","the","evaluation","metrics",".","See","`","~greykite.common.viz.timeseries_plotting.plot_forecast_vs_actual","`","return","value","for","how","to","plot","the","figure","and","add","customization",".","See","Also","--","--","--","--","`","~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation","`",":","called","by","this","function","`","~greykite.common.viz.timeseries_plotting.plot_multivariate","`",":","called","by","this","function"]
code: def plot_flexible_grouping_evaluation(
      self,
      which="train",
      groupby_time_feature=None,
      groupby_sliding_window_size=None,
      groupby_custom_column=None,
      map_func_dict=None,
      agg_kwargs=None,
      extend_col_names=False,
      y_col_style_dict="auto-fill",
      default_color="rgba(0, 145, 202, 1.0)",
      xlabel=None,
      ylabel=None,
      title=None,
      showlegend=True,
  ):
      """Plots group-wise evaluation metrics. Whereas
      `~greykite.framework.output.univariate_forecast.UnivariateForecast.plot_grouping_evaluation`
      shows one metric, this can show any number of custom metrics.
  
      For example:
  
          * Mean and quantiles of squared error by group.
          * Mean and quantiles of residuals by group.
          * Mean and quantiles of actual and forecast by group.
          * % of actuals outside prediction intervals by group
          * any combination of the above metrics by the same group
  
      See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`
      for details.
  
      which: `str`
          "train" or "test". Which dataset to evaluate.
      groupby_time_feature : `str` or None, optional
          If provided, groups by a column generated by
          `~greykite.common.features.timeseries_features.build_time_features_df`.
          See that function for valid values.
      groupby_sliding_window_size : `int` or None, optional
          If provided, sequentially partitions data into groups of size
          ``groupby_sliding_window_size``.
      groupby_custom_column : `pandas.Series` or None, optional
          If provided, groups by this column value. Should be same length as the DataFrame.
      map_func_dict : `dict` [`str`, `callable`] or None, default None
          Grouping evaluation metric specification, along with ``agg_kwargs``.
          See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`.
      agg_kwargs : `dict` or None, default None
          Grouping evaluation metric specification, along with ``map_func_dict``.
          See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`.
      extend_col_names : `bool` or None, default False
          How to name the grouping metrics.
          See `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`.
      y_col_style_dict: `dict` [`str`, `dict` or None] or "plotly" or "auto" or "auto-fill", default "auto-fill"
          The column(s) to plot on the y-axis, and how to style them. The names should match
          those generated by ``agg_kwargs`` and ``extend_col_names``.
          The function
          `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation`
          can be used to check the column names.
  
          For convenience, start with "auto-fill" or "plotly", then adjust styling as needed.
  
          See `~greykite.common.viz.timeseries_plotting.plot_multivariate` for details.
  
      default_color: `str`, default "rgba(0, 145, 202, 1.0)" (blue)
          Default line color when ``y_col_style_dict`` is one of "auto", "auto-fill".
      xlabel : `str` or None, default None
          x-axis label. If None, default is ``x_col``.
      ylabel : `str` or None, default None
          y-axis label. If None, y-axis is not labeled.
      title : `str` or None, default None
          Plot title. If None and ``ylabel`` is provided, a default title is used.
      showlegend : `bool`, default True
          Whether to show the legend.
  
      Returns
      -------
      fig : `plotly.graph_objects.Figure`
          Interactive plotly graph showing the evaluation metrics.
  
          See `~greykite.common.viz.timeseries_plotting.plot_forecast_vs_actual`
          return value for how to plot the figure and add customization.
  
      See Also
      --------
      `~greykite.framework.output.univariate_forecast.UnivariateForecast.get_flexible_grouping_evaluation` : called by this function
      `~greykite.common.viz.timeseries_plotting.plot_multivariate` : called by this function
      """
      grouped_df = self.get_flexible_grouping_evaluation(
          which=which,
          groupby_time_feature=groupby_time_feature,
          groupby_sliding_window_size=groupby_sliding_window_size,
          groupby_custom_column=groupby_custom_column,
          map_func_dict=map_func_dict,
          agg_kwargs=agg_kwargs,
          extend_col_names=extend_col_names,
      )
  
      x_col = grouped_df.index.name
      grouped_df.reset_index(inplace=True)
      fig = plot_multivariate(
          grouped_df,
          x_col=x_col,
          y_col_style_dict=y_col_style_dict,
          default_color=default_color,
          xlabel=xlabel,
          ylabel=ylabel,
          title=title,
          showlegend=showlegend,
      )
      return fig
  
imports: from functools import partial
  import pandas
  from sklearn.metrics import mean_squared_error
  from greykite.common import constants
  from greykite.common.evaluation import ElementwiseEvaluationMetricEnum
  from greykite.common.evaluation import EvaluationMetricEnum
  from greykite.common.evaluation import add_finite_filter_to_scorer
  from greykite.common.evaluation import calc_pred_coverage
  from greykite.common.evaluation import calc_pred_err
  from greykite.common.evaluation import fraction_outside_tolerance
  from greykite.common.evaluation import r2_null_model_score
  from greykite.common.python_utils import apply_func_to_columns
  from greykite.common.time_properties import infer_freq
  from greykite.common.viz.timeseries_plotting import add_groupby_column
  from greykite.common.viz.timeseries_plotting import flexible_grouping_evaluation
  from greykite.common.viz.timeseries_plotting import grouping_evaluation
  from greykite.common.viz.timeseries_plotting import plot_forecast_vs_actual
  from greykite.common.viz.timeseries_plotting import plot_multivariate
  from greykite.common.viz.timeseries_plotting import plot_univariate
  from greykite.framework.input.univariate_time_series import UnivariateTimeSeries
label_desc: Use image node_id 9 for calling the UnivariateForecast obj's underlying member method code with example usage: obj.plot_flexible_grouping_evaluation(which, groupby_time_feature, groupby_sliding_window_size, groupby_custom_column, map_func_dict, agg_kwargs, extend_col_names, y_col_style_dict, default_color, xlabel, ylabel, title, showlegend) and returns: fig