tabular/src/autogluon/tabular/task/tabular_prediction/predictor.py
def feature_importance(self, dataset=None, model=None, features=None, feature_stage='original', subsample_size=1000, time_limit=None, num_shuffle_sets=None, include_confidence_band=True, silent=False):
        """
        Calculates feature importance scores for the given model via permutation importance. Refer to https://explained.ai/rf-importance/ for an explanation of permutation importance.
        A feature's importance score represents the performance drop that results when the model makes predictions on a perturbed copy of the dataset where this feature's values have been randomly shuffled across rows.
        A feature score of 0.01 would indicate that the predictive performance dropped by 0.01 when the feature was randomly shuffled.
        The higher the score a feature has, the more important it is to the model's performance.
        If a feature has a negative score, this means that the feature is likely harmful to the final model, and a model trained with the feature removed would be expected to achieve a better predictive performance.
        Note that calculating feature importance can be a very computationally expensive process, particularly if the model uses hundreds or thousands of features. In many cases, this can take longer than the original model training.
        To estimate how long `feature_importance(model, dataset, features)` will take, it is roughly the time taken by `predict_proba(dataset, model)` multiplied by the number of features.

        Note: For highly accurate importance and p_value estimates, it is recommend to set `subsample_size` to at least 5,000 if possible and `num_shuffle_sets` to at least 10.

        Parameters
        ----------
        dataset : str or :class:`TabularDataset` or `pandas.DataFrame` (optional)
            This dataset must also contain the label-column with the same column-name as specified during `fit()`.
            If specified, then the dataset is used to calculate the feature importance scores.
            If str is passed, `dataset` will be loaded using the str value as the file path.
            If not specified, the original dataset used during `fit()` will be used if `cache_data=True`. Otherwise, an exception will be raised.
            Do not pass the training data through this argument, as the feature importance scores calculated will be biased due to overfitting.
                More accurate feature importances will be obtained from new data that was held-out during `fit()`.
        model : str, default = None
            Model to get feature importances for, if None the best model is chosen.
            Valid models are listed in this `predictor` by calling `predictor.get_model_names()`
        features : list, default = None
            List of str feature names that feature importances are calculated for and returned, specify None to get all feature importances.
            If you only want to compute feature importances for some of the features, you can pass their names in as a list of str.
            Valid feature names change depending on the `feature_stage`.
                To get the list of feature names for `feature_stage='transformed'`, call `list(predictor.transform_features().columns)`.
                To get the list of feature names for `feature_stage=`transformed_model`, call `list(predictor.transform_features(model={model_name}).columns)`.
        feature_stage : str, default = 'original'
            What stage of feature-processing should importances be computed for.
            Options:
                'original':
                    Compute importances of the original features.
                    Warning: `dataset` must be specified with this option, otherwise an exception will be raised.
                'transformed':
                    Compute importances of the post-internal-transformation features (after automated feature engineering). These features may be missing some original features, or add new features entirely.
                    An example of new features would be ngram features generated from a text column.
                    Warning: For bagged models, feature importance calculation is not yet supported with this option when `dataset=None`. Doing so will raise an exception.
                'transformed_model':
                    Compute importances of the post-model-transformation features. These features are the internal features used by the requested model. They may differ greatly from the original features.
                    If the model is a stack ensemble, this will include stack ensemble features such as the prediction probability features of the stack ensemble's base (ancestor) models.
        subsample_size : int, default = 1000
            The number of rows to sample from `dataset` when computing feature importance.
            If `subsample_size=None` or `dataset` contains fewer than `subsample_size` rows, all rows will be used during computation.
            Larger values increase the accuracy of the feature importance scores.
            Runtime linearly scales with `subsample_size`.
        time_limit : float, default = None
            Time in seconds to limit the calculation of feature importance.
            If None, feature importance will calculate without early stopping.
            A minimum of 1 full shuffle set will always be evaluated. If a shuffle set evaluation takes longer than `time_limit`, the method will take the length of a shuffle set evaluation to return regardless of the `time_limit`.
        num_shuffle_sets : int, default = None
            The number of different permutation shuffles of the data that are evaluated.
            Larger values will increase the quality of the importance evaluation.
            It is generally recommended to increase `subsample_size` before increasing `num_shuffle_sets`.
            Defaults to 3 if `time_limit` is None or 10 if `time_limit` is specified.
            Runtime linearly scales with `num_shuffle_sets`.
        include_confidence_band: bool, default = True
            If True, will include output columns 'p99_high' and 'p99_low' which indicates that the true feature importance will be between 'p99_high' and 'p99_low' 99% of the time (99% confidence interval).
            Increasing `subsample_size` and `num_shuffle_sets` will tighten the band.
        silent : bool, default = False
            Whether to suppress logging output.

        Returns
        -------
        Pandas `pandas.DataFrame` of feature importance scores with 6 columns:
            index: The feature name.
            'importance': The estimated feature importance score.
            'stddev': The standard deviation of the feature importance score. If NaN, then not enough num_shuffle_sets were used to calculate a variance.
            'p_value': P-value for a statistical t-test of the null hypothesis: importance = 0, vs the (one-sided) alternative: importance > 0.
                Features with low p-value appear confidently useful to the predictor, while the other features may be useless to the predictor (or even harmful to include in its training data).
                A p-value of 0.01 indicates that there is a 1% chance that the feature is useless or harmful, and a 99% chance that the feature is useful.
                A p-value of 0.99 indicates that there is a 99% chance that the feature is useless or harmful, and a 1% chance that the feature is useful.
            'n': The number of shuffles performed to estimate importance score (corresponds to sample-size used to determine confidence interval for true score).
            'p99_high': Upper end of 99% confidence interval for true feature importance score.
            'p99_low': Lower end of 99% confidence interval for true feature importance score.
        """
        dataset = self.__get_dataset(dataset) if dataset is not None else dataset
        if (dataset is None) and (not self._trainer.is_data_saved):
            raise AssertionError('No dataset was provided and there is no cached data to load for feature importance calculation. `cache_data=True` must be set in the `TabularPrediction.fit()` call to enable this functionality when dataset is not specified.')

        if num_shuffle_sets is None:
            num_shuffle_sets = 10 if time_limit else 3

        fi_df = self._learner.get_feature_importance(model=model, X=dataset, features=features, feature_stage=feature_stage,
                                                    subsample_size=subsample_size, time_limit=time_limit, num_shuffle_sets=num_shuffle_sets, silent=silent)

        if include_confidence_band:
            import scipy.stats
            num_features = len(fi_df)
            confidence_threshold = 0.99
            p99_low_dict = dict()
            p99_high_dict = dict()
            for i in range(num_features):
                fi = fi_df.iloc[i]
                mean = fi['importance']
                stddev = fi['stddev']
                n = fi['n']
                if stddev == np.nan or n == np.nan or mean == np.nan or n == 1:
                    p99_high = np.nan
                    p99_low = np.nan
                else:
                    t_val_99 = scipy.stats.t.ppf(1 - (1 - confidence_threshold) / 2, n-1)
                    p99_high = mean + t_val_99 * stddev / math.sqrt(n)
                    p99_low = mean - t_val_99 * stddev / math.sqrt(n)
                p99_high_dict[fi.name] = p99_high
                p99_low_dict[fi.name] = p99_low
            fi_df['p99_high'] = pd.Series(p99_high_dict)
            fi_df['p99_low'] = pd.Series(p99_low_dict)
        return fi_df

        Calculates feature importance scores for the given model via permutation importance. Refer to https://explained.ai/rf-importance/ for an explanation of permutation importance.
        A feature's importance score represents the performance drop that results when the model makes predictions on a perturbed copy of the dataset where this feature's values have been randomly shuffled across rows.
        A feature score of 0.01 would indicate that the predictive performance dropped by 0.01 when the feature was randomly shuffled.
        The higher the score a feature has, the more important it is to the model's performance.
        If a feature has a negative score, this means that the feature is likely harmful to the final model, and a model trained with the feature removed would be expected to achieve a better predictive performance.
        Note that calculating feature importance can be a very computationally expensive process, particularly if the model uses hundreds or thousands of features. In many cases, this can take longer than the original model training.
        To estimate how long `feature_importance(model, dataset, features)` will take, it is roughly the time taken by `predict_proba(dataset, model)` multiplied by the number of features.

        Note: For highly accurate importance and p_value estimates, it is recommend to set `subsample_size` to at least 5,000 if possible and `num_shuffle_sets` to at least 10.

        Parameters
        ----------
        dataset : str or :class:`TabularDataset` or `pandas.DataFrame` (optional)
            This dataset must also contain the label-column with the same column-name as specified during `fit()`.
            If specified, then the dataset is used to calculate the feature importance scores.
            If str is passed, `dataset` will be loaded using the str value as the file path.
            If not specified, the original dataset used during `fit()` will be used if `cache_data=True`. Otherwise, an exception will be raised.
            Do not pass the training data through this argument, as the feature importance scores calculated will be biased due to overfitting.
                More accurate feature importances will be obtained from new data that was held-out during `fit()`.
        model : str, default = None
            Model to get feature importances for, if None the best model is chosen.
            Valid models are listed in this `predictor` by calling `predictor.get_model_names()`
        features : list, default = None
            List of str feature names that feature importances are calculated for and returned, specify None to get all feature importances.
            If you only want to compute feature importances for some of the features, you can pass their names in as a list of str.
            Valid feature names change depending on the `feature_stage`.
                To get the list of feature names for `feature_stage='transformed'`, call `list(predictor.transform_features().columns)`.
                To get the list of feature names for `feature_stage=`transformed_model`, call `list(predictor.transform_features(model={model_name}).columns)`.
        feature_stage : str, default = 'original'
            What stage of feature-processing should importances be computed for.
            Options:
                'original':
                    Compute importances of the original features.
                    Warning: `dataset` must be specified with this option, otherwise an exception will be raised.
                'transformed':
                    Compute importances of the post-internal-transformation features (after automated feature engineering). These features may be missing some original features, or add new features entirely.
                    An example of new features would be ngram features generated from a text column.
                    Warning: For bagged models, feature importance calculation is not yet supported with this option when `dataset=None`. Doing so will raise an exception.
                'transformed_model':
                    Compute importances of the post-model-transformation features. These features are the internal features used by the requested model. They may differ greatly from the original features.
                    If the model is a stack ensemble, this will include stack ensemble features such as the prediction probability features of the stack ensemble's base (ancestor) models.
        subsample_size : int, default = 1000
            The number of rows to sample from `dataset` when computing feature importance.
            If `subsample_size=None` or `dataset` contains fewer than `subsample_size` rows, all rows will be used during computation.
            Larger values increase the accuracy of the feature importance scores.
            Runtime linearly scales with `subsample_size`.
        time_limit : float, default = None
            Time in seconds to limit the calculation of feature importance.
            If None, feature importance will calculate without early stopping.
            A minimum of 1 full shuffle set will always be evaluated. If a shuffle set evaluation takes longer than `time_limit`, the method will take the length of a shuffle set evaluation to return regardless of the `time_limit`.
        num_shuffle_sets : int, default = None
            The number of different permutation shuffles of the data that are evaluated.
            Larger values will increase the quality of the importance evaluation.
            It is generally recommended to increase `subsample_size` before increasing `num_shuffle_sets`.
            Defaults to 3 if `time_limit` is None or 10 if `time_limit` is specified.
            Runtime linearly scales with `num_shuffle_sets`.
        include_confidence_band: bool, default = True
            If True, will include output columns 'p99_high' and 'p99_low' which indicates that the true feature importance will be between 'p99_high' and 'p99_low' 99% of the time (99% confidence interval).
            Increasing `subsample_size` and `num_shuffle_sets` will tighten the band.
        silent : bool, default = False
            Whether to suppress logging output.

        Returns
        -------
        Pandas `pandas.DataFrame` of feature importance scores with 6 columns:
            index: The feature name.
            'importance': The estimated feature importance score.
            'stddev': The standard deviation of the feature importance score. If NaN, then not enough num_shuffle_sets were used to calculate a variance.
            'p_value': P-value for a statistical t-test of the null hypothesis: importance = 0, vs the (one-sided) alternative: importance > 0.
                Features with low p-value appear confidently useful to the predictor, while the other features may be useless to the predictor (or even harmful to include in its training data).
                A p-value of 0.01 indicates that there is a 1% chance that the feature is useless or harmful, and a 99% chance that the feature is useful.
                A p-value of 0.99 indicates that there is a 99% chance that the feature is useless or harmful, and a 1% chance that the feature is useful.
            'n': The number of shuffles performed to estimate importance score (corresponds to sample-size used to determine confidence interval for true score).
            'p99_high': Upper end of 99% confidence interval for true feature importance score.
            'p99_low': Lower end of 99% confidence interval for true feature importance score.
        
Calculates feature importance scores for the given model via permutation importance.

For highly accurate importance and p_value estimates, it is recommend to set `subsample_size` to at least 5,000 if possible and `num_shuffle_sets` to at least 10.

Parameters



Returns

Pandas `pandas.DataFrame` of feature importance scores with 6 columns:
index: The feature name.
'importance': The estimated feature importance score.
'stddev': The standard deviation of the feature importance score. If NaN, then not enough num_shuffle_sets were used to calculate a variance.
'p_value': P-value for a statistical t-test of the null hypothesis: importance = 0, vs the (one-sided) alternative: importance > 0.
Features with low p-value appear confidently useful to the predictor, while the other features may be useless to the predictor (or even harmful to include in its training data).
A p-value of 0.01 indicates that there is a 1% chance that the feature is useless or harmful, and a 99% chance that the feature is useful.
A p-value of 0.99 indicates that there is a 99% chance that the feature is useless or harmful, and a 1% chance that the feature is useful.
'n': The number of shuffles performed to estimate importance score (corresponds to sample-size used to determine confidence interval for true score).
'p99_high': Upper end of 99% confidence interval for true feature importance score.
'p99_low': Lower end of 99% confidence interval for true feature importance score.
Calculates feature importance scores for the given model via permutation importance . For highly accurate importance and p_value estimates it is recommend to set ` subsample_size ` to at least 5 000 if possible and ` num_shuffle_sets ` to at least 10 . Parameters Returns Pandas ` pandas . DataFrame ` of feature importance scores with 6 columns : index : The feature name . ' importance ' : The estimated feature importance score . ' stddev ' : The standard deviation of the feature importance score . If NaN then not enough num_shuffle_sets were used to calculate a variance . ' p_value ' : P - value for a statistical t - test of the null hypothesis : importance = 0 vs the ( one - sided ) alternative : importance > 0 . Features with low p - value appear confidently useful to the predictor while the other features may be useless to the predictor ( or even harmful to include in its training data ) . A p - value of 0 . 01 indicates that there is a 1% chance that the feature is useless or harmful and a 99% chance that the feature is useful . A p - value of 0 . 99 indicates that there is a 99% chance that the feature is useless or harmful and a 1% chance that the feature is useful . ' n ' : The number of shuffles performed to estimate importance score ( corresponds to sample - size used to determine confidence interval for true score ) . ' p99_high ' : Upper end of 99% confidence interval for true feature importance score . ' p99_low ' : Lower end of 99% confidence interval for true feature importance score .
def feature_importance(self, dataset=None, model=None, features=None, feature_stage='original', subsample_size=1000, time_limit=None, num_shuffle_sets=None, include_confidence_band=True, silent=False):
        dataset = self.__get_dataset(dataset) if dataset is not None else dataset
        if (dataset is None) and (not self._trainer.is_data_saved):
            raise AssertionError('No dataset was provided and there is no cached data to load for feature importance calculation. `cache_data=True` must be set in the `TabularPrediction.fit()` call to enable this functionality when dataset is not specified.')
        if num_shuffle_sets is None:
            num_shuffle_sets = 10 if time_limit else 3
        fi_df = self._learner.get_feature_importance(model=model, X=dataset, features=features, feature_stage=feature_stage,
                                                    subsample_size=subsample_size, time_limit=time_limit, num_shuffle_sets=num_shuffle_sets, silent=silent)
        if include_confidence_band:
            import scipy.stats
            num_features = len(fi_df)
            confidence_threshold = 0.99
            p99_low_dict = dict()
            p99_high_dict = dict()
            for i in range(num_features):
                fi = fi_df.iloc[i]
                mean = fi['importance']
                stddev = fi['stddev']
                n = fi['n']
                if stddev == np.nan or n == np.nan or mean == np.nan or n == 1:
                    p99_high = np.nan
                    p99_low = np.nan
                else:
                    t_val_99 = scipy.stats.t.ppf(1 - (1 - confidence_threshold) / 2, n-1)
                    p99_high = mean + t_val_99 * stddev / math.sqrt(n)
                    p99_low = mean - t_val_99 * stddev / math.sqrt(n)
                p99_high_dict[fi.name] = p99_high
                p99_low_dict[fi.name] = p99_low
            fi_df['p99_high'] = pd.Series(p99_high_dict)
            fi_df['p99_low'] = pd.Series(p99_low_dict)
        return fi_df
def feature_importance ( self , dataset = None , model = None , features = None , feature_stage = 'original' , subsample_size = 1000 , time_limit = None , num_shuffle_sets = None , include_confidence_band = True , silent = False ) : dataset = self . __get_dataset ( dataset ) if dataset is not None else dataset if ( dataset is None ) and ( not self . _trainer . is_data_saved ) : raise AssertionError ( 'No dataset was provided and there is no cached data to load for feature importance calculation. `cache_data=True` must be set in the `TabularPrediction.fit()` call to enable this functionality when dataset is not specified.' ) if num_shuffle_sets is None : num_shuffle_sets = 10 if time_limit else 3 fi_df = self . _learner . get_feature_importance ( model = model , X = dataset , features = features , feature_stage = feature_stage , subsample_size = subsample_size , time_limit = time_limit , num_shuffle_sets = num_shuffle_sets , silent = silent ) if include_confidence_band : import scipy . stats num_features = len ( fi_df ) confidence_threshold = 0.99 p99_low_dict = dict ( ) p99_high_dict = dict ( ) for i in range ( num_features ) : fi = fi_df . iloc [ i ] mean = fi [ 'importance' ] stddev = fi [ 'stddev' ] n = fi [ 'n' ] if stddev == np . nan or n == np . nan or mean == np . nan or n == 1 : p99_high = np . nan p99_low = np . nan else : t_val_99 = scipy . stats . t . ppf ( 1 - ( 1 - confidence_threshold ) / 2 , n - 1 ) p99_high = mean + t_val_99 * stddev / math . sqrt ( n ) p99_low = mean - t_val_99 * stddev / math . sqrt ( n ) p99_high_dict [ fi . name ] = p99_high p99_low_dict [ fi . name ] = p99_low fi_df [ 'p99_high' ] = pd . Series ( p99_high_dict ) fi_df [ 'p99_low' ] = pd . Series ( p99_low_dict ) return fi_df
Calculates feature importance scores for the given model via permutation importance.
Calculates feature importance scores for the given model via permutation importance .
"""
        Calculates feature importance scores for the given model via permutation importance. Refer to https://explained.ai/rf-importance/ for an explanation of permutation importance.
        A feature's importance score represents the performance drop that results when the model makes predictions on a perturbed copy of the dataset where this feature's values have been randomly shuffled across rows.
        A feature score of 0.01 would indicate that the predictive performance dropped by 0.01 when the feature was randomly shuffled.
        The higher the score a feature has, the more important it is to the model's performance.
        If a feature has a negative score, this means that the feature is likely harmful to the final model, and a model trained with the feature removed would be expected to achieve a better predictive performance.
        Note that calculating feature importance can be a very computationally expensive process, particularly if the model uses hundreds or thousands of features. In many cases, this can take longer than the original model training.
        To estimate how long `feature_importance(model, dataset, features)` will take, it is roughly the time taken by `predict_proba(dataset, model)` multiplied by the number of features.

        Note: For highly accurate importance and p_value estimates, it is recommend to set `subsample_size` to at least 5,000 if possible and `num_shuffle_sets` to at least 10.

        Parameters
        ----------
        dataset : str or :class:`TabularDataset` or `pandas.DataFrame` (optional)
            This dataset must also contain the label-column with the same column-name as specified during `fit()`.
            If specified, then the dataset is used to calculate the feature importance scores.
            If str is passed, `dataset` will be loaded using the str value as the file path.
            If not specified, the original dataset used during `fit()` will be used if `cache_data=True`. Otherwise, an exception will be raised.
            Do not pass the training data through this argument, as the feature importance scores calculated will be biased due to overfitting.
                More accurate feature importances will be obtained from new data that was held-out during `fit()`.
        model : str, default = None
            Model to get feature importances for, if None the best model is chosen.
            Valid models are listed in this `predictor` by calling `predictor.get_model_names()`
        features : list, default = None
            List of str feature names that feature importances are calculated for and returned, specify None to get all feature importances.
            If you only want to compute feature importances for some of the features, you can pass their names in as a list of str.
            Valid feature names change depending on the `feature_stage`.
                To get the list of feature names for `feature_stage='transformed'`, call `list(predictor.transform_features().columns)`.
                To get the list of feature names for `feature_stage=`transformed_model`, call `list(predictor.transform_features(model={model_name}).columns)`.
        feature_stage : str, default = 'original'
            What stage of feature-processing should importances be computed for.
            Options:
                'original':
                    Compute importances of the original features.
                    Warning: `dataset` must be specified with this option, otherwise an exception will be raised.
                'transformed':
                    Compute importances of the post-internal-transformation features (after automated feature engineering). These features may be missing some original features, or add new features entirely.
                    An example of new features would be ngram features generated from a text column.
                    Warning: For bagged models, feature importance calculation is not yet supported with this option when `dataset=None`. Doing so will raise an exception.
                'transformed_model':
                    Compute importances of the post-model-transformation features. These features are the internal features used by the requested model. They may differ greatly from the original features.
                    If the model is a stack ensemble, this will include stack ensemble features such as the prediction probability features of the stack ensemble's base (ancestor) models.
        subsample_size : int, default = 1000
            The number of rows to sample from `dataset` when computing feature importance.
            If `subsample_size=None` or `dataset` contains fewer than `subsample_size` rows, all rows will be used during computation.
            Larger values increase the accuracy of the feature importance scores.
            Runtime linearly scales with `subsample_size`.
        time_limit : float, default = None
            Time in seconds to limit the calculation of feature importance.
            If None, feature importance will calculate without early stopping.
            A minimum of 1 full shuffle set will always be evaluated. If a shuffle set evaluation takes longer than `time_limit`, the method will take the length of a shuffle set evaluation to return regardless of the `time_limit`.
        num_shuffle_sets : int, default = None
            The number of different permutation shuffles of the data that are evaluated.
            Larger values will increase the quality of the importance evaluation.
            It is generally recommended to increase `subsample_size` before increasing `num_shuffle_sets`.
            Defaults to 3 if `time_limit` is None or 10 if `time_limit` is specified.
            Runtime linearly scales with `num_shuffle_sets`.
        include_confidence_band: bool, default = True
            If True, will include output columns 'p99_high' and 'p99_low' which indicates that the true feature importance will be between 'p99_high' and 'p99_low' 99% of the time (99% confidence interval).
            Increasing `subsample_size` and `num_shuffle_sets` will tighten the band.
        silent : bool, default = False
            Whether to suppress logging output.

        Returns
        -------
        Pandas `pandas.DataFrame` of feature importance scores with 6 columns:
            index: The feature name.
            'importance': The estimated feature importance score.
            'stddev': The standard deviation of the feature importance score. If NaN, then not enough num_shuffle_sets were used to calculate a variance.
            'p_value': P-value for a statistical t-test of the null hypothesis: importance = 0, vs the (one-sided) alternative: importance > 0.
                Features with low p-value appear confidently useful to the predictor, while the other features may be useless to the predictor (or even harmful to include in its training data).
                A p-value of 0.01 indicates that there is a 1% chance that the feature is useless or harmful, and a 99% chance that the feature is useful.
                A p-value of 0.99 indicates that there is a 99% chance that the feature is useless or harmful, and a 1% chance that the feature is useful.
            'n': The number of shuffles performed to estimate importance score (corresponds to sample-size used to determine confidence interval for true score).
            'p99_high': Upper end of 99% confidence interval for true feature importance score.
            'p99_low': Lower end of 99% confidence interval for true feature importance score.
        """
self
dataset
model
features
feature_stage
subsample_size
time_limit
num_shuffle_sets
include_confidence_band
silent