from feature_selector import FeatureSelector

import pandas as pd

"""
Explanation: Introduction: Feature Selector Usage
In this notebook we will walk-through using the FeatureSelector class for selecting features to remove from a dataset. This class has five methods for finding features to remove: 

Find columns with a missing fraction greater than a specified threshold
Find features with only a single unique value
Find collinear features as identified by a correlation coefficient greater than a specified value
Find features with 0.0 importance from a gradient boosting machine
Find features that do not contribute to a specified cumulative feature importance from the gradient boosting machine

The FeatureSelector is a work in progress! Any contributions on GitHub are appreciated.
End of explanation
"""


train = pd.read_csv('data/credit_example.csv')
train_labels = train['TARGET']
train.head()

"""
Explanation: Example Dataset
This dataset was used as part of the Home Credit Default Risk competition on Kaggle. It is intended for a supervised machine learning classification task where the objective is to predict if a client will default on a loan. The entire dataset can be downloaded [here] and we will work with a small sample of 10,000 rows. 
The feature selector was designed to be used for machine learning tasks, but can be applied to any dataset.\ The feature importance based methods do require a supervised machine learning problem.
End of explanation
"""


train = train.drop(columns = ['TARGET'])

"""
Explanation: There are several categorical columns in the dataset. The FeatureSelector handles these using one-hot encoding when using the feature importance based methods.
End of explanation
"""


fs = FeatureSelector(data = train, labels = train_labels)

"""
Explanation: Implementation
The FeatureSelector has five functions for identifying columns to remove:

identify_missing
identify_single_unique
identify_collinear
identify_zero_importance 
identify_low_importance

These methods find the features to drop according to specified criteria. The identified features are stored in the ops attribute (a Python dictionary) of the FeatureSelector. We can remove the identified features manually or use the remove function in the  FeatureSelector for actually removing the features. 
Create the Instance
The FeatureSelector only requires a dataset with observations in the rows and features in the columns (standard structured data). We are working with a classifified machine learning problem so we also pass in training labels.
End of explanation
"""


fs.identify_missing(missing_threshold=0.6)

"""
Explanation: 1. Missing Values
The first feature selection method is straightforward: find any columns with a missing fraction greater than a specified threshold. For this example we will use a threhold of 0.6 which corresponds to finding features with more than 60% missing values. (This method does not one-hot encode the features first).
End of explanation
"""


missing_features = fs.ops['missing']
missing_features[:10]

"""
Explanation: The features identified for removal can be accessed through the ops dictionary of the FeatureSelector object.
End of explanation
"""


fs.plot_missing()

"""
Explanation: We can also plot a histogram of the missing column fraction for all columns in the dataset.
End of explanation
"""


fs.missing_stats.head(10)

"""
Explanation: For detailed information on the missing fractions, we can access the missing_stats attribute, a dataframe of the missing fractions for all features.
End of explanation
"""


fs.identify_single_unique()

single_unique = fs.ops['single_unique']
single_unique

"""
Explanation: 2. Single Unique Value
The next method is straightforward: find any features that have only a single unique value. (This does not one-hot encode the features).
End of explanation
"""


fs.plot_unique()

"""
Explanation: We can plot a histogram of the number of unique values in each feature of the dataset.
End of explanation
"""


fs.unique_stats.sample(5)

"""
Explanation: Finally, we can access a dataframe with the number of unique values per feature.
End of explanation
"""


fs.identify_collinear(correlation_threshold=0.975)

correlated_features = fs.ops['collinear']
correlated_features[:5]

"""
Explanation: 3. Collinear (highly correlated) Features
This method finds pairs of collinear features based on the Pearson correlation coefficient. For each pair above the specified threshold (in terms of absolute value), it identifies one of the variables to be removed. We need to pass in a correlation_threshold. 
This method is based on code found at https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/
For each pair, the feature that will be removed is the one that comes last in terms of the column ordering in the dataframe. (This method does not one-hot encode the data beforehand unless one_hot=True. Therefore correlations are only calculated between numeric columns)
End of explanation
"""


fs.plot_collinear()

"""
Explanation: We can view a heatmap of the correlations above the threhold. The features which will be dropped are on the x-axis.
End of explanation
"""


fs.plot_collinear(plot_all=True)

fs.identify_collinear(correlation_threshold=0.98)
fs.plot_collinear()

"""
Explanation: To plot all of the correlations in the data, we can pass in plot_all = True to the plot_collinear function.
End of explanation
"""


fs.record_collinear.head()

"""
Explanation: To view the details of the corelations above the threshold, we access the record_collinear attribute which is a dataframe. The drop_feature will be removed and for each feature that will be removed, there may be several correlations it has with the corr_feature that are above the correlation_threshold.
End of explanation
"""


fs.identify_zero_importance(task = 'classification', eval_metric = 'auc', 
                            n_iterations = 10, early_stopping = True)

"""
Explanation: 4. Zero Importance Features
This method relies on a machine learning model to identify features to remove. It therefore requires a supervised learning problem with labels. The method works by finding feature importances using a gradient boosting machine implemented in the LightGBM library. 
To reduce variance in the calculated feature importances, the model is trained a default 10 times. The model is also by default trained with early stopping using a validation set (15% of the training data) to identify the optimal number of estimators to train. The following parameters can be passed to the identify_zero_importance method:

task: either classification or regression. The metric and labels must match with the task
eval_metric: the metric used for early stopping (for example auc for classification or l2 for regression). To see a list of available metrics, refer to the LightGBM docs
n_iterations: number of training runs. The feature importances are averaged over the training runs (default = 10)
early_stopping: whether to use early stopping when training the model (default = True). Early stopping stops training estimators (decision trees) when the performance on a validation set no longer decreases for a specified number of estimators (100 by default in this implementation). Early stopping is a form of regularization used to prevent overfitting to training data

The data is first one-hot encoded for use in the model. This means that some of the zero importance features may be created from one-hot encoding. To view the one-hot encoded columns, we can access the one_hot_features of the FeatureSelector.
Note of caution: in contrast to the other methods, the feature imporances from a model are non-deterministic (have a little randomness). The results of running this method can change each time it is run.
End of explanation
"""


one_hot_features = fs.one_hot_features
base_features = fs.base_features
print('There are %d original features' % len(base_features))
print('There are %d one-hot features' % len(one_hot_features))

"""
Explanation: Running the gradient boosting model requires one hot encoding the features. These features are saved in the one_hot_features attribute of the FeatureSelector. The original features are saved in the base_features.
End of explanation
"""


fs.data_all.head(10)

"""
Explanation: The data attribute of the FeatureSelector holds the original dataframe. After one-hot encoding, the data_all attribute holds the original data plus the one-hot encoded features.
End of explanation
"""


zero_importance_features = fs.ops['zero_importance']
zero_importance_features[10:15]

"""
Explanation: There are a number of methods we can use to inspect the results of the feature importances. First we can access the list of features with zero importance.
End of explanation
"""


fs.plot_feature_importances(threshold = 0.99, plot_n = 12)

"""
Explanation: Plot Feature Importances
The feature importance plot using plot_feature_importances will show us the plot_n most important features (on a normalized scale where the features sum to 1). It also shows us the cumulative feature importance versus the number of features. 
When we plot the feature importances, we can pass in a threshold which identifies the number of features required to reach a specified cumulative feature importance. For example, threshold = 0.99 will tell us the number of features needed to account for 99% of the total importance.
End of explanation
"""


fs.feature_importances.head(10)

"""
Explanation: All of the feature importances are accessible in the feature_importances attribute of the FeatureSelector
End of explanation
"""


one_hundred_features = list(fs.feature_importances.loc[:99, 'feature'])
len(one_hundred_features)

"""
Explanation: We could use these results to select only the 'n' most important features. For example, if we want the top 100 most importance, we could do the following.
End of explanation
"""


fs.identify_low_importance(cumulative_importance = 0.99)

"""
Explanation: 5. Low Importance Features
This method builds off the feature importances from the gradient boosting machine (identify_zero_importance must be run first) by finding the lowest importance features not needed to reach a specified cumulative total feature importance. For example, if we pass in 0.99, this will find the lowest important features that are not needed to reach 99% of the total feature importance. 
When using this method, we must have already run identify_zero_importance and need to pass in a cumulative_importance that accounts for that fraction of total feature importance.
Note of caution: this method builds on the gradient boosting model features importances and again is non-deterministic. I advise running these two methods several times with varying parameters and testing each resulting set of features rather than picking one number and sticking to it.
End of explanation
"""


low_importance_features = fs.ops['low_importance']
low_importance_features[:5]

"""
Explanation: The low importance features to remove are those that do not contribute to the specified cumulative importance. These are also available in the ops dictionary.
End of explanation
"""


train_no_missing = fs.remove(methods = ['missing'])

train_no_missing_zero = fs.remove(methods = ['missing', 'zero_importance'])

"""
Explanation: Removing Features
Once we have identified the features to remove, we have a number of ways to drop the features. We can access any of the feature lists in the removal_ops dictionary and remove the columns manually. We also can use the remove method, passing in the methods that identified the features we want to remove.
This method returns the resulting data which we can then use for machine learning. The original data will still be accessible in the data attribute of the Feature Selector.
Be careful of the methods used for removing features! It's a good idea to inspect the features that will be removed before using the remove function.
End of explanation
"""


all_to_remove = fs.check_removal()
all_to_remove[10:25]

"""
Explanation: To remove the features from all of the methods, pass in method='all'. Before we do this, we can check how many features will be removed using check_removal. This returns a list of all the features that have been idenfitied for removal.
End of explanation
"""


train_removed = fs.remove(methods = 'all')

"""
Explanation: Now we can remove all of the features idenfitied.
End of explanation
"""


train_removed_all = fs.remove(methods = 'all', keep_one_hot=False)

print('Original Number of Features', train.shape[1])
print('Final Number of Features: ', train_removed_all.shape[1])

"""
Explanation: Handling One-Hot Features
If we look at the dataframe that is returned, we may notice several new columns that were not in the original data. These are created when the data is one-hot encoded for machine learning. To remove all the one-hot features, we can pass in keep_one_hot = False to the remove method.
End of explanation
"""


fs = FeatureSelector(data = train, labels = train_labels)

fs.identify_all(selection_params = {'missing_threshold': 0.6, 'correlation_threshold': 0.98, 
                                    'task': 'classification', 'eval_metric': 'auc', 
                                     'cumulative_importance': 0.99})

train_removed_all_once = fs.remove(methods = 'all', keep_one_hot = True)

fs.feature_importances.head()

"""
Explanation: Alternative Option for Using all Methods
If we don't want to run the identification methods one at a time, we can use identify_all to run all the methods in one call. For this function, we need to pass in a dictionary of parameters to use for each individual identification method. 
The following code accomplishes the above steps in one call.
End of explanation
"""


