"""
## Random forest and Random Search on red wine quality:
![Random forest: credit: github user kjw0612](https://camo.githubusercontent.com/5afa4bc35f56871811e34f442baacc6cc098bd12a9320062846efe1bed011275/68747470733a2f2f33312e6d656469612e74756d626c722e636f6d2f37393637306561626539336364643434386331356635626362313938643066622f74756d626c725f696e6c696e655f6e386533393859624b76317330347263332e706e67)
In this notebook Eswar sai and I have worked on Random forest regression and feature importance based feature selection using it. This is a good notebook for someone who is starting out with random forest. But before all that, <br/>
### What Is Random Forest?
Random forest is a supervised learning algorithm. The "forest" it builds, is an ensemble of decision trees, usually trained with the “bagging” method. The general idea of the bagging method is that a combination of learning models increases the overall result.<br/>
### How Random Forest Works:

Put simply: random forest builds multiple decision trees and merges them together to get a more accurate and stable prediction.<br/>

One big advantage of random forest is that it can be used for both classification and regression problems, which form the majority of current machine learning systems. Let's look at random forest in classification, since classification is sometimes considered the building block of machine learning.<br/>

Random forest has nearly the same hyperparameters as a decision tree or a bagging classifier. Fortunately, there's no need to combine a decision tree with a bagging classifier because you can easily use the classifier-class of random forest. With random forest, you can also deal with regression tasks by using the algorithm's regressor.<br/>

Random forest adds additional randomness to the model, while growing the trees. Instead of searching for the most important feature while splitting a node, it searches for the best feature among a random subset of features. This results in a wide diversity that generally results in a better model.<br/>

Therefore, in random forest, only a random subset of the features is taken into consideration by the algorithm for splitting a node. You can even make trees more random by additionally using random thresholds for each feature rather than searching for the best possible thresholds (like a normal decision tree does).<br/>

Random forest was first introduced by Briemann in his paper of 2001. You can read the details of random forest's inner working, which is called the CART algorithm, in the [following blog](https://shyambhu20.blogspot.com/2020/01/Random-forest-model.html).<br/>
Now, let's discuss the contents of this notebook:<br/>
### contents:
(1) [Basic data exploration](#section1)<br/>
(2) [Basic Random forest regressor fitting](#section2)<br/>
(3) [Fine tuning using RandomizedSearchCV](#random)<br/>
(4) [Fine tuning using GridSearchCV](#grid)<br/>
<br/>
Enjoy reading! and if you like the effort, consider showing your appreciation with a upvote!
References:
(1) [Description taken from this article](https://builtin.com/data-science/random-forest-algorithm)<br/>

"""
"""
### Data loading
"""
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
df= pd.read_csv('/kaggle/input/red-wine-quality-cortez-et-al-2009/winequality-red.csv')
df.head()
df.describe().T
"""
### <a id = 'section1'> Basic data exploration</a>
"""
df.isnull().sum()
from scipy import stats
z = np.abs(stats.zscore(df))
print(z)
"""
### Z-score based outlier removal
"""
threshold = 3
print(np.where(z > 3))
print(z[13][9])
df_o = df[(z < 3).all(axis=1)]
df.shape
df_o.shape
"""
### Train-test-split
"""
from sklearn.model_selection import train_test_split
X = df_o.drop(columns = 'quality')
y = df_o['quality']
X.head()
y.head()
"""
### <a id='section1'>Basic Random forest fitting</a>
"""
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=100)
clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)
from sklearn import metrics
print('Accuracy: ', metrics.accuracy_score(y_test,y_pred))
df.columns
import pandas as pd
feature_imp = pd.Series(clf.feature_importances_, index=df_o.columns[:11]).sort_values(ascending=False)
feature_imp
"""
### Feature importance visualization
"""
%matplotlib inline
import seaborn as sns

sns.barplot(x=feature_imp, y=feature_imp.index)

plt.xlabel('Feature Importance Score')
plt.ylabel('Features')
plt.title("Visualizing Important Features")
plt.legend()
plt.show()
"""
### <a id = 'random'>Fine tuning using RandomSearchCV</a>
"""
#Random Search Cross Validation

from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(random_state = 42)
from pprint import pprint
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(rf.get_params())
from sklearn.model_selection import RandomizedSearchCV
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(2, 14, num = 7)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}
pprint(random_grid)
# Use the random grid to search for best hyperparameters
# First create the base model to tune
rf = RandomForestRegressor()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)
# Fit the random search model
rf_random.fit(X_train,y_train)
rf_random.best_params_
def evaluate(model, X_test, y_test):
    predictions = model.predict(X_test)
    errors = abs(predictions - y_test)
    mape = 100 * np.mean(errors / y_test)
    accuracy = 100 - mape
    print('Model Performance')
    print('Average Error: {:0.4f} degrees.'.format(np.mean(errors)))
    print('Accuracy = {:0.2f}%.'.format(accuracy))
    
    return accuracy
base_model = RandomForestRegressor(n_estimators = 10, random_state = 42)
base_model.fit(X_train, y_train)
base_accuracy = evaluate(base_model, X_test,y_test)
best_random = rf_random.best_estimator_
random_accuracy = evaluate(best_random, X_test, y_test)
print('Improvement of {:0.2f}%.'.format( 100 * (random_accuracy - base_accuracy) / base_accuracy))
"""
### <a id='grid'>Fine tuning with GridSearchCV </a>
"""
#Grid Search with Cross Validation

from sklearn.model_selection import GridSearchCV
# Create the parameter grid based on the results of random search 
param_grid = {
    'bootstrap': [True],
    'max_depth': [8, 10, 12, 14],
    'max_features': [2, 3],
    'min_samples_leaf': [3, 4, 5],
    'min_samples_split': [8, 10, 12],
    'n_estimators': [100, 200, 300, 1000]
}
# Create a based model
rf = RandomForestRegressor()
# Instantiate the grid search model
grid_search = GridSearchCV(estimator = rf, param_grid = param_grid, 
                          cv = 3, n_jobs = -1, verbose = 2)
# Fit the grid search to the data
grid_search.fit(X_train, y_train)
grid_search.best_params_
best_grid = grid_search.best_estimator_
grid_accuracy = evaluate(best_grid, X_test, y_test)
"""
#### Here ends our treaties with Random forest. Thanks for reading the notebook. Happy kaggling!
"""