# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] toc=true
# <h1>Table of Contents<span class="tocSkip"></span></h1>
# <div class="toc"><ul class="toc-item"><li><span><a href="#Introduction" data-toc-modified-id="Introduction-1">Introduction</a></span></li><li><span><a href="#Why-Random-Forest?" data-toc-modified-id="Why-Random-Forest?-2">Why Random Forest?</a></span><ul class="toc-item"><li><span><a href="#Advantages" data-toc-modified-id="Advantages-2.1">Advantages</a></span></li><li><span><a href="#Disadvantages" data-toc-modified-id="Disadvantages-2.2">Disadvantages</a></span></li></ul></li><li><span><a href="#How-Random-Forest-Works?" data-toc-modified-id="How-Random-Forest-Works?-3">How Random Forest Works?</a></span></li><li><span><a href="#Random-Forests-vs-Decision-Trees" data-toc-modified-id="Random-Forests-vs-Decision-Trees-4">Random Forests vs Decision Trees</a></span></li><li><span><a href="#Project---Flower-Classification-Using-Random-Forest-Algorithm" data-toc-modified-id="Project---Flower-Classification-Using-Random-Forest-Algorithm-5">Project - Flower Classification Using Random Forest Algorithm</a></span></li><li><span><a href="#Data-Load-and-Initial-Exploration" data-toc-modified-id="Data-Load-and-Initial-Exploration-6">Data Load and Initial Exploration</a></span><ul class="toc-item"><li><span><a href="#Changing-Data-Format" data-toc-modified-id="Changing-Data-Format-6.1">Changing Data Format</a></span></li></ul></li><li><span><a href="#Data-Pre-processing" data-toc-modified-id="Data-Pre-processing-7">Data Pre-processing</a></span></li><li><span><a href="#Train-the-Model-and-Predict" data-toc-modified-id="Train-the-Model-and-Predict-8">Train the Model and Predict</a></span></li><li><span><a href="#Evaluate-the-Model" data-toc-modified-id="Evaluate-the-Model-9">Evaluate the Model</a></span></li><li><span><a href="#Hyperparameter-Tuning" data-toc-modified-id="Hyperparameter-Tuning-10">Hyperparameter Tuning</a></span></li></ul></div>
# -

# # Random Forest

# ## Introduction
#
# Random forest is a supervised learning algorithm. The "forest" it builds, is an collection of decision trees, usually trained with the “bagging” method. The general idea of the bagging method is that a combination of learning models increases the overall result.
#
# i.e. random forest builds multiple decision trees and merges them together to get a more accurate and stable prediction.
#
# It is one of the most used algorithms, because of its simplicity and diversity (it can be used for both classification and regression tasks).

# ## Why Random Forest?
#
# Why go to the trouble of creating a forest of decision trees when a single decision tree already works? 
#
# Trees are easy to build, easy to use, and easy to interpret BUT they are not very accurate. Decision Trees have some intrinsic limitations that a Random Forest helps to address - Decision Trees are prone to overfitting - i.e. they work great with training data but their accuracy is not so great with classifying new samples.
#
# Random Forests combine the simplicity of Decision Trees with flexibility to deliver a vast improvement in accuracy.

# ### Advantages
#
# - Random forests is considered as a highly accurate and robust method because of the number of decision trees participating in the process.
# - It does not suffer from the overfitting problem. The main reason is that it takes the average of all the predictions, which cancels out the biases.
# - The algorithm can be used in both classification and regression problems.
# - Random forests can also handle missing values. There are two ways to handle these: using median values to replace continuous variables, and computing the proximity-weighted average of missing values.
# - You can get the relative feature importance, which helps in selecting the most contributing features for the classifier.

# ### Disadvantages
#
# - Random forests is slow in generating predictions because it has multiple decision trees. Whenever it makes a prediction, all the trees in the forest have to make a prediction for the same given input and then perform voting on it. This whole process is time-consuming.
# - The model is difficult to interpret compared to a decision tree, where you can easily make a decision by following the path in the tree.

# ## How Random Forest Works?
#
# Here's a short video on how to build a Random Forest.

# +
## Run this cell (shift+enter) to see the video

from IPython.display import IFrame
IFrame("https://www.youtube.com/embed/J4Wdy0Wc_xQ", width="814", height="509")
# -

# ## Random Forests vs Decision Trees
# - Random forests is a set of multiple decision trees.
# - Deep decision trees may suffer from overfitting, but random forests prevents overfitting by creating trees on random subsets.
# - Decision trees are computationally faster.
# - Random forests is difficult to interpret, while a decision tree is easily interpretable and can be converted to rules.

# ## Project - Flower Classification Using Random Forest Algorithm
#
# You will be building a model on the iris flower dataset, which is a very famous classification set. It comprises the sepal length, sepal width, petal length, petal width, and type of flowers. There are three species or classes: setosa, versicolor, and virginia. You will build a model to classify the type of flower. The dataset is available in the scikit-learn library or you can download it from the UCI Machine Learning Repository.
#
# Here's a link to the dataset if you wish to explore it further - http://archive.ics.uci.edu/ml/datasets/Iris/

import numpy as np 
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
import matplotlib
# %matplotlib inline

# ## Data Load and Initial Exploration
#
# The Iris dataset come preloaded in the scikitlearn library.

# +
#Import scikit-learn dataset library
from sklearn import datasets

#Load dataset
iris = datasets.load_iris()
# -

# Let's take a quick look at the features of our dataset and the different flower types that the samples are classified into - 

# print the names of the four features
print(iris.feature_names)

# print the label species(setosa, versicolor,virginica)
print(iris.target_names)

# ### Changing Data Format
#
# The `iris` object that holds our dataset is not a pandas dataframe. Let's store the data in a dataframe to enable further analysis and modeling.

type(iris)

# +
## converting to a dataframe

data=pd.DataFrame({
    'sepal length':iris.data[:,0],
    'sepal width':iris.data[:,1],
    'petal length':iris.data[:,2],
    'petal width':iris.data[:,3],
    'species':iris.target
})
# -

# Let's take a quick look at the data before we move ahead - 

data.head()

data['species'].value_counts()

data.hist()

# ## Data Pre-processing
#
# We will not need to do much pre-processing in this case. The data is already in the form we need it. We will do two quick things though - 
#
# - split the features and the labels
# - split the data into train and test datasets

# +
# Import train_test_split function
from sklearn.model_selection import train_test_split

X=data[['sepal length', 'sepal width', 'petal length', 'petal width']]  # Features
y=data['species']  # Labels

# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3) # 70% training and 30% test

# -

# ## Train the Model and Predict

# +
#Import Random Forest Model
from sklearn.ensemble import RandomForestClassifier

#Create a Gaussian Classifier
clf=RandomForestClassifier(n_estimators=100)

#Train the model using the training sets y_pred=clf.predict(X_test)
clf.fit(X_train,y_train)

y_pred=clf.predict(X_test)

# -

# ## Evaluate the Model

# Test scores for the test dataset - 

#Import scikit-learn metrics module for accuracy calculation
from sklearn import metrics
# Model Accuracy, how often is the classifier correct?
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

# Test scores for the train dataset - 

print("Accuracy:",metrics.accuracy_score(y_train, clf.predict(X_train)))

# ## Hyperparameter Tuning
#
# Now that you've built a Random Forest classifier how can you improve the accuracy?
#
# We covered Hyperparameter Tuning in the Decision Trees notebook. Let's see how we can apply that to our current project. We will be ranking the importance of features and then selectively use the most important features to re-train our model.
#
# The RandomForestClassifier helps us get feature importance scores - 

feature_imp = pd.Series(clf.feature_importances_,index=iris.feature_names).sort_values(ascending=False)
feature_imp

# Now that we know the top three features that are pertinent for our model, let's do feature selection accordingly - 

# Split dataset into features and labels
X=data[['petal length', 'petal width','sepal length']]  # Removed feature "sepal width"
y=data['species']                                       
# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.70, random_state=5) # 70% training and 30% test

# Let's retrain our model - 

# +
#Create a Gaussian Classifier
clf=RandomForestClassifier(n_estimators=100)

#Train the model using the training sets y_pred=clf.predict(X_test)
clf.fit(X_train,y_train)

# prediction on test set
y_pred=clf.predict(X_test)
# -

# ... and now for the evaluation. Did the feature selection lead to a favorable outcome?

# Model Accuracy, how often is the classifier correct?
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

# Great! As you might notice, our accuracy has gone up from 0.933 to 0.952.

#
# This project was created using a variety of online resources as references. This informative tutorial by <NAME> was used extensively - https://www.datacamp.com/community/tutorials/random-forests-classifier-python

StarcoderdataJupyterScriptsDedupFiltered