"""
# <center>House Price Prediction</center> 
#### <center>ML Foundation Project</center>
"""
"""
## Table of Contents
 
1. [Problem Statement](#section1)</br>
    - 1.1 [Introduction](#section101)<br/>
    
<br>

2. [Data Loading and Description](#section2)</br>
    - 2.1 [Importing Packages](#section201)<br/>
    - 2.2 [Importing Dataset](#section202)<br/>
    
<br>

3. [Data profiling](#section3)</br>
    - 3.1 [Understanding Data](#section301)<br/>
    - 3.2 [Data Preprofiling](#section302)<br/>
    - 3.3 [Data Preprocessing](#section303)<br/>
    - 3.4 [Seperating columns with respect to data types (Num, Cat and Bool)](#section304)<br/>
    - 3.5 [Copying columns from Numerical Column to Categrical](#section305)<br/>
    - 3.6 [Treating Missing Values](#section306)<br/> 
  
<br>

4. [Data Transformation](#section4)</br>
    - 4.1 [How many movies are released per year over the decade 2006-2016?](#section401)<br/>
    - 4.2 [Label Encoding Categorical(hdC) Data](#section402)<br/>
    - 4.3 [Dummifying Variables of the Boolean(hdB) Data](#section403)<br/>
    - 4.4 [Merging all the columns](#section404)<br/>
    - 4.5 [Outliers Detection and Handling](#section405)<br/>
    - 4.6 [Pandas Post-profiling](#section406)<br/>

<br>

5. [Exploratory Data Analysis](#section5)</br>
    - 5.1 [Count of the Houses Built in each years](#section501)<br/>
    - 5.2 [How many Houses were sold per year?](#section501)<br/>
    - 5.3 [Plotting the variation in Sale Price with change in the Lot Area and MSZoning of the house](#section503)<br/>
    - 5.4 [Understanding the distribution of data for selected columns](#section504)<br/>
    - 5.5 [Understanding the impact of Remodelling on the SalePrice](#section505)<br/>
    - 5.6 [Relationship between Price and Other selected columns](#section506)<br/>
    - 5.7 [Which Neighborhood house are expensive?](#section507)<br/>
    - 5.8 [Finding Correlation between the columns](#section508)<br/>

<br>

6. [Data Preparation for Model Input](#section6)</br>
    - 6.1 [Scaling the Data](#section601)<br/>
    - 6.2 [Splitting the Data](#section602)<br/>

<br>

7. [Model Implementation](#section7)</br>    
    - 7.1 [Linear Regression Model](#section701)<br/>
    - 7.2 [DecisionTree Regressor Model](#section702)<br/>            
    - 7.3 [RandomForest Regressor Model](#section703)<br/>

<br>

8. [Metrics for Model Performance](#section8)</br>

<br>

9. [Interpreting Model Coefficients](#section9)</br>
    - 9.1 [Plotting Coefficients of the Columns](#section901)<br/>
    - 9.2 [Cases for tuning the Model](#section902)<br/>

<br>

10. [Conclusion](#section10)</br>  

"""
"""
<a id=section1></a>
## 1. Problem statement

The dataset consists of 1460 houses sale information which are built over the period 1872-2010. The data consists of various variables for each house such as, MSZoning, LotArea, Street, Utilties, saletype, yearsold and many. The information from these variables will help in creating a Machine Learning Model, which would predict the Sale Price when an information of a new house with its characteristics are passed.
"""
"""
<a id=section101></a>
### 1.1 Introduction:

The Exploratory Data Analysis is to practice Python skills learned till now on a structured data set including loading, inspecting, wrangling, exploring, and drawing conclusions from data. The notebook has observations with each step in order to explain thoroughly how to approach the data set. Based on the observation some questions are arised and answered in the notebook for the reference though not all of them are explored in the analysis.

After EDA, The Data is treated for outliers and then, split into Train and Test sets of 75% and 25% respectively. The Models used to train and predict are Linear Regression and DecisionTree Regressor with each model in two ways such as, Basic(default) and GridSearchCV(best_estimator). The predictions from models are collected into DataFrame for comparisions and performance tuning.
"""
"""
<a id=section2></a>
## 2. Data Loading and Description:

The dataset comprises of __1460 observations of 81 columns__. Below is the table showing names of all the columns and respective description.
"""
"""
| Column Name        | Description                                                                              |
| -------------------|:----------------------------------------------------------------------------------------:| 
|SalePrice    | the property's sale price in dollars. This is the target variable that you're trying to predict.|
|MSSubClass   | The building class|
|MSZoning     | The general zoning classification|
|LotFrontage  | Linear feet of street connected to property|
|LotArea      | Lot size in square feet|
|Street       | Type of road access|
|Alley        | Type of alley access|
|LotShape     | General shape of property|
|LandContour  | Flatness of the property|
|Utilities    | Type of utilities available|
|LotConfig    | Lot configuration|
|LandSlope    | Slope of property|
|Neighborhood | Physical locations within Ames city limits|
|Condition1   | Proximity to main road or railroad|
|Condition2   | Proximity to main road or railroad (if a second is present)|
|BldgType     | Type of dwelling|
|HouseStyle   | Style of dwelling|
|OverallQual  | Overall material and finish quality|
|OverallCond  | Overall condition rating|
|YearBuilt    | Original construction date|
|YearRemodAdd | Remodel date|
|RoofStyle    | Type of roof|
|RoofMatl     | Roof material|
|Exterior1st  | Exterior covering on house|
|Exterior2nd  | Exterior covering on house (if more than one material)|
|MasVnrType   | Masonry veneer type|
|MasVnrArea   | Masonry veneer area in square feet|
|ExterQual    | Exterior material quality|
|ExterCond    | Present condition of the material on the exterior|
|Foundation   | Type of foundation|
|BsmtQual     | Height of the basement|
|BsmtCond     | General condition of the basement|
|BsmtExposure | Walkout or garden level basement walls|
|BsmtFinType1 | Quality of basement finished area|
|BsmtFinSF1   | Type 1 finished square feet|
|BsmtFinType2 | Quality of second finished area (if present)|
|BsmtFinSF2   | Type 2 finished square feet|
|BsmtUnfSF    | Unfinished square feet of basement area|
|TotalBsmtSF  | Total square feet of basement area|
|Heating      | Type of heating|
|HeatingQC    | Heating quality and condition|
|CentralAir   | Central air conditioning|
|Electrical   | Electrical system|
|1stFlrSF     | First Floor square feet|
|2ndFlrSF     | Second floor square feet|
|LowQualFinSF | Low quality finished square feet (all floors)|
|GrLivArea    | Above grade (ground) living area square feet|
|BsmtFullBath | Basement full bathrooms|
|BsmtHalfBath | Basement half bathrooms|
|FullBath     | Full bathrooms above grade|
|HalfBath     | Half baths above grade|
|Bedroom      | Number of bedrooms above basement level|
|Kitchen      | Number of kitchens|
|KitchenQual  | Kitchen quality|
|TotRmsAbvGrd | Total rooms above grade (does not include bathrooms)|
|Functional   | Home functionality rating|
|Fireplaces   | Number of fireplaces|
|FireplaceQu  | Fireplace quality|
|GarageType   | Garage location|
|GarageYrBlt  | Year garage was built|
|GarageFinish | Interior finish of the garage|
|GarageCars   | Size of garage in car capacity|
|GarageArea   | Size of garage in square feet|
|GarageQual   | Garage quality|
|GarageCond   | Garage condition|
|PavedDrive   | Paved driveway|
|WoodDeckSF   | Wood deck area in square feet|
|OpenPorchSF  | Open porch area in square feet|
|EnclosedPorch| Enclosed porch area in square feet|
|3SsnPorch    | Three season porch area in square feet|
|ScreenPorch  | Screen porch area in square feet|
|PoolArea     | Pool area in square feet|
|PoolQC       | Pool quality|
|Fence        | Fence quality|
|MiscFeature  | Miscellaneous feature not covered in other categories|
|MiscVal      | $Value of miscellaneous feature|
|MoSold       | Month Sold|
|YrSold       | Year Sold|
|SaleType     | Type of sale|
|SaleCondition| Condition of sale|
"""
"""
<a id=section201></a>
### 2.1 Importing Packages:
"""
import numpy as np                      # Implements multi-dimensional array and matrices
import pandas as pd                     # For data manipulation and analysis
import pandas_profiling
import matplotlib.pyplot as plt         # Plotting library for Python programming language and it's numerical mathematics extension NumPy
import seaborn as sns                   # Provides a high level interface for drawing attractive and informative statistical graphics
%matplotlib inline
sns.set()

from subprocess import check_output     # To check output
"""
<a id=section202></a>
### 2.2 Importing Dataset:
"""
#Reading House Data
hd = pd.read_csv('https://raw.githubusercontent.com/insaid2018/Term-2/master/Projects/house_data.csv')
"""
<a id=section3></a>
## 3. Data Profiling:

- Firstly, I will __understand the dataset__ using various pandas functionalities.
- Then, with the help of __pandas profiling__ I will find which columns of the dataset need preprocessing.
- In __preprocessing__, I will deal with erronous and missing values. 
- Then after, I will do __pandas profiling__ again, to see how preprocessing has transformed the dataset.
"""
"""
<a id=section301></a>
### 3.1 Understanding Data:
- Displaying the first five rows of the data to understand variables.
- Finding the shape of dataset.
"""
hd.head(10)                                # Display the first five rows of the dataset.
hd.tail(10)
hd.shape                                          # Shape of the Dataset
"""
- Next, I will do descriptive statistics for numerical variables.
- This will help in finding distribution, Standard Deviations and min-max of numerical columns
"""
hd.describe()    # Descriptive statistics for the numerical variables in the Dataset.
"""
- And, Display a random 10 rows to see variations in the Data.
"""
hd.sample(10)                            # Displays a random 10 rows from the dataset.
"""
- Inorder to handle the data better, finding the Data Types and Null Values. 
"""
hd.info()                         # Displays Data_Types and Null_Values in the Dataset.
"""
### Observations from Data Profiling:

- There are missing values(/NaN) in the data.
- Dataset consists of different dtypes.
- Skewness is observed in some columns.

Now, I have inputs for starting off with pre-profiling.
"""
"""
<a id=section302></a>
### 3.2 Pre profiling:

- With pandas profiling, an __interactive HTML report__ gets generated, which gives breif description about the columns of the dataset like the __counts, Histograms and Correlations__. Detailed information about __each column__, __coorelation between different columns__ and a __sample__ of dataset.<br/>
- It also gives __visual interpretation__ of each column in the data.
- Spread of the data is better understood by the distribution plot. 
- Grannular level analysis of each column.
"""
import pandas_profiling                          # Get a quick overview for all the variables using pandas_profiling.
profile = pandas_profiling.ProfileReport(hd)
profile.to_file('House_Data_before_preprocessing.html')   # HTML file will be downloaded to local workspace.
"""
Here, I have done Pandas Profiling before preprocessing our dataset, so I have named the html file as __House_Data_before_preprocessing.html__. Looking at the file will help developing useful insights from the dataset. <br/>

Below are insights from the preprofiling:
- There are columns with missing values on average of 4-8%.
- Also, some columns comprises of >90 zeros.
- Columns should be divided based on dtypes.
- Such as, 'Numerical and Float' of one set, 'Categorical' of another set and 'Boolean' of seperate set... for better handling.
- Categorical Columns should be carefully handled during encoding.
- Columns with no insights needs to be addressed.
- All the columns with 'Basement' characteristics are positively correlated to one another. 
- Same goes for 'Garage' involved columns as well.
"""
"""
<a id=section303></a>
### 3.3 Pre processing:
"""
"""
- Dealing with missing Values.
- Dealing with zeros.
- Splitting column data for more insights.
- Dropping the coulumn unsoughtful for data insights.
"""
"""
### 3.3.1 Dealing with missing Values:
"""
"""
- Creating a list for dropping the columns which are having >5% of missing values.
"""
DrpLst=[]  #DropList for droping the columns which are having >5% missing Values.
for i in hd.columns:
    if((hd[i].count())/1460)<0.95:
        DrpLst.append(i)
DrpLst
"""
### 3.3.2 Dealing with zeros:

"""
"""
- Appending the above created list with the column names which are having >80% zeros values. Considering that columns with >80% zeros would not help in making the predictions.
"""
for i in hd.columns:                   #DropList for droping the columns which are having >80% Zero Values.
    if(((hd[i]==0).sum())/1460)>0.80:
        if(i not in DrpLst):
            DrpLst.append(i)
len(DrpLst)  #There are 19 columns, which are having higher missing Values and Zeros.
"""
### 3.3.3 Dropping Missing Values and Zeros
"""
"""
- Now dropping the 19 columns from the data set which are having >5% missing values and >80% zeros.
- Also, Dropping the 'Id' column which is non-intrepretable.
"""
#hd.shape
DrpLst
DrpLst.append('Id')  #Dropping the column Id
hd = hd.drop(DrpLst, axis=1)
"""
<a id=section304></a>
### 3.4 Seperating columns with respect to data types (Num, Cat and Bool).

- Firstly, data should be seperated according to the dtypes before begining with preprocessing of the data.
- This helps in treating the missing values accordingly.
"""
#Code to seperate the Dataset on the basis of 'dtypes'
hdN=pd.DataFrame()
hdB=pd.DataFrame(hd['CentralAir'])
hdC=pd.DataFrame()

for i in hd.columns:
    if (hd[i].dtype=='int64') | (hd[i].dtype=='float64'):
        hdN[i]=hd[i]
    elif (hd[i].dtype=='object') & (i!='CentralAir'):
        hdC[i]=hd[i]
"""
<a id=section305></a>
### 3.5 Copying columns from Numerical Column to Categrical.
- The list of columns with years information is to be copied to categorical set(hdC).
- because, when replacing NaN values with the mean values during the treatment. 
- Missing or Nan Values in the Years involved columns would be filled with the mean.
"""
Lst=['YearBuilt','YearRemodAdd','MoSold','YrSold'] #List of columns which are categorical but, present in Numerical Data.

#Adding the Lst columns from Numerical(hdN) to Categorical(hdC) Data
for i in Lst:
    hdC[i]=hdN[i]


#Dropping Lst columns from Numerical(hdN) Data
hdN=hdN.drop(Lst,axis=1)
hdN.columns
"""
<a id=section306></a>
### 3.6 Treating Missing Values
"""
"""
### 3.6.1 Numerical
- Replacing the missing values in a column with the mean of values of that column.
"""
#Replacing Missing Values with the Mean 

for i in hdN.columns:
    if((hdN[i].count())!=1460):
        hdN[i]= hdN[i].replace(np.NaN , hdN[i].mean())

#hdN.isnull().sum() 
hdN.astype('int')
hdN.info()
"""
### 3.6.2 Categorical
"""
"""
- Finding the column with highest unique values. So that, we could group by and then, fill the NaN values with mode of the values of that column.
"""
hdC.nunique()  #Finding the nunique values   
"""
- Treating missing values with the mode after group by of 'YearBuilt' Column.
"""
for i in hdC.columns:
    if((hdC[i].count())!=1460):
        hdC[i] = hdC[i].fillna(hdC[i].mode()[0])
hdC.info()
hdC.columns
"""
## Feature Engineering
### Combining the less weigted categories in a column to single category to address the skewnes.
"""
#Reading House Data
#hd = pd.read_csv('https://raw.githubusercontent.com/insaid2018/Term-2/master/Projects/house_data.csv')
def FtEng(df,lst,k,pfx):
    for col in lst:
        pf = pfx
        ln = len(df[col])
        vlc = pd.DataFrame(df[col].value_counts())
        vlc['cats']=vlc.index
        
        
        for r in vlc[col]:
            if (r<(k*ln)):
                pf = pf + vlc[vlc[col]==r]['cats'][0]



        if (len(pf)<20):
            for t in vlc[col]:
                if (t<(k*ln)):
                    u = vlc[vlc[col]==t]['cats'][0]
                    df.loc[df[col]==u, col] = pf
            print('The new category of '+ col +' is '+ pf)


        elif (len(pf)>20):
            for e in vlc[col]:
                if (e<(k*ln)):
                    v = vlc[vlc[col]==e]['cats'][0]
                    df.loc[df[col]==v, col] = 'Others'
            print('The new category of '+ col +' is Others')

    return df

            #print(u)
    
CatColList = ['MSZoning','Street','LotShape','LandContour','Utilities','LotConfig','LandSlope','Neighborhood',
              'Condition1','BldgType','HouseStyle','RoofStyle']

hdC_Dist = FtEng(df=hdC,lst=CatColList, k=0.10, pfx='Skw_')
hdC.shape
hdC_Dist.shape
"""
<a id=section4></a>
## 4 Data Transformation

- Transformation of categorical and boolean columns are required in order to input data to the model.
- In this project, Categorical columns is transformed using Label Encoder.
- And Boolean Data is transformed using dummyification.
"""
"""
<a id=section401></a>
### 4.1 Label Encoding Categorical(hdC) Data:
- In simple words, Label Encoder will label the categories of a column in numerical representation.
- Post this, all the columns will be of 'int64'.
"""
"""
# Dist (UnSkew Testing)
"""
from sklearn.preprocessing import LabelEncoder
hdC_Dist = hdC_Dist.apply(LabelEncoder().fit_transform) # fitting on hdC set of data
#hdC_Dist.info()  
from sklearn.preprocessing import LabelEncoder
hdC = hdC.apply(LabelEncoder().fit_transform) # fitting on hdC set of data
hdC.info()          # Checking the Dtype
"""
<a id=section402></a>
### 4.2 Dummifying Variables of the Boolean(hdB) Data
- Dummyfication would make N categories in a column to N-1 number of columns.
- In this case, No as 0 and Yes as 1.
- Since, drop_first is True only the second column such as, CentralAir_Yes will be returned. 
"""
hdB=(pd.get_dummies(hdB, prefix='CentralAir', drop_first=True)).astype('int')
hdB.info()
"""
<a id=section403></a>
### 4.3 Merging all the columns
"""
"""
- Concatenating all the data sets that were seperated based on dtype for better handling. 
- The Final dataset:hdM to be passed to the Model.
"""
"""
# Dist (UnSkew Testing)
"""
hdM_Dist = pd.concat([hdN,hdB,hdC_Dist], axis=1)
hdM = pd.concat([hdN,hdB,hdC], axis=1)
hdM
"""
<a id=section404></a>
### 4.4 Outliers Detection and Handling
- Treating the outliers of the data using Inter Quantile Range.
- IQR is one of the most effective techniques in removing the outliers.
- The values over the quantile range are treated as outliers and removed.
"""
#Function for Outlier removal using Inter Qunatile Range
def OutlrRem(df,col):
    Q1 = df[col].quantile(0.25)
    Q3 = df[col].quantile(0.75)
    IQR = Q3-Q1
    LwrR = df[col].min()
    UprR = Q3 + (1.5*IQR)
    df = df.loc[(df[col]>LwrR)&(df[col]<UprR)]
    return df
hdM.nunique().nlargest(12)
hdM.shape  #Checking the shape before Outlier Removal
OutLst = ['LotArea','GrLivArea','BsmtUnfSF','1stFlrSF','TotalBsmtSF','SalePrice',
          'BsmtFinSF1','GarageArea','2ndFlrSF','MasVnrArea','WoodDeckSF','OpenPorchSF']
for j in OutLst:
    OutlrRem(hdM,j)

hdM.shape #Checking the shape after Outlier Removal
"""
- No change in the rows shows that there are no outliers in the Data.
"""
"""
<a id=section405></a>
### 4.5 Pandas Post-profiling
"""
import pandas_profiling
prof = pandas_profiling.ProfileReport(hdM)
prof.to_file('House_Data_after_preprocessing.html')
"""
<a id=section5></a>
## 5. Exploratory Data Analysis
"""
"""
<a id=section501></a>
### 5.1 Count of the Houses Built in each years
"""
plt.figure(figsize=(50,50))
#plt.title('House built over the years')
abc = sns.catplot("YearBuilt", data=hd, aspect=5, kind="count", color='Skyblue')
abc.set_xticklabels(rotation=90)
"""
### Observations:
- Number of houses sold that are built in 2006 are >60, which is the highest.
- Houses Built during the peroid 1978-1991 are lower than previous years, so do the sales.
- 268 out of 1460 houses sold were built during the years: 2003-2007. 

"""
"""
<a id=section502></a>
### 5.2 How many Houses were sold per year?
"""
plt.figure(figsize=(5,5))
#plt.title('House built over the years')
abc = sns.catplot("YrSold", data=hd, aspect=2, kind="count", color='Skyblue')
abc.set_xticklabels(rotation=90)
"""
### Observations:
- Nearly 23% houses were sold in 2009 and it is the highest.
- Lowest number of houses were sold in the year 2010.
"""
"""
<a id=section503></a>
### 5.3 Plotting the variation in Sale Price with change in the Lot Area and MSZoning of the house
"""
c=sns.scatterplot(x="LotArea", y="SalePrice",data=hdM)
"""
- From above, It can been seen that there are some outliers in LotArea. These houses are of very large area. Since, the count is very less and can overfit the model, removing them.
"""
#hd = hd[hd['LotArea']<45000]  # For Visualization and understanding hd(raw) and hdM(processed) are used.
hdM = hdM[hdM['LotArea']<45000]
c=sns.scatterplot(x="LotArea", y="SalePrice",hue='MSZoning',data=hd) #Outliers are removed.
plt.legend(bbox_to_anchor=(1.05, 0.9), loc=2, borderaxespad=0.)
"""
### Observations:
- The houses of Residential Low zone are spread all over the SalePrice.
- SalePrice for most of the houses in Residential Medium zone are in the range: 50,000 - 1,80,000 dollars.
- The houses above $1,60,000 saleprice are of Floating Village zone.
- Houses above 20,000 sq.ft. Lot Area are of Residential Low Density Zone.
"""
"""
<a id=section504></a>
### 5.4 Understanding the distribution of data for selected columns 
"""
"""
- The selected columns are ['MSZoning','BldgType','HouseStyle','RoofStyle','ExterQual','ExterCond','Foundation','BsmtQual', 'KitchenQual','Electrical','SaleType','SaleCondition']
"""
plt.figure(figsize=(20,20))
plt.tight_layout()
sns.countplot('MSZoning',data=hd, ax=plt.subplot2grid((4,3), (0,0)))
sns.countplot('SaleCondition',data=hd,ax=plt.subplot2grid((4,3), (0,1)))
sns.countplot('BldgType',data=hd, ax=plt.subplot2grid((4,3), (0,2)))
sns.countplot('HouseStyle',data=hd, ax=plt.subplot2grid((4,3), (1,0)))
sns.countplot('RoofStyle',data=hd, ax=plt.subplot2grid((4,3), (1,1)))
sns.countplot('ExterQual',data=hd,ax=plt.subplot2grid((4,3), (1,2)))
sns.countplot('ExterCond',data=hd,ax=plt.subplot2grid((4,3), (2,0)))
sns.countplot('Foundation',data=hd,ax=plt.subplot2grid((4,3), (2,1)))
sns.countplot('BsmtQual',data=hd,ax=plt.subplot2grid((4,3), (2,2)))
sns.countplot('KitchenQual',data=hd,ax=plt.subplot2grid((4,3), (3,0)))
sns.countplot('Electrical',data=hd,ax=plt.subplot2grid((4,3), (3,1)))
sns.countplot('SaleType',data=hd,ax=plt.subplot2grid((4,3), (3,2)))

plt.show() 
"""
### Observations:
- In MSZoning Column, the RL category is of >75% and the RM is of 15%.
- In SaleCondition, Normal sale is >75%, followed by approx. 10% of Abnorml and Partial each.
- In Building Type column, 1Fam is again >75%.
- The column: HouseStyle is also skewed but better than above columns. Because, 1Story is nearly 50% followed by 2Story of 23%.
- RoofStyle is Gable category for 75% houses.
- External Quality and External Condition have high similar such as, TA of 60% and Gd of around 20%.
- Basement Quality and Kitchen Quality are highly similar.
- Electrical and Sale Type are highly skewed, such as SBrkr and WD categories are of 80% respectively.
"""
"""
<a id=section505></a>
### 5.5 Understanding the impact of Remodelling on the SalePrice
"""
sns.jointplot("SalePrice","YearRemodAdd", data=hdM, kind='hex', color='DarkBlue')
"""
- From Above, It can been seen that there are some outliers in SalePrice. These houses are of high price aroun 700000. Since, the count is very less and they can overfit the model, removing them.
"""
#hdM=hdM[hdM['SalePrice']<400000]
hd=hd[hd['SalePrice']<400000]  # Using 'hd' dataset here such that to show the year with encoding in the plot
sns.jointplot("SalePrice","YearRemodAdd", data=hd, kind='hex', color='DarkBlue')
"""
### Observations:
- Density of the houses remodelled after 2000 is high.
- SalePrice is nearly normal distributed.
- On the otherhand, YearRemodAdd is unevenly distributed. And it makes sense because, remodelling depends on the condition of the house.
- There were very few houses remodelled during 1980-1990.
"""
"""
<a id=section506></a>
### 5.6 Relationship between Price and ['GarageArea','TotalBsmtSF','OverallCond','OverallQual']
"""
g = sns.PairGrid(hd, y_vars=['SalePrice'], x_vars=['GarageArea','TotalBsmtSF','OverallCond','OverallQual'], height=4)
g.map(sns.regplot)
plt.show()
"""
### Observations:

#### SalePrice vs GarageArea & TotalBsmtSF:
- An important observation from first two graphs is that SalePrice is low when there is no Garage and Basement.
- GarageArea and TotalBasementSquareFeet are linearly related to SalePrice individually.
- There are few outliers in both the graphs.

#### SalePrice vs OverallCond:
- It is very strange that the SalePrice is low even when the Over All Condition is High.
- House condition is negative related with Price.

#### SalePrice vs OverallQual:
- As the Over all Quality increases, Sale Price increases.
- Very few outliers are seen.

"""
"""
<a id=section507></a>
### 5.7 Which Neighborhood house are expensive?
"""
hd['Neighborhood'].unique()
plt.figure(figsize=(15,5))
c = sns.scatterplot(x="Neighborhood", y="SalePrice",hue='YearBuilt',data=hd)
plt.legend(bbox_to_anchor=(1.05, 0.9), loc=2, borderaxespad=0.)
c.set_xticklabels(( 'CollgCr','Veenker', 'Crawfor', 'NoRidge', 'Mitchel', 'Somerst','NWAmes', 
                   'OldTown', 'BrkSide', 'Sawyer', 'NridgHt', 'NAmes','SawyerW', 'IDOTRR', 
                   'MeadowV', 'Edwards', 'Timber', 'Gilbert','ClearCr', 'NPkVill', 'StoneBr', 
                   'Blmngtn', 'BrDale', 'SWISU','Blueste'),rotation=90);
"""
### Observations:
- The Graph shows that houses of the Neighborhood: 'NridgHt' are sold more. At the same, 'SalePrice' is from above average price to highest price and those houses are latest built.
- Only the houses of neighborhood:'Crawfor' are sold at higher price even though, they are built before 1950.
- All the houses built after 1950 are sold above 150000(approx.).
"""
"""
<a id=section508></a>
### 5.8 Finding correlation between the columns
"""
plt.figure(figsize=(50,50))
sns.heatmap(hdM.corr(), annot=True)
"""
- The plot is too complicated and confusing to understand. Hence, we shall concenrate only on the columns with higher correlation.
"""
t = 0.8 #Thershold for Correlation
cr = hdM.corr()
hdCor = cr[(cr>t)|(cr<(-t))]
plt.figure(figsize=(20,20))
sns.heatmap(hdCor, annot=True)
"""
## Observations:

Below are pair of columns which are correlated more than 80% threshold:
- [OverallQual and CentralAir_Y]
- [GrLivArea and TotalRmsAbvGr]
- [SalePrice and OverallQual]
- [Garage Area and Garage Cars]
- [Exterior1st and Exterior2nd]

In fact, one among the pairs can be dropped in order to reduce the computing time of the model. For now, no changes are done to the final 'hdM' dataset.

"""
"""
<a id=section6></a>
## 6. Data Preparation for Model Input
"""
"""
<a id=section601></a>
### 6.1 Scaling the Data
"""
"""
- The values in each column is of different scales.
- And this can lead to problems when fitting.
- Scaling will help to Normalize data to single scaler.
- Here, StandardScaler is used for scaling.
"""
from sklearn.preprocessing import StandardScaler
def scaling(x):
    sclr = StandardScaler().fit(x)  #Fitting the data 
    hdtf = sclr.transform(x)        #Transforming the data
    return hdtf

def postscaling(y,z):
    hdSc=pd.DataFrame(y)
    hdSc.columns = z.columns
    return hdSc
hdSc = postscaling(scaling(hdM),hdM) #Executing the function to check for errors.
hdSc = postscaling(scaling(hdM_Dist),hdM_Dist)
"""
<a id=section602></a>
### 6.2 Splitting the Data
"""
"""
- Functions for splitting the Data into X and Y before doing Train-Test Split
"""
def splitX(hdSc):
    X = pd.DataFrame()
    X = hdSc.loc[:,hdSc.columns!='SalePrice']
    return X

def splitY(hdSc):
    Y = pd.DataFrame()
    Y = hdSc['SalePrice']
    return Y
    
"""
- Calling the functions for execution and error check.
"""
X = splitX(hdSc)
Y = splitY(hdSc)
print('Shape of X is ', X.shape)   #Printing X Shape
print('Shape of X is ', Y.shape)   #Printing Y Shape
"""
- Splitting the data X and Y into Train-Test Split with Test size of 25%.
"""
from sklearn.model_selection import train_test_split
def splitTrTt(X,Y):
    return train_test_split(X,Y, random_state=1, test_size=0.25)
X_train, X_test, Y_train, Y_test = splitTrTt(X,Y)  #Calling the function to check for errors
#Printing the shapes of split train and test data.
def printshape():
    print('X_Train Shape : ',X_train.shape)
    print('X_Test Shape : ',X_test.shape)
    print('X_Train Shape : ',Y_train.shape)
    print('X_Test Shape : ',Y_test.shape)

printshape()
"""
<a id=section7></a>
## 7. Model Implementation
"""
"""
<a id=section701></a>
### 7.1 Linear Regression Model
"""
"""
### 7.1.1 Linear Regression Deafult Model
- Linear Regression is choosen as a primary model for train and prediction.
- The model below is basic meaning without any parameters.
"""
from sklearn.linear_model import LinearRegression
def lnrReg(X_train, Y_train, X_test):
    lr = LinearRegression()                  #Instantiating the Model
    lr = lr.fit(X_train, Y_train)            #Training the Model with X_Train and Y_Train
    Y_pred_train = lr.predict(X_train)       #Prediction on X_Train
    Y_pred_test = lr.predict(X_test)         #Prediction on Y_Train
    return Y_pred_train,Y_pred_test

Y_pred_train,Y_pred_test = lnrReg(X_train, Y_train, X_test)

from sklearn import metrics
print('Train Acc' + format(metrics.mean_absolute_error(Y_train,Y_pred_train),'.5f'))

print('Test Acc' + format(metrics.mean_absolute_error(Y_test,Y_pred_test),'.5f'))
Y_pred_train,Y_pred_test = lnrReg(X_train, Y_train, X_test)

from sklearn import metrics
print('Train Acc' + format(metrics.mean_squared_error(Y_train,Y_pred_train),'.5f'))

print('Test Acc' + format(metrics.mean_squared_error(Y_test,Y_pred_test),'.5f'))
"""
### 7.1.2 Linear Regression GridSearchCV Model
- Linear Regression with GridSearchCV for train and prediction.
- The model below would perform GridSearchCV within the parameters provided such as, **{'normalize':[True,False], 'copy_X':[True,False],'fit_intercept':[True,False]}**
- And then, considers the best estimator for predictions.
"""
from sklearn.model_selection import GridSearchCV

def LinGridCV(X_train, Y_train,X_test):
    parameters = {'normalize':[True,False], 'copy_X':[True,False],'fit_intercept':[True,False]}
    lr = LinearRegression()
    lr = GridSearchCV(lr,parameters,cv=15)
    lr.fit(X_train, Y_train)
    
    lin = lr.best_estimator_
    print('Best Linear Regressor GSCV Estimator is',lin)
    lin = lin.fit(X_train, Y_train)
    Y_pred_train_LinGCV = lin.predict(X_train)
    Y_pred_test_LinGCV = lin.predict(X_test)
    return Y_pred_train_LinGCV,Y_pred_test_LinGCV

"""
<a id=section702></a>
### 7.2 Decision Tree Regressor
"""
"""
### 7.2.1 Decision Tree Regressor Default Model
- DecisionTree Regressor is choosen as a secondary model for train and prediction.
- The model below is basic meaning without any parameters.
"""
from sklearn.tree import DecisionTreeRegressor
def DecTreReg(X_train, Y_train,X_test):
    dtr = DecisionTreeRegressor(random_state=3)
    dtr = dtr.fit(X_train, Y_train)
    Y_pred_train_dtr = dtr.predict(X_train)
    Y_pred_test_dtr = dtr.predict(X_test)
    return Y_pred_train_dtr,Y_pred_test_dtr
Y_pred_train_dtr,Y_pred_test_dtr = DecTreReg(X_train, Y_train, X_test)

from sklearn import metrics
print('Train Acc' + format(metrics.mean_absolute_error(Y_train,Y_pred_train_dtr),'.5f'))

print('Test Acc' + format(metrics.mean_absolute_error(Y_test,Y_pred_test_dtr),'.5f'))
"""
### 7.2.2 Decision Tree Regressor Grid SearchCV Model
- DecisionTree Regressor with GridSearchCV for train and prediction.
- The model below would perform GridSearchCV within the parameters provided such as, **{'criterion':['mse','mae'],'max_depth': range(2,20),'max_features': ['auto', 'sqrt', 'log2', None]}**
- And then, considers the best estimator for predictions.
"""
def dtrGridCV(X_train, Y_train,X_test):
    parameters = {'criterion':['mse','mae'],'max_depth': range(2,20),
                             'max_features': ['auto', 'sqrt', 'log2', None]}
    dt = DecisionTreeRegressor()
    dt = GridSearchCV(dt,parameters, cv=10)
    dt.fit(X_train, Y_train)
    
    dct = dt.best_estimator_
    print('Best Decision Tree GSCV Estimator is',dct)
    dct = dct.fit(X_train, Y_train)
    Y_pred_train_dtrGCV = dct.predict(X_train)
    Y_pred_test_dtrGCV = dct.predict(X_test)
    return Y_pred_train_dtrGCV,Y_pred_test_dtrGCV
"""
<a id=section703></a>
### 7.3 Random Forest Regressor
"""
"""
### 7.3.1 Random Forest Regressor Default Model
- Random Forest Regressor with default parameters for train and prediction.
"""
from sklearn.ensemble import RandomForestRegressor
def RanForReg(X_train, Y_train,X_test):
    rfr = RandomForestRegressor(random_state=4)
    rfr = rfr.fit(X_train, Y_train)
    Y_pred_train_rfr = rfr.predict(X_train)
    Y_pred_test_rfr = rfr.predict(X_test)
    return Y_pred_train_rfr,Y_pred_test_rfr
Y_pred_train_rfr,Y_pred_test_rfr = RanForReg(X_train, Y_train, X_test)

from sklearn import metrics
print('Train Acc' + format(metrics.mean_absolute_error(Y_train,Y_pred_train_rfr),'.5f'))

print('Test Acc' + format(metrics.mean_absolute_error(Y_test,Y_pred_test_rfr),'.5f'))
Y_pred_train_rfr,Y_pred_test_rfr = RanForReg(X_train, Y_train, X_test)

from sklearn import metrics
print('Train Acc' + format(metrics.mean_squared_error(Y_train,Y_pred_train_rfr),'.5f'))

print('Test Acc' + format(metrics.mean_squared_error(Y_test,Y_pred_test_rfr),'.5f'))
"""
### 7.3.2 Random Forest Regressor GridSearchCV Model
- Random Forest Regressor with GridSearchCV for train and prediction.
- The model below would perform GridSearchCV within the parameters provided such as, **{'n_estimators':[50,100,150,200],'criterion':['mse', 'mae'],'max_depth': range(3,8),'min_samples_leaf':range(2,4),'max_features': ['auto', 'sqrt', 'log2', None]}**
- And then, considers the best estimator for predictions.
"""
def rfrGridCV(X_train, Y_train,X_test):

    param_grid = {'n_estimators':[50,100,150,200],'criterion':['mse', 'mae'],'max_depth': range(3,8),
                  'min_samples_leaf':range(2,4),'max_features': ['auto', 'sqrt', 'log2', None]}
    rf = RandomForestRegressor(random_state=5)
    rf = GridSearchCV(estimator=rf,param_grid=param_grid, cv=10)
    rf.fit(X_train, Y_train)
    
    rfc = rf.best_estimator_
    print('Best Random Forest GSCV Estimator is',rfc)
    rfc = rfc.fit(X_train, Y_train)
    Y_pred_train_rfcGCV = rfc.predict(X_train)
    Y_pred_test_rfcGCV = rfc.predict(X_test)
    return Y_pred_train_rfcGCV,Y_pred_test_rfcGCV
"""
<a id=section8></a>
## 8. Metrics for Model Performance
"""
"""
- Function to iterate over the above models and predict values.
- The predicted values are stored into Data Frame for comparision.
- MAE, MSE, RMSE and R-Squared metrics are used to measure the performance of the model.
- Not only the predictions on Test but, Predictions on Train are also considered for better insights.
"""
def Modl(x):
    
    #Scaling
    hdSc = postscaling(scaling(x),x)
    
    
    X = splitX(hdSc)  #X Split
    Y = splitY(hdSc)  #Y Split

    
    
    #Train-Test Split
    X_train, X_test, Y_train, Y_test = splitTrTt(X,Y)
    Y_pred_train,Y_pred_test = lnrReg(X_train, Y_train,X_test)
    Y_pred_train_dtr,Y_pred_test_dtr = DecTreReg(X_train, Y_train,X_test)
    Y_pred_train_LinGCV, Y_pred_test_LinGCV = LinGridCV(X_train, Y_train,X_test)
    Y_pred_train_dtrGCV, Y_pred_test_dtrGCV = dtrGridCV(X_train, Y_train,X_test)
    Y_pred_train_rfr,Y_pred_test_rfr = RanForReg(X_train, Y_train,X_test)
    Y_pred_train_rfcGCV,Y_pred_test_rfcGCV = rfrGridCV(X_train, Y_train,X_test)
    
    
    a = []
    c = [(Y_pred_train,Y_pred_test),(Y_pred_train_dtr,Y_pred_test_dtr),
         (Y_pred_train_LinGCV, Y_pred_test_LinGCV),(Y_pred_train_dtrGCV, Y_pred_test_dtrGCV),
         (Y_pred_train_rfr,Y_pred_test_rfr),(Y_pred_train_rfcGCV,Y_pred_test_rfcGCV)]

    
    #For loop to iterate over the models for calculation of metrics
    from sklearn import metrics
    for i,j in c: 
        metlst = [format(metrics.mean_absolute_error(Y_train,i),'.5f'),
                  format(metrics.mean_absolute_error(Y_test,j),'.5f'),
                  format(metrics.mean_squared_error(Y_train,i),'.5f'),
                  format(metrics.mean_squared_error(Y_test,j),'.5f'),
                  format(np.sqrt(metrics.mean_squared_error(Y_train,i)),'.5f'),
                  format(np.sqrt(metrics.mean_squared_error(Y_test,j)),'.5f'),
                  format(metrics.r2_score(Y_train,i),'.5f'),
                  format(metrics.r2_score(Y_test,j),'.5f')]
        a.append(metlst)

    #DataFrame for storing all the measuring values from models    
    b = pd.DataFrame(a,
                     columns=['MAE_Train', 'MAE_Test','MSE_Train', 'MSE_Test','RMSE_Train', 'RMSE_Test','R2_Train', 'R2_Test'],
                     index = ['Linear Regression','DecisionTree Reg','Linear Grid SearchCV','DecisionTree Reg GSCV',
                              'RandomForest Reg', 'RandomForest Reg GSCV']).astype('float')
    return b

#Modl(hdM)               #Running the function above with hdM Data
# Results are as in below block
"""
Best Linear Regressor GSCV Estimator is LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

Best Decision Tree GSCV Estimator is DecisionTreeRegressor(ccp_alpha=0.0, criterion='mae', max_depth=8,
                      max_features='sqrt', max_leaf_nodes=None,
                      min_impurity_decrease=0.0, min_impurity_split=None,
                      min_samples_leaf=1, min_samples_split=2,
                      min_weight_fraction_leaf=0.0, presort='deprecated',
                      random_state=None, splitter='best')
                      

Best Random Forest GSCV Estimator is RandomForestRegressor(bootstrap=True, ccp_alpha=0.0, criterion='mse',
                      max_depth=7, max_features='sqrt', max_leaf_nodes=None,
                      max_samples=None, min_impurity_decrease=0.0,
                      min_impurity_split=None, min_samples_leaf=2,
                      min_samples_split=2, min_weight_fraction_leaf=0.0,
                      n_estimators=150, n_jobs=None, oob_score=False,
                      random_state=5, verbose=0, warm_start=False)
                                     
|     |MAE_Train|MAE_Test|MSE_Train|MSE_Test|RMSE_Train|RMSE_Test|R2_Train|R2_Test|
|-----|:-------:|:------:|:-------:|:------:|:--------:|:-------:|:------:|:-----:|
|Linear Regression|0.24518|0.24761|0.15789|0.15816|0.39735|0.39769|0.83844|0.85142|
|DecisionTree Reg|0.00000|0.31187|0.00000|0.19404|0.00000|0.44050|1.00000|0.81771|
|Linear Grid SearchCV|0.24518|0.24761|0.15789|0.15816|0.39735|0.39769|0.83844|0.85142|
|DecisionTree Reg GSCV|0.18998|0.34592|0.10917|0.27036|0.33041|0.51996|0.88829|0.74601|
|RandomForest Reg|0.08362|0.20027|0.02121|0.09132|0.14563|0.30219|0.97830|0.91421|
|RandomForest Reg GSCV|0.17308|0.22712|0.07388|0.14217|0.27181|0.37706|0.92440|0.86643|
"""
"""
**Observations post running all the Models**
"""
"""
- The metrics obtained from different version of models shows that **Random Forest Regressor Default** Model gave the better results.
- On the otherhand, **Linear Regression: Basic and Grid SearchCV** Models gave same results.
- Key point is, Default models are better performers than parameters tuned Grid SearchCV ones.
- Linear Regression or RandomForest Regressor Model with default parameters could be used for predictions based on time consumption factor.
"""
"""
<a id=section9></a>
## 9. Interpreting Model Coefficients
"""
"""
<a id=section901></a>
### 9.1 Plotting Coefficients of the columns
- Let us understand more about the model and each columns influence on Sale Price of the house.
- For that, the best performed model(Linear Regression Default Model) from above table is fit and coefficients are extracted.
- The coefficient against each column is collected in data frame.
- Plotting the values from data frame will show the influence or dependency of each column.
"""
# Model Instantiating 
from sklearn.linear_model import LinearRegression
lr = LinearRegression()               # Linear Regressor with default parameters
lr = lr.fit(X_train,Y_train)


cols=X_train.columns.to_list()         # Passing columns of X_train to list called 'cols'
cols.insert(0,'Intercept')             # Inserting 'Intercept' at the top for 'cols' list
coef = lr.coef_.tolist()               # Passing coefficients of the Default to list called 'cols'
coef.insert(0, lr.intercept_)          # Inserting Intercept value to 'coef' list


coefs = pd.DataFrame(coef, columns=['Coefficients'], index=cols)  #Creating DataFrame for better plotting
"""
- Plotting the coefficient for each column and the intercept.
"""
coefs.plot(kind='barh', figsize=(12, 12))
"""
- From the Coefficients Plot, It is understood that some of the columns are having very low coefficient values.
- And, these columns could create noise for the model.
- Hence, data will be fed to the model by dropping the columns based on different thresholds.
- This will help in fine tuning the Model.
"""
"""
<a id=section902></a>
### 9.2 Cases for tuning the Model
- Let us divide the tuning into different cases based on thresholds.
- Below is the function for checking the metrics for each case.
"""
def Mdl(x):  
  #Scaling
  hdSc = postscaling(scaling(x),x)
      
  X = splitX(hdSc)  #X Split
  Y = splitY(hdSc)  #Y Split

      
  #Train-Test Split
  X_train, X_test, Y_train, Y_test = splitTrTt(X,Y)
  
  #Model Implementation
  Y_pred_train,Y_pred_test = lnrReg(X_train, Y_train,X_test)
  Y_pred_train_rfr,Y_pred_test_rfr = RanForReg(X_train, Y_train,X_test)

  from sklearn import metrics
  print('Linear Regression Model Output')
  print('MAE :', format(metrics.mean_absolute_error(Y_test,Y_pred_test),'.5f'))
  print('MSE :', format(metrics.mean_squared_error(Y_test,Y_pred_test),'.5f'))
  print('RMSE :' , format(np.sqrt(metrics.mean_squared_error(Y_test,Y_pred_test)),'.5f'))
  print('R2 Score :' , format(metrics.r2_score(Y_test,Y_pred_test),'.5f'))

  print('RandomForest Regressor Model Output')
  print('MAE :', format(metrics.mean_absolute_error(Y_test,Y_pred_test_rfr),'.5f'))
  print('MSE :', format(metrics.mean_squared_error(Y_test,Y_pred_test_rfr),'.5f'))
  print('RMSE :' , format(np.sqrt(metrics.mean_squared_error(Y_test,Y_pred_test_rfr)),'.5f'))
  print('R2 Score :' , format(metrics.r2_score(Y_test,Y_pred_test_rfr),'.5f'))
"""
#### **Case-1:** Dropping the columns with Coefficient values within the threshold: <0.02 and >-0.02
 
"""
hdCoef1 = hdM.drop(['MoSold','SaleType','PavedDrive','Electrical','HeatingQC','Heating','BsmtFinType2',
                    'Foundation','ExterCond','Condition1','LotConfig','Utilities','Street','MSZoning',
                    'CentralAir_Y','OpenPorchSF','GarageArea','BedroomAbvGr'], axis=1)
Mdl(hdCoef1)
"""
#### **Case-2:** Dropping the columns with Coefficient values within the threshold: <0.01 and >-0.01
"""
hdCoef2 = hdM.drop(['Heating','BsmtFinType2','ExterCond','HouseStyle','LandContour','LotShape','HalfBath','FullBath'], axis=1)
Mdl(hdCoef2)
"""
- Above experiment shows:
 - In Case-1, Models are under performing.
 - In Case-2, Linear Regression model is giving slightly improved metrics but, RandomForest Regressor results didn't get better.
"""
"""
<a id=section10></a>
## 10. Conclusion
"""
"""
   From the metrics obtained by performing experiments with different Models, It can be concluded that **Linear Regression** and **Random Forest Regressor** are better fit for this data. And, Removing the Columns with coefficient values <0.01 and >-0.01 has shown slight improvements in the metrics.

"""
"""
Although, Most of the aspects of cleaning, handling, transforming and many... in processing data are covered in the project. There is still room for experiments. Hence, Below are some key points to be considered for future improvements.
- Impute missing values(>5%) of the dropped columns.
- Check importance of >80% zero value columns when predicting Sale Price.
- Try to reduced the skewness in columns with feature engineering techniques.
Ex: Merge lower categories of a column into single category.
- Use Random SearchCV while passing data to make the model more generalized.
- Try other Regression Models to check for better predictions.
"""