# This Python 3 environment comes with many helpful analytics libraries installed # It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python # For example, here's several helpful packages to load import numpy as np # linear algebra import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv) # Input data files are available in the read-only "../input/" directory`````````` # For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory import os for dirname, _, filenames in os.walk('/kaggle/input'): for filename in filenames: print(os.path.join(dirname, filename)) # You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" # You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session """ ___ """ """ **House Price prediction:**<br> I'm attempting this competition a second time here, my first attempt was 4 months ago, with a public score of 0.16806, when I didn't analyse the data extensively and feature selection was based on assumptions. My goal this time is to get better at my **EXPLORATORY DATA ANALYSIS-EDA** and overall process and not just to improvise the score. We will be analysing data extensively based on statistics and not on assumptions<br><br> **Update:** EDA and Feature Engineered data in this notebook coupled with a basic Linear Regression model got an improvement in the score(rmse: 0.14285) compared to my previous notebook attempt using Hyperparameter tuned Lasso Regression(rmse: 0.16806), I think that implies deeper Data Analysis alone can greatly improve the performance without the need for using complex models for training. May be that's why we hear lot of Data Scientists stressing on the importance of preparing and working on the data rather than fancy models.<br><br>Please upvote if you find the notebook useful """ """ *** """ """ Our goal is to predict the SalePrice of a house by training our model on a given set of data.<br> In this dataset, we have 81 columns and 1460 rows.<br> SalePrice column is the **target/dependant variable** whose value, the model should be able to predict <br> Remaining 80 columns are the **features/independent variables** whose values will affect the value of the target. <br> This is a regression problem since the prediciton variable is a continuous variable (Price value of a house) """ """ ___ """ """ ## Index #### Exploratory Data Analysis & Feature Selection 1. [Inspect data](#1.-Inspect-Data) 2. [Handle missing values](#2.-Handle-missing-values) 2.1 [Missing values in Numerical features](#2.1-Missing-values-in-Numerical-features)<br> 2.2 [Missing values in Categorical features](#2.2-Missing-values-in-Categorical-features) 3. [Data analysis & Feature selection - Numerical](#3.-Data-analysis-&-Feature-selection---Numerical) 3.1 [Pairwise Correlation](#3.1-Pairwise-Correlation) 3.2 [Correlation matrix & Heatmap](#3.2-Correlation-matrix-&-Heatmap) 3.3 [Feature Analysis based on Correlation](#3.3.-Feature-Analysis-based-on-Correlation) 4. [Data analysis & Feature selection - Categorical](#4.-Data-analysis-&-Feature-selection---Categorical) #### Data Pre-Processing 5. [Skewness and Normalisation](#5.-Skewness-and-Normalisation) 6. [Training set-Validation set split, Pre-processing steps: OneHotEncoding, Standardisation](#6.-Training-set-Validation-set-split,-Pre-processing-steps:-OneHotEncoding,-Standardisation) #### Model Training, Evaluation & Validation 7. [Extreme Gradient Boosting(XGB) Regressor](#7.-Extreme-Gradient-Boosting(XGB)-Regressor) 7.1 [Hyperparameter Tuning](#7.1-Hyperparameter-Tuning) 7.2 [Plotting Decision trees](#7.2-Plotting-Decision-trees) 7.3 [Feature importance](#7.3-Feature-importance) 7.4 [Predicted vs Actual plot](#7.4-Predicted-vs-Actual-plot) 7.5 [Residual plot](#7.5-Residual-plot) 7.6 [Learning Curve-Bias/variance Tradeoff](#7.6-Learning-Curve-Bias/variance-Tradeoff) #### Target Prediction 8. [Predict target for Test Data](#8.-Predict-target-for-Test-Data)<br><br> References: - [Inspiration from Gabriel Atkin's ML process pipeline video](https://www.youtube.com/watch?v=zwYHloLXH0c&t=5885s) - [Kaggle notebook by Abu Bakar](https://www.kaggle.com/bakar31/eda-house-price-prediction) - [My earlier attempt of this problem](https://www.kaggle.com/iravad/house-price-advanced-regression-techniques-lasso) """ """ We import the library requirements here """ # import libraries pd.options.display.max_columns=None pd.options.display.max_rows=85 import pandas as pd import numpy as np import seaborn as sns import matplotlib.pyplot as plt from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.preprocessing import OneHotEncoder import xgboost as xgb from xgboost import plot_tree, plot_importance from sklearn.model_selection import GridSearchCV from sklearn.metrics import mean_absolute_error, mean_squared_error from sklearn.model_selection import learning_curve """ Then we read in the Train, Test and sample_submission datasets """ # read training and testing CSV data to dataframes train & test respectively train = pd.read_csv("../input/house-prices-advanced-regression-techniques/train.csv") test = pd.read_csv("../input/house-prices-advanced-regression-techniques/test.csv") sample_submission = pd.read_csv("/kaggle/input/house-prices-advanced-regression-techniques/sample_submission.csv") """ ## 1. Inspect Data """ """ We start inspecting the Training data, let's take a visual look at the features and Target and let's try to understand the given set of properties of a house. We may not understand the data in-depth for now, that's okay. Let's just get to know about the data first. We will do a full exploratory analysis later to get into the technicalities of each feature """ # inspect the shape of the train & test dataframes print(train.shape) print(test.shape) """ **Train** has 1460 data observations, 80 features + 1 target(SalePrice column)<br> **Test** has 1459 data observations, 80 features<br> We have to train a model using the given train data and predict SalePrice for the test data<br> The **EDA findings from train data will be used for preparing and pre-processing the test data** """ """ Most datasets for predicitons have both *Numerical* and *Categorical* data<br>Features like __LotFrontage, LotArea__ have numerical data giving details of area in square feet<br>Features like __Lotshape, Utilities__ denote categories like Reg(Regular)/AllPub(All public Utilities (E,G,W,& S)) """ # inspect train train """ Looking at the datatypes, non-null count of all the features, we will roughly have an idea about the columns and their missing values in general """ # dataframe information train.info() """ Let's observe the satistical details for __all numerical features__ like count, minimun value, maximum value, mean value, standard deviation etc. <br> This will help us to understand the data more statistically <br> For example, <br> In the entire dataset which is the oldest year of construction and latest year of construction of house?<br>Also, observe the columns __MSSubClass, OverallQual__ & __OverallCond__, could you spot anything different in these columns? """ # statistics of train train.describe() """ The __YearBuilt__ - *Original construction date*, statistics shows min value of original construction as 1872 and max values as 2010!<br> Please note that __MSSubClass, OverallQual__ & __OverallCond__ have **numerical data** but on a closer inspection of min, 25%, 50%, 75% & max values, they are **Categorical variables** indicating categories & **ratings** as numbers! We can confirm this by looking at the data description of these features """ """ MSSubClass: Identifies the type of dwelling involved in the sale. 20 1-STORY 1946 & NEWER ALL STYLES 30 1-STORY 1945 & OLDER 40 1-STORY W/FINISHED ATTIC ALL AGES 45 1-1/2 STORY - UNFINISHED ALL AGES 50 1-1/2 STORY FINISHED ALL AGES 60 2-STORY 1946 & NEWER 70 2-STORY 1945 & OLDER 75 2-1/2 STORY ALL AGES 80 SPLIT OR MULTI-LEVEL 85 SPLIT FOYER 90 DUPLEX - ALL STYLES AND AGES 120 1-STORY PUD (Planned Unit Development) - 1946 & NEWER 150 1-1/2 STORY PUD - ALL AGES 160 2-STORY PUD - 1946 & NEWER 180 PUD - MULTILEVEL - INCL SPLIT LEV/FOYER 190 2 FAMILY CONVERSION - ALL STYLES AND AGES OverallQual: Rates the overall material and finish of the house 10 Very Excellent 9 Excellent 8 Very Good 7 Good 6 Above Average 5 Average 4 Below Average 3 Fair 2 Poor 1 Very Poor OverallCond: Rates the overall condition of the house 10 Very Excellent 9 Excellent 8 Very Good 7 Good 6 Above Average 5 Average 4 Below Average 3 Fair 2 Poor 1 Very Poor """ """ let's convert them into **Categorical datatypes** """ # converting 'int' datatype to 'object' datatype for train data train["MSSubClass"] = train["MSSubClass"].astype(str) train["OverallQual"] = train["OverallQual"].astype(str) train["OverallCond"] = train["OverallCond"].astype(str) # converting 'int' datatype to 'object' datatype for test data test["MSSubClass"] = test["MSSubClass"].astype(str) test["OverallQual"] = test["OverallQual"].astype(str) test["OverallCond"] = test["OverallCond"].astype(str) """ Let's see **the list of Numerical and Categorical features** in the dataset """ # categorical columns include "object" datatypes cat_cols = train.select_dtypes(include="object").columns cat_cols # numerical columns exclude "object" datatypes num_cols = train.select_dtypes(exclude="object").columns num_cols """ Let's take a look at the **target** """ # SalePrice column train["SalePrice"] # SalePrice histogram plt.figure(figsize=(10, 8)) sns.histplot(train["SalePrice"], kde=True) """ The SalePrice for maximum number of houses in the data is in between the price range 100000 and 200000. The data is **skewed** with more houses in the price range upto $ 300000 and on the right side of the histogram, there are very less number of houses with more SalePrice """ """ #### [Go to Index](#Index) """ """ *** """ """ ## 2. Handle missing values """ """ It's always good to analyse for any missing values in a dataset and clean the dataset before training our model """ """ Now, it's time for **datacleaning**<br>There are many ways to deal with null values: - drop the rows with null values(we're not choosing this method here since information will be lost if rows are dropped) - **drop** the entire column with null values - if the data is missing at random, - if column becomes redundant & - if we don't have a reason to have that column for training - **impute** missing values - if the data is not missing at random, - if we can find why the values are missing & - we have a reason to keep the column for training Before deciding on the method(dropping or imputing), list out all the Null/missing values from the entire dataset """ # missing value in train data missing_values = train.isna().sum().sort_values(ascending=False) missing_values # visualising null values in train sns.heatmap(train.isnull(), cbar=False) """ There are missing values in both categorical and numerical features, each of them need separate treatment for dealing with missing values<br> First, let's see the missing values in __Numerical features__ """ """ #### [Go to Index](#Index) """ """ *** """ """ ## 2.1 Missing values in Numerical features """ # missing values in numerical features of train (train.select_dtypes(exclude='object')).isnull().sum().sort_values(ascending=False).head(10) """ For numerical variables, we will **impute** missing values with the mean/median of the column data (depending on the type of value column<br> for example, for area values we can impute using mean and for age/year values we can impute using median values) """ """ __LotFrontage__ has 259 null values<br>LotFrontage: Linear feet of street connected to property """ # getting unique values and their counts in LotFrontage column train["LotFrontage"].value_counts() """ Let's observe this feature's relationship with target """ #plot feature LotFrontage vs target SalePrice plt.figure(figsize=(8,4)) ax=sns.regplot(y="LotFrontage", x="SalePrice", data=train) plt.setp(ax.get_xticklabels(), rotation=90) plt.show() """ *LotFrontage* has a linear relationship with *SalePrice*<br>let's impute the missing values using the mean value of the column """ # impute mean for missing values train["LotFrontage"].fillna(train["LotFrontage"].mean(), inplace=True) # impute missing values in test data test["LotFrontage"].fillna(train["LotFrontage"].mean(), inplace=True) """ **GarageYrBlt** is the next numerical feature with 81 missing values<br>GarageYrBlt: Year garage was built <br>Let's analyse if the year the garage was built is the same as the *YearBuilt*(the year the house was built). <br> Garages *might* have been built along with the house at the same time, let's look into the statistcs<br>If they're the same, we can drop the *GarageYrBuilt* feature since it may be regarded as duplicate information """ train[["GarageYrBlt", "YearBuilt"]] # plotting GarageYrBlt and YearBuilt vs SalePrice plt.figure(figsize=(8,4)) sns.scatterplot(x="GarageYrBlt", y="SalePrice", data=train, color = 'r', alpha=0.4) sns.scatterplot(x="YearBuilt", y="SalePrice", data=train, color='b', alpha=0.4) plt.legend(labels = ["GarageYrBlt", "YearBuilt"]) """ The values look same except for few, many data points overlap<br>Statistics can help to get the real numbers! """ # comparing if the row values are same for the features compare = np.where(train["GarageYrBlt"] == train["YearBuilt"], True, False) comparison = np.array(compare) np.unique(comparison, return_counts=True) """ The values are same for 1089 data points(75% of data)<br>so we cannot drop the *GarageYrBlt* feature, for 25% of houses, Garage built year is different.<br> Let's locate the null values in the dataframe and investigate further how to impute those missing 81 values """ # some of the null value rows in GarageYrBlt train.loc[train["GarageYrBlt"].isnull()].head() """ Looking in the dataframe, we have null values in all columns related to Garage(GarageType, GarageYrBlt, GarageFinish, GarageCars, GarageArea, GarageQual, GarageCond) for those 81 rows. That implies **those houses don't have a garage, so we cannot impute mean value here for *GarageYrBlt*, impute '0' for the null values** """ # impute mean for missing values train["GarageYrBlt"].fillna(0, inplace=True) # impute missing values in test data test["GarageYrBlt"].fillna(0, inplace=True) """ Now, we've managed the missing values for Numerical Features. Let's analyse the missing values in __Categorical features__ """ """ #### [Go to Index](#Index) """ """ *** """ """ ## 2.2 Missing values in Categorical features """ # missing values in categorical features of train (train.select_dtypes(include='object')).isnull().sum().sort_values(ascending=False).head(20) """ We're having more null values in these columns:<br> **PoolQC 1453<br> MiscFeature 1406<br> Alley 1369<br> Fence 1179**<br><br>They are **more than 80% of missing values** in each of the above columns, if missing values are more than 30% of total data we can drop the columns assuming that they don't contribute to overall performance. Imputing them with mean/median/mode without proper analysis will lead to bias in the model<br>Let's **analyse the reason** for the large percentage of missing data in certain features. Sometimes, missing values are not because of entry error/at random, they may be missing intentionally because the data is not applicable/intended to be in any of the categories """ """ Exploring __PoolQC__ feature: """ """ Why do you think *PoolQC* has 1453 null values? Investigate...<br>PoolQC: Pool quality Ex Excellent Gd Good TA Average/Typical Fa Fair NA No Pool """ train["PoolQC"].value_counts() """ PoolArea: Pool area in square feet <br>is a feature related to *PoolQC*, inspect """ # get unique PoolArea value counts train["PoolArea"].value_counts() """ Now, it's clear that **PoolArea is 0** for same 1453 datapoints, that means 1453 houses in data **don't have pool**<br>Clearly *PoolArea* & *PoolQC* have **very low variance/nearly zero variance**, they won't help in improving the model performance, so we can drop *PoolArea & PoolQC* """ """ #### Variance is the measure of spread between the unique values/categories in a column. If more than 95% of the values in the column belong to a specific category then we say the column has low variance. The 5% of values in other categories cannot help in prediction and the column is considered to be a constant. The column will not improve the model performance in predicting the target, so we can drop the column """ # drop PoolArea & PoolQC columns train.drop(columns=["PoolArea", "PoolQC"], axis=1, inplace=True) # drop PoolArea & PoolQC columns in test data test.drop(columns=["PoolArea", "PoolQC"], axis=1, inplace=True) """ Next column with more null values is __MiscFeature__, let's investigate the reason<br> MiscFeature: Miscellaneous feature not covered in other categories Elev Elevator Gar2 2nd Garage (if not described in garage section) Othr Other Shed Shed (over 100 SF) TenC Tennis Court NA None """ # getting unique values and their counts in MiscFeature column train["MiscFeature"].value_counts() """ *MiscFeature* has 54 datapoints in 4 categories, we suspect missing 1406 datapoints may belong to the 'NA-None' category<br>Even if we impute the 1406(96%) datapoints with 'NA', the overall **variance** for the column will be **very low** & it will not contribute to the performance of the model. So drop the column """ # drop MiscFeature column train.drop(columns=["MiscFeature"], axis=1, inplace=True) # drop MiscFeature column in test data test.drop(columns=["MiscFeature"], axis=1, inplace=True) """ Next feature to have more missing values is **Alley**, it has 1369 null values<br> Alley: Type of alley access to property Grvl Gravel Pave Paved NA No alley access """ # getting unique value count for Alley train["Alley"].value_counts() """ *Alley* has only 2 categories with 91 values<br> Missing 1369 values contribute to 93.7% of total data(threshold for low variance is 95%), so cannot be considered low variance & we cannot drop the column<br>Missing values must belong to 3rd category *NA - No alley access*<br>Let's impute 'NA' for missing values """ # impute 'NA' for null values in Alley train["Alley"].fillna('NA', inplace=True) # impute null values for Alley in test data test["Alley"].fillna('NA', inplace=True) """ Next feature with missing values is **Fence**, 1179 values are missing, let's analyse:<br> Fence: Fence quality GdPrv Good Privacy MnPrv Minimum Privacy GdWo Good Wood MnWw Minimum Wood/Wire NA No Fence """ # getting counts of unique values in Fence train["Fence"].value_counts() """ By analysing at the count values in categories, the 1179 missing values beong to the missing category *NA - No Fence*, let's impute 'NA' """ # impute 'NA' for null values in Fence train["Fence"].fillna('NA', inplace=True) # impute null values for Fence in test data test["Fence"].fillna('NA', inplace=True) """ **FireplaceQu** has 690 missing values<br> FireplaceQu: Fireplace quality Ex Excellent - Exceptional Masonry Fireplace Gd Good - Masonry Fireplace in main level TA Average - Prefabricated Fireplace in main living area or Masonry Fireplace in basement Fa Fair - Prefabricated Fireplace in basement Po Poor - Ben Franklin Stove NA No Fireplace """ # getting counts of unique values in FireplaceQu train["FireplaceQu"].value_counts() """ Missing values may be *NA-No fireplce*. we have another feature __Fireplaces__ that shows the number of fireplaces in the house, let's analyse this feature """ # getting counts of unique values in Fireplaces train["Fireplaces"].value_counts() """ There are 690 counts of zero fireplaces, that matches with our 690 missing values in *FirelaceQu*, so impute them as 'NA - No Fireplace' """ # impute 'NA' for missing values in FireplaceQu train["FireplaceQu"].fillna("NA", inplace=True) # impute for missing values in FireplaceQu in test data test["FireplaceQu"].fillna("NA", inplace=True) """ **GarageType, GarageFinish, GarageQual, GarageCond** all describing the properties of Garage, have 81 values missing<br>The 81 missing values are from **same rows** (we confirmed this when analysing the **GarageYrBlt** column) """ """ GarageType: Garage location 2Types More than one type of garage Attchd Attached to home Basment Basement Garage BuiltIn Built-In (Garage part of house - typically has room above garage) CarPort Car Port Detchd Detached from home NA No Garage GarageFinish: Interior finish of the garage Fin Finished RFn Rough Finished Unf Unfinished NA No Garage GarageQual: Garage quality Ex Excellent Gd Good TA Typical/Average Fa Fair Po Poor NA No Garage GarageCond: Garage condition Ex Excellent Gd Good TA Typical/Average Fa Fair Po Poor NA No Garage """ """ All these features have a category 'NA' for 'No garage', the missing 81 values belong to 'NA'<br>Impute missing values as 'NA' """ # imputing 'NA' for missing values train["GarageType"].fillna('NA', inplace=True) train["GarageFinish"].fillna('NA', inplace=True) train["GarageQual"].fillna('NA', inplace=True) train["GarageCond"].fillna('NA', inplace=True) # imputing 'NA' for missing values in test data test["GarageType"].fillna('NA', inplace=True) test["GarageFinish"].fillna('NA', inplace=True) test["GarageQual"].fillna('NA', inplace=True) test["GarageCond"].fillna('NA', inplace=True) """ __BsmtExposure, BsmtFinType2__ both features have 38 missing values, let's see if the missing values are from the same rows<br> BsmtExposure: Refers to walkout or garden level walls Gd Good Exposure Av Average Exposure (split levels or foyers typically score average or above) Mn Mimimum Exposure No No Exposure NA No Basement BsmtFinType2: Rating of basement finished area (if multiple types) GLQ Good Living Quarters ALQ Average Living Quarters BLQ Below Average Living Quarters Rec Average Rec Room LwQ Low Quality Unf Unfinshed NA No Basement """ # getting uniuqe value counts for BsmtExposure train["BsmtExposure"].value_counts() # getting uniuq value counts for BsmtFinType2 train["BsmtFinType2"].value_counts() train.loc[train["BsmtExposure"].isnull()].head() """ The missing 38 values are from **same rows** and category *NA - No Basement* is missing from both columns<br> Impute the missing values as 'NA' """ # filling 'NA' for missing values train["BsmtExposure"].fillna('NA', inplace=True) train["BsmtFinType2"].fillna('NA', inplace=True) # filling 'NA' for missing values in test data test["BsmtExposure"].fillna('NA', inplace=True) test["BsmtFinType2"].fillna('NA', inplace=True) """ __BsmtCond, BsmtFinType1, BsmtQual__ features have 37 missing values, let's see if the missing values are from the same rows<br> BsmtQual: Evaluates the height of the basement Ex Excellent (100+ inches) Gd Good (90-99 inches) TA Typical (80-89 inches) Fa Fair (70-79 inches) Po Poor (<70 inches NA No Basement BsmtCond: Evaluates the general condition of the basement Ex Excellent Gd Good TA Typical - slight dampness allowed Fa Fair - dampness or some cracking or settling Po Poor - Severe cracking, settling, or wetness NA No Basement BsmtFinType1: Rating of basement finished area GLQ Good Living Quarters ALQ Average Living Quarters BLQ Below Average Living Quarters Rec Average Rec Room LwQ Low Quality Unf Unfinshed NA No Basement """ # unique value counts for BsmtQual train["BsmtQual"].value_counts() # unique value counts for BsmtCond train["BsmtCond"].value_counts() # unique value counts for BsmtFinType1 train["BsmtFinType1"].value_counts() train.loc[train["BsmtQual"].isnull()].head() """ The missing 37 values are *NA-No Basement*, they are from same rows as 'NA' is missing from above 3 features """ # impute 'NA' for null values train["BsmtQual"].fillna('NA', inplace=True) train["BsmtCond"].fillna('NA', inplace=True) train["BsmtFinType1"].fillna('NA', inplace=True) # impute 'NA' for null values in test data test["BsmtQual"].fillna('NA', inplace=True) test["BsmtCond"].fillna('NA', inplace=True) test["BsmtFinType1"].fillna('NA', inplace=True) """ Checking for missing values in train again """ # null values in train train.isnull().sum().sort_values(ascending=False).head(5) """ 3 more Categorical features having missing values, let's analyse __MasVnrType & MasVnrArea__ together since they both represent the same property & both have 8 missing values MasVnrType: Masonry veneer type BrkCmn Brick Common BrkFace Brick Face CBlock Cinder Block None None Stone Stone MasVnrArea: Masonry veneer area in square feet """ # getting unique value count for MasVnrArea train["MasVnrArea"].value_counts() # getting unique value count for MasVnrType train["MasVnrType"].value_counts() """ In *MasVnrType*, "CBlock(Cinder Block)" category is missing, however without statistics confirmation, we cannot impute 'CBlock' for the 8 missing values, so we impute most frequent values in both columns: - Impute **0** for missing values in *MasVnrArea* - Impute **mode**('None' category) for *MasVnrType* """ # impute most frequent values for MasVnrArea & MasVnrType train["MasVnrArea"].fillna(0, inplace=True) train["MasVnrType"].fillna(train["MasVnrType"].mode()[0], inplace=True) # impute most frequent values for MasVnrArea & MasVnrType in test data test["MasVnrArea"].fillna(0, inplace=True) test["MasVnrType"].fillna(train["MasVnrType"].mode()[0], inplace=True) """ Last feature with missing value is __Electrical__, with 1 missing value Electrical: Electrical system SBrkr Standard Circuit Breakers & Romex FuseA Fuse Box over 60 AMP and all Romex wiring (Average) FuseF 60 AMP Fuse Box and mostly Romex wiring (Fair) FuseP 60 AMP Fuse Box and mostly knob & tube wiring (poor) Mix Mixed """ # getting unique value counts for Electrical train["Electrical"].value_counts() """ Let's impute the most frequent category for the missing value """ # impute most frequent category for null value train["Electrical"].fillna(train["Electrical"].mode()[0], inplace=True) # impute most frequent category for null value in test data test["Electrical"].fillna(train["Electrical"].mode()[0], inplace=True) """ We have successfully imputed missing values in numerical & categorical features! **No null values** in the train dataset now<br>Along the way we analysed the related features and removed 3 redundant features!! """ # check for null values in train sns.heatmap(train.isnull(), cbar=False) # checking null values in test data test.isnull().sum() """ **Test data has null values in more columns** than train data, we have to clean the test data by **imputing mean** for numerical columns and **mode** for categorical columns. We cannot drop the columns with missing values, since it will lead to shape mismatch later during the model training stage """ # impute mean for numerical columns and mode for categorical columns in test data test["MSZoning"].fillna(test["MSZoning"].mode()[0], inplace=True) test["Exterior1st"].fillna(test["Exterior1st"].mode()[0], inplace=True) test["Exterior2nd"].fillna(test["Exterior2nd"].mode()[0], inplace=True) test["BsmtFinSF1"].fillna(test["BsmtFinSF1"].mean(), inplace=True) test["TotalBsmtSF"].fillna(test["TotalBsmtSF"].mean(), inplace=True) test["KitchenQual"].fillna(test["KitchenQual"].mode()[0], inplace=True) test["Functional"].fillna(test["Functional"].mode()[0], inplace=True) test["GarageCars"].fillna(test["GarageCars"].mean(), inplace=True) test["SaleType"].fillna(test["SaleType"].mode()[0], inplace=True) # check the shape of train and test after data cleaning print(train.shape) print(test.shape) """ #### [Go to Index](#Index) """ """ *** """ """ ## 3. Data analysis & Feature selection - Numerical """ """ Now that we took care of all missing values in the train data, let's first analyse Numerical features by selecting relevant features & dropping redundant features using statistics.<br> **Correlation matrix & heatmap** can be used to explore numerical features and their correlation values """ # numerical features in train train.select_dtypes(exclude='object') """ ## 3.1 Pairwise Correlation """ """ Exploring the relationships between features is crucial in data analysis in order to select the best set of features. A pairplot helps to analyse each numerical feature in relation to every other numerical feature in a dataset<br> Since we have a lot of numerical features in train data, a single pairplot with all numerical columns becomes complex and takes a long time to execute. Let's plot 3 separate pairplots for better visualisation """ sns.pairplot(train, diag_kind='kde', vars = ["SalePrice", 'LotFrontage', 'LotArea', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF'], corner=True) """ The diagonal histograms in the pairplot show the distributions of individual columns along the axes and each scatter plot shows the relationship between columns along the x and y axes. For example, - We can see strong positive linear relationship between columns **SalePrice and 1stFlrSF(1st floor area)** where the plot is a upward diagonal. - But we can infer that **LotArea and YrBuilt** don't have a linear relationship. - Pairplot can tell us about very few datapoints that are far away from the clusters, they are outliers that have extreme values deviating from the normal distribution. - There can be **univariate**(that deviate from other datapoints in an individual feature) and **bivariate outliers**(that have abnormal readings/values when 2 features are taken in account)<br> Similar insights can be derived from the below pairplots too """ sns.pairplot(train, diag_kind='kde', vars = ["SalePrice", 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd', 'Fireplaces'], corner=True) """ **GrLivArea, TotRmsAbvGrd and BedroomAbvGr** have strong positive correlation with SalePrice """ sns.pairplot(train, diag_kind='kde', vars = ["SalePrice", 'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'MiscVal', 'MoSold', 'YrSold'], corner=True) """ **GarageArea** will be an important feature in determining *SalePrice* """ """ So far we have visually analysed the features and their relationships. To make better decisions on selecting the best features, let's look at the statistics """ """ ## 3.2 Correlation matrix & Heatmap """ """ Correlation values are statistical readings of relationships among numerical features<br> Correlation matrix gives the relationship between each variable with respect to every other variable in the dataset, the correlation can be: - **positive** - when a variable responds to change in other variable in a positive way - **negative** - when a variable responds to change in other variable in a negative way - **zero** - no relationship between the 2 variables """ # correlation matrix of train data train.corr() """ A **Correlation Matrix** gives the statistical relationship between every column to every other other column in a dataframe. The diagonal values are 1 as they are correlation values of column with itself. We can also see how each column is related to target. The correlation matrix can be **visualised using a heatmap** """ # visualising correaltion using a heatmap plt.figure(figsize=(12,12)) sns.heatmap(train.corr(), cmap="rocket") """ In the above heatmap, lighter the hue, more the correlation & darker the hue, lesser the correlation between variables along X & Y axis<br>We can see the highly correlated features with target (*SalePrice* in Y-axis bottom)as lighter colors in the bottom of the plot """ """ Below features have high correlation with target as observed from the heatmap: * OverallQual * GeLivArea * GarageCArs * GArageArea * TotalBcmtSF * 1stFlrSF * YearBuilt * YearRemodAdd etc. """ """ Let's look at the statistics of correlation value of each variable vs target<br> Calculate the **correlation of every numerical feature against SalePrice** and sort them from max to min values """ # calculating correaltion with SalePrice from max to min num_cols_corr = train.corrwith(train["SalePrice"]).sort_values(ascending=False) num_cols_corr """ - A value of 1 is correlation with itself, so *SalePrice* has 1 - We have values starting from 0.79 for *OverallQual* to 0.04 for *3SsnPorch* and then we have negative correaltion for some features - Correlation **above 0.3 can be taken as high positive correlation and below -0.3 as high negative correlation** - Negatively correlation also affects target prediction, so consider both +ve, -ve highly correlated features - Also, if the correlation between 2 features are **similar**, we can drop one of the features as similarly correlated features will add to the complexity of the model and don't necessarily improve the performance """ """ In Train data, we have correlation from 0.74 to 0.04 in positive value and from -0.01 to -0.13 in negative value. Any value above 0.3 can be considered as strong correlation in positive and negative side. We don't have strong negative correlation here, so let's consider only positive correlation values<br> Let's **retain the features with threshold > 0.3** and drop features with threshold < 0.3 """ # filtering columns with < 0.3 correlation with target numcols_to_drop = num_cols_corr[num_cols_corr < 0.3].index numcols_to_drop # drop columns with weak correlation from train data train.drop(numcols_to_drop, axis=1, inplace=True) # drop columns with weak correlation from test data test.drop(numcols_to_drop, axis=1, inplace=True) """ ## 3.3. Feature Analysis based on Correlation """ """ Let's explore the features based on their correlation values<br> #### 3.3.1 Analysing features having high correlation(>0.3) with target - GrLivArea: Above grade (ground) living area square feet - 1stFlrSF: First Floor square feet - 2ndFlrSF: Second floor square feet - __GrLivArea__ has 0.708624 correlation with *SalePrice*, when we analyse this feature in the dataframe, this shows the *living area above ground* and this column is the sum of __1stFlrSF & 2ndFlrSF__, so we can simplify our training data by dropping the 1stFlrSF & 2ndFlrSF which are already included in *GrLivArea* """ train.head() # drop 1stFlrSF, 2ndFlrSF from train train.drop(columns=["1stFlrSF", "2ndFlrSF"], axis=1, inplace=True) # drop 1stFlrSF, 2ndFlrSF from test data test.drop(columns=["1stFlrSF", "2ndFlrSF"], axis=1, inplace=True) """ #### 3.3.2 Analysing features with similar correlation values - GarageCars: Size of garage in car capacity - GarageArea: Size of garage in square feet - __GarageArea & GarageCars__ both explain Garage size in different metrics, so we can drop *GarageArea* & retain *GarageCars* which has high correlation with *Saleprice* compared to *GarageArea* """ # drop GarageArea from train train.drop(columns=["GarageArea"], axis=1, inplace=True) # drop GarageArea from test data test.drop(columns=["GarageArea"], axis=1, inplace=True) """ #### 3.3.3 Irrelevant Features - Since we dropped columns having very low correlation, the features that are related to these dropped columns would become irrelevant or have no contribution to the training data, we can identify those features and remove them form train data """ # columns dropped earlier due to low correlation with SalePrice numcols_to_drop """ The feature *BsmtFinSF2*(Type 2 finished square feet) has been dropped now, so the feature explaining about the rating of BsmtFinSF2 will become redundant, so let's drop the feature *BsmtFinType2* """ """ BsmtFinSF2: Type 2 finished square feet BsmtFinType2: Rating of basement finished area (if multiple types) GLQ Good Living Quarters ALQ Average Living Quarters BLQ Below Average Living Quarters Rec Average Rec Room LwQ Low Quality Unf Unfinshed NA No Basement """ # drop BsmtFinType2 from train train.drop(columns=["BsmtFinType2"], axis=1, inplace=True) # drop BsmtFinType2 from test data test.drop(columns=["BsmtFinType2"], axis=1, inplace=True) """ #### [Go to Index](#Index) """ """ *** """ """ ## 4. Data analysis & Feature selection - Categorical """ """ Until now, we have analysed only the numeric features and their correlation using heatmap & correlation matrix. The dataset has numerous categorical features. Let's analyse the categorical data, select relevant features & drop redundant features using statistics """ # categorical features in train train.select_dtypes(include='object') """ Let's plot all the categorical variables against our target *SalePrice* and analyse their relationship. The **variance** on the *SalePrice* for each unique category of a cloumn is shown as a box plot. The plots will give a lot of insights into the data. This will be our initial step in deciding which variables are important in predicting the target and which variables would not be crucial """ # plotting categorical columns vs Saleprice fig, ax = plt.subplots(11,4, figsize=(30, 70)) for col, subplot in zip(train.select_dtypes(include='object'), ax.flatten()): sns.boxplot(x=train[col], y="SalePrice", data=train, ax=subplot) for label in subplot.get_xticklabels(): label.set_rotation(90) """ Some of the insights from the above plots: - **Utilities** column has only 1 category *AllPub* affecting the *SalePrice*, the entire column could be redundant - **Exterior1st & Exterior2nd** columns have very similar correlation with *SalePrice* for almost all categories, they may be duplicates - Mean value of *SalePrice* for all categories in **Functional** column is almost same """ """ To confirm the above points, let's do a thorough statistical exploration of the columns before acting upon the features<br> What to lookout for in categorical columns? - As we go through the dataframe, observe the values for each column. If we find suspicious distribution of values, analyse them statistically - We can observe more than 1 feature explaining same/similar properties of the house, we could analyse and drop one of them - If a column has same category mostly, analyse the variance of the column. Drop if the column has a very low variance """ """ #### - Street appears to have same category *Paved* for most of the datapoints, let's verify it's variance by analysing the unique category counts Street: Type of road access to property Grvl Gravel Pave Paved """ # unique value count of categories in Street train["Street"].value_counts() """ 99.6% of rows have *paved* category, only 0.4% of total data has *Grvl* category, this column has **very low variance** and will not help in predicting the target, we can drop this feature """ # drop Street column from train train.drop(columns=["Street"], axis=1, inplace=True) # drop Street column from test data test.drop(columns=["Street"], axis=1, inplace=True) """ #### - Similar/identical categorical features *LandContour & LandSlope* **both explain flatness** of the property in 2 different ways """ """ LandContour: Flatness of the property Lvl Near Flat/Level Bnk Banked - Quick and significant rise from street grade to building HLS Hillside - Significant slope from side to side Low Depression LandSlope: Slope of property Gtl Gentle slope Mod Moderate Slope Sev Severe Slope """ # LandContour, LandSlope columns in train train[["LandContour", "LandSlope"]].head(20) # getting unique value counts of LandContour, LandSlope print(train["LandContour"].value_counts()) print(train["LandSlope"].value_counts()) # plotting LandContour, LandSlope vs SalePrice fig, axes = plt.subplots(1, 2, figsize=(12, 4)) sns.histplot(ax=axes[0], x="LandContour", y="SalePrice", data=train) sns.histplot(ax=axes[1], x="LandSlope", y="SalePrice", data=train) """ *LandSlope* has simplified categories compared to *LandContour*, so let's drop *LandContour* """ # drop LandContour from train train.drop(columns=["LandContour"], axis=1, inplace=True) train.shape # drop LandContour from test data test.drop(columns=["LandContour"], axis=1, inplace=True) """ #### - Utilities appears to have same category value for many datapoints, let's check it's variance Utilities: Type of utilities available AllPub All public Utilities (E,G,W,& S) NoSewr Electricity, Gas, and Water (Septic Tank) NoSeWa Electricity and Gas Only ELO Electricity only """ # getting value count of categories in utilities train["Utilities"].value_counts() """ Clearly *Utilities* won't help to improve the performance of the model, it has **very low variance** in data, since all the houses fall into 1 category(well, except for only 1), so drop *Utilities* feature """ # drop Utilities column from train train.drop(columns=["Utilities"], axis=1, inplace=True) # drop Utilities column from test test.drop(columns=["Utilities"], axis=1, inplace=True) """ #### - Other 2 similar features that explain proximity are: Condition1: Proximity to various conditions Artery Adjacent to arterial street Feedr Adjacent to feeder street Norm Normal RRNn Within 200' of North-South Railroad RRAn Adjacent to North-South Railroad PosN Near positive off-site feature--park, greenbelt, etc. PosA Adjacent to postive off-site feature RRNe Within 200' of East-West Railroad RRAe Adjacent to East-West Railroad Condition2: Proximity to various conditions (if more than one is present) Artery Adjacent to arterial street Feedr Adjacent to feeder street Norm Normal RRNn Within 200' of North-South Railroad RRAn Adjacent to North-South Railroad PosN Near positive off-site feature--park, greenbelt, etc. PosA Adjacent to postive off-site feature RRNe Within 200' of East-West Railroad RRAe Adjacent to East-West Railroad """ # count of unique values in Condition1, Condition2 print(train["Condition1"].value_counts()) print(train["Condition2"].value_counts()) """ Drop **Condition2** since almost all(1445) houses have 'Norm' category and this column has low variance """ # drop Condition2 from train train.drop(columns=["Condition2"], axis=1, inplace=True) # drop Condition2 from test test.drop(columns=["Condition2"], axis=1, inplace=True) """ #### - BldgType & HouseStyle are entirely different features, as we observe from the descriptions and datapoints, so they don't need any action BldgType: Type of dwelling 1Fam Single-family Detached 2FmCon Two-family Conversion; originally built as one-family dwelling Duplx Duplex TwnhsE Townhouse End Unit TwnhsI Townhouse Inside Unit HouseStyle: Style of dwelling 1Story One story 1.5Fin One and one-half story: 2nd level finished 1.5Unf One and one-half story: 2nd level unfinished 2Story Two story 2.5Fin Two and one-half story: 2nd level finished 2.5Unf Two and one-half story: 2nd level unfinished SFoyer Split Foyer SLvl Split Level """ """ #### - OverallQual & OverallCond are different properties. On a visual inspection of the data, the ratings are different in the 2 columns for each datapoint. So no action is needed OverallQual: Rates the overall material and finish of the house 10 Very Excellent 9 Excellent 8 Very Good 7 Good 6 Above Average 5 Average 4 Below Average 3 Fair 2 Poor 1 Very Poor OverallCond: Rates the overall condition of the house 10 Very Excellent 9 Excellent 8 Very Good 7 Good 6 Above Average 5 Average 4 Below Average 3 Fair 2 Poor 1 Very Poor """ """ #### - Let's analyse RoofStyle & RoofMatl both explaining about roof structure, they seem to have same values when we observe in the dataframe: RoofStyle: Type of roof Flat Flat Gable Gable Gambrel Gabrel (Barn) Hip Hip Mansard Mansard Shed Shed RoofMatl: Roof material ClyTile Clay or Tile CompShg Standard (Composite) Shingle Membran Membrane Metal Metal Roll Roll Tar&Grv Gravel & Tar WdShake Wood Shakes WdShngl Wood Shingles """ """ Both clearly explain different properties, let's observe the category counts """ # plot the category count for RoofStyle, RoofMatl fig, axes = plt.subplots(1, 2, figsize=(12,4)) sns.countplot(ax=axes[0], x="RoofStyle", data=train) sns.countplot(ax=axes[1], x="RoofMatl", data=train) ax=plt.xticks(rotation=90) print(train["RoofStyle"].value_counts()) print(train["RoofMatl"].value_counts()) """ In *RoofMatl* 98% of datapoints fall under 1 category *CompShg*, so this feature won't be of importance in prediciting the target<br>Since it's **variance is less**, let's drop RoofMatl """ # drop RoofMatl from train train.drop(columns=["RoofMatl"], axis=1, inplace=True) # drop RoofMatl from test test.drop(columns=["RoofMatl"], axis=1, inplace=True) """ #### 5.5 Exterior2nd is the exterior covering of house, if more than 1 material is used other than in Exterior1st, let's analyse them Exterior1st: Exterior covering on house AsbShng Asbestos Shingles AsphShn Asphalt Shingles BrkComm Brick Common BrkFace Brick Face CBlock Cinder Block CemntBd Cement Board HdBoard Hard Board ImStucc Imitation Stucco MetalSd Metal Siding Other Other Plywood Plywood PreCast PreCast Stone Stone Stucco Stucco VinylSd Vinyl Siding Wd Sdng Wood Siding WdShing Wood Shingles Exterior2nd: Exterior covering on house (if more than one material) AsbShng Asbestos Shingles AsphShn Asphalt Shingles BrkComm Brick Common BrkFace Brick Face CBlock Cinder Block CemntBd Cement Board HdBoard Hard Board ImStucc Imitation Stucco MetalSd Metal Siding Other Other Plywood Plywood PreCast PreCast Stone Stone Stucco Stucco VinylSd Vinyl Siding Wd Sdng Wood Siding WdShing Wood Shingles """ # getting value counts in categories of Exterior1st, Exterior2nd print(train["Exterior1st"].value_counts()) print(train["Exterior2nd"].value_counts()) # observe the 2 features in dataframe train[["Exterior1st", "Exterior2nd"]].head(20) # comparing the values for all datapoints across the entire dataframe compare = np.where(train["Exterior2nd"] == train["Exterior1st"], True, False) comparison = np.array(compare) np.unique(comparison, return_counts=True) # plotting Exterior1st, Exterior2nd vs SalePrice fig, axes = plt.subplots(1, 2, figsize=(24, 4)) sns.boxplot(ax=axes[0], x="Exterior1st", y="SalePrice", data=train) sns.boxplot(ax=axes[1], x="Exterior2nd", y="SalePrice", data=train) """ Though visually it seems both columns have similar categories and similar correlation with target, 15% of values differ for both columns, only 85% have the same material used for both exterior covering of the house<br>So we cannot drop *Exterior2nd*</mark> """ """ #### 5.6 ExterQual & ExterCond seem to explain about the quality and condition of the exterior, are they same? ExterQual: Evaluates the quality of the material on the exterior Ex Excellent Gd Good TA Average/Typical Fa Fair Po Poor ExterCond: Evaluates the present condition of the material on the exterior Ex Excellent Gd Good TA Average/Typical Fa Fair Po Poor """ # category counts in ExterQual, ExterCond print(train["ExterQual"].value_counts()) print(train["ExterCond"].value_counts()) fig, axes = plt.subplots(1, 2, figsize=(14,6)) sns.boxplot(ax=axes[0], x="ExterQual", y="SalePrice", data=train) sns.boxplot(ax=axes[1], x="ExterCond", y="SalePrice", data=train) """ Both are entirely different, *ExterQual* reflects the quality of exterior whereas *ExterCond* reflects the present condition of the exterior irrespective of quality, let's retain these features for training """ """ #### - BsmtQual & BsmtCond explain the height and general condition of the basement respectively BsmtQual: Evaluates the height of the basement Ex Excellent (100+ inches) Gd Good (90-99 inches) TA Typical (80-89 inches) Fa Fair (70-79 inches) Po Poor (<70 inches NA No Basement BsmtCond: Evaluates the general condition of the basement Ex Excellent Gd Good TA Typical - slight dampness allowed Fa Fair - dampness or some cracking or settling Po Poor - Severe cracking, settling, or wetness NA No Basement """ # getting value count for categories in BsmtQual, BsmtCond print(train["BsmtQual"].value_counts()) print(train["BsmtCond"].value_counts()) """ The category count is significant, we cannot drop any of the feature here, let's retain both """ """ #### - Heating column seems to have same category for most datapoint, let's analyse Heating: Type of heating Floor Floor Furnace GasA Gas forced warm air furnace GasW Gas hot water or steam heat Grav Gravity furnace OthW Hot water or steam heat other than gas Wall Wall furnace """ train["Heating"].value_counts() """ 98% of houses in the data use same type of heating(GasA), so it's **variance negligible**, drop the *Heating* feature """ # drop Heating from train train.drop(columns=["Heating"], axis=1, inplace=True) # drop Heating from test test.drop(columns=["Heating"], axis=1, inplace=True) """ #### - The related column *Heating QC: Heating quality and condition* - even if the type of heating is same throughout, the quality of heating may vary and it may be correlated with target. Let's analyse HeatingQC: Heating quality and condition Ex Excellent Gd Good TA Average/Typical Fa Fair Po Poor """ # category count for HeatingQC train["HeatingQC"].value_counts() # plot HeatingQC vs SalePrice plt.figure(figsize=(8,4)) sns.boxplot(x="HeatingQC", y="SalePrice", data=train) """ We have all 5 categories of heating quality from Excellent to poor, as shown in the plot greatly correlates with the target. So this feature is important """ """ #### - When going through the *CentralAir*, it looks like, it has the same category *Y* throughout. If the category is same for all the datapoints, then we can drop the column. Let's analyse for counts CentralAir: Central air conditioning N No Y Yes """ # getting value count for CentralAir train["CentralAir"].value_counts() """ We have 95(6.5% of total data) houses without central air conditioning. The variance is high enough to retain this feature in our train data """ """ #### - GarageQual & GarageCond on visual inspection have same categories, let's explore them together GarageQual: Garage quality Ex Excellent Gd Good TA Typical/Average Fa Fair Po Poor NA No Garage GarageCond: Garage condition Ex Excellent Gd Good TA Typical/Average Fa Fair Po Poor NA No Garage """ print(train["GarageQual"].value_counts()) print(train["GarageCond"].value_counts()) fig, axes = plt.subplots(1, 2, figsize=(14,6)) sns.stripplot(ax=axes[0], x="GarageQual", y="SalePrice", data=train) sns.stripplot(ax=axes[1], x="GarageCond", y="SalePrice", data=train) """ *GarageQual* represents the Garage quality but *GarageCond* represents the condition of Garage. Except for few datapoints, 2 **columns are very similar in the category count and their correlation with *SalePrice***, let's drop *GarageCond* """ # drop GarageCond from train train.drop(columns=["GarageCond"], axis=1, inplace=True) # drop GarageCond from test test.drop(columns=["GarageCond"], axis=1, inplace=True) """ #### - PavedDrive feature has same category *'Y'* in visual check, let's analyse the variance PavedDrive: Paved driveway Y Paved P Partial Pavement N Dirt/Gravel """ train["PavedDrive"].value_counts() """ Only 91% of data have category *'Y'*, so the column cannot be dropped """ """ #### - Finally let's analyse *SaleType* or *SaleCondition* features SaleType: Type of sale WD Warranty Deed - Conventional CWD Warranty Deed - Cash VWD Warranty Deed - VA Loan New Home just constructed and sold COD Court Officer Deed/Estate Con Contract 15% Down payment regular terms ConLw Contract Low Down payment and low interest ConLI Contract Low Interest ConLD Contract Low Down Oth Other SaleCondition: Condition of sale Normal Normal Sale Abnorml Abnormal Sale - trade, foreclosure, short sale AdjLand Adjoining Land Purchase Alloca Allocation - two linked properties with separate deeds, typically condo with a garage unit Family Sale between family members Partial Home was not completed when last assessed (associated with New Homes) """ train["SaleType"].value_counts() train["SaleCondition"].value_counts() """ *SaleType or SaleCondition* columns have no abnormality in their category values, so no change is needed here """ # check the shape of train and test after feature selection print(train.shape) print(test.shape) """ We have analysed all the features(both Numerical & Categorical) and based on the statistics, we dropped some features that are redundant, duplicate and those don't contribute in improving the performance of the model. The number of features have been **reduced from 80 to 50**, a **simplified model** will help in better prediciton of target """ """ #### [Go to Index](#Index) """ """ *** """ """ ## 5. Skewness and Normalisation """ """ As we analysed the data, the distribution plots in the pairplot show that target variable *SalePrice* and some features are **highly skewed**. Data should follow normal distribution for the model to successfully predict target, because regression based models assume **normally distributed data for good predcition results** """ """ #### Skewness refers to the asymmetrical/distorted distribution of data along the mean. A normal distribution resembles a bell curve with data equally distributed on either side of the mean value of the data. Skewed data has most of the values either to the right or left of the mean. Log Transformation is one of the methods used to approximate skewed data to normal distrbution """ # skewness of target, Saleprice plt.figure(figsize=(8,6)) sns.histplot(train["SalePrice"]) # skewness of numerical columns in train data train.hist(bins=20, figsize=(20, 10)) plt.show() """ As we can see here, most of the columns have skewed to the right side. **Right skewed data** model can be good in predicting only lower value ranges but predcition will be poor in higher value ranges due to less data in higher value range. The data has to be normalised before training it on an algorithm """ # calculate skewness of numerical columns and sort values from maximun to minimum skew = (train.select_dtypes(exclude='object').skew(axis=0).sort_values(ascending=False)) skew """ **Skewness value** of < -1 and > 1 is considered that the data is highly skewed<br> Features with skewness >1 are positively skewed and features with skewness < -1 are negatively skewed. we don't have highly negatively skewed data in train dataset<br>**Filter** features having highly positively skewed data, **skewness > 1**<br>Drop *SalePrice* from the list, we'll deal with target skew later """ # columns in train data with skewness > 1 skew_train = skew[skew > 1].index # only skewed features skew_cols = skew_train.drop(["SalePrice"]) # columns to be normalised skew_cols # applying log transformation to features in train data for cols in skew_cols: train[cols] = np.log1p(train[cols]) train # applying log transformation to features in test data for cols in skew_cols: test[cols] = np.log1p(test[cols]) # applying log transformation to target train["SalePrice"] = np.log1p(train["SalePrice"]) train["SalePrice"] # visualising normalised train data train.hist(bins=20, figsize=(20, 10)) plt.show() """ After log transformation, the data has been approximated to a normal distribution """ # visualising normalised target sns.histplot(train["SalePrice"]) """ #### [Go to Index](#Index) """ """ *** """ """ ## 6. Training set-Validation set split, Pre-processing steps: OneHotEncoding, Standardisation """ """ Separate the target *SalePrice* from the train data, the **features** are stored in the variable **X** and the **target** in **y** """ X = train.drop(["SalePrice"], axis=1) y = train["SalePrice"] print("Shape of X:", X.shape) print("Shape of y:", y.shape) # filter the categorical columns from X cat_cols = X.select_dtypes(include="object").columns cat_cols """ The model should be trained on a set of data and **validated on another new set of data**. Here we split the features and target in the ratio of 70% training data and 30% validation data. A validation dataset is needed to validate the results of a trained model. We will train a model using X_train, y_train and use our model on X_val to predict y values """ # train test split - split X, y into X_train, X_val, y_train, y_val X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.30, random_state=42) print("Shape of X_train:", X_train.shape) print("Shape of y_train:", y_train.shape) print("Shape of X_val:", X_val.shape) print("Shape of y_val:", y_val.shape) """ Machine learning algorithms can only take numerical inputs, so we need to **encode the categories** in categorical columns into numerical values<br> **One hot encoding** is one of the methods to encode categories. We will build a pipeline below using **ColumnTransformer** to do this task, where it will encode the categorical columns but will 'passthrough' the numerical columns without any change """ # define column transformer pipeline categorical_transformer = Pipeline(steps=[("onehot", OneHotEncoder(handle_unknown="ignore"))]) col_transform = ColumnTransformer( transformers=[("categorical", categorical_transformer,cat_cols)], remainder='passthrough' ) # fit-transform column transformer on the training set X_train and based on the parameters, transform the validation set X_val X_train_encoded = col_transform.fit_transform(X_train) X_val_encoded = col_transform.transform(X_val) print("Shape of X_train after encoding:", X_train_encoded.shape) print("Shape of X_val after encoding:", X_val_encoded.shape) # apply column transformer and transform the test data test_encoded = col_transform.transform(test) print("Shape of test data after encoding:", test_encoded.shape) """ One hot encoding method will increase the number of columns since it encodes and transforms each category in a column into a new column. It also change the names of the categorical features into numericals, we need our **original column names** for evaluating our model, so get the original feature names form encoded data. after encoding, the column names have their respective category """ # this code will get the original feature names along with their categories onehot_cols = col_transform.named_transformers_["categorical"].named_steps["onehot"].get_feature_names(cat_cols) """ We have extracted only the categorical column names after encoding, to get all the column names from the training data, append the numerical columns to the list """ # append the numerical columns with the encoded columns original_feature_names = list(onehot_cols) + list(train.select_dtypes(exclude="object").columns.drop(["SalePrice"])) original_feature_names """ The data on which our model is to be trained has all the different ranges of values and different metrics in data such as area in square feet, years, ratings from 1 to 10 etc., to get a uniformity in values, we need to **scale the data**. **StandardScaler** is one method of standardisation of data """ # fit-transform StandardScale on the training set X_train and based on the parameters, transform the validation set X_val scaler = StandardScaler(with_mean=False) X_train_scaled = scaler.fit_transform(X_train_encoded) X_val_scaled = scaler.transform(X_val_encoded) # apply StandardScaler and standardise the test data test_scaled = scaler.transform(test_encoded) """ #### [Go to Index](#Index) """ """ *** """ """ ## 7. Extreme Gradient Boosting(XGB) Regressor """ """ We have completed the pre-processing of training and validation datasets. Now, we are ready to train the data on our model.<br>We are training the data on an XGBoost model. **Extreme Gradient(XG) Boosting** is an improvisation of an **ensemble algorithm** called Gradient Boosting and is known for great accuracy and speed """ """ Here is a complete reference to XGBoost Classification & Regression models (https://medium.com/sfu-cspmp/xgboost-a-deep-dive-into-boosting-f06c9c41349) """ #