"""
---
# Boston Housing Prices Prediction: Linear Regression
---
"""
"""
---
## Objective
---

The problem on hand is to predict the housing prices of a town or a suburb based on the features of the locality provided to us. In the process, we need to identify the most important features in the dataset. We need to employ techniques of data preprocessing and build a linear regression model that predicts the prices for us. 
"""
"""
----------------------------
## Data Information
---------------------------

Each record in the database describes a Boston suburb or town. The data was drawn from the Boston Standard Metropolitan Statistical Area (SMSA) in 1970. Detailed attribute information can be found below-

Attribute Information (in order):
- **CRIM:**     per capita crime rate by town
- **ZN:**       proportion of residential land zoned for lots over 25,000 sq.ft.
- **INDUS:**    proportion of non-retail business acres per town
- **CHAS:**     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
- **NOX:**      nitric oxides concentration (parts per 10 million)
- **RM:**       average number of rooms per dwelling
- **AGE:**     proportion of owner-occupied units built before 1940
- **DIS:**      weighted distances to five Boston employment centers
- **RAD:**      index of accessibility to radial highways
- **TAX:**      full-value property-tax rate per 10,000 dollars
- **PTRATIO:**  pupil-teacher ratio by town
- **B - 1000(Bk - 0.63)^2** where Bk is the proportion of blacks by town
- **LSTAT:**    %lower status of the population
- **MEDV:**     Median value of owner-occupied homes in 1000 dollars
"""
"""
## Importing the required libraries
"""
# import libraries for data manipulation
import pandas as pd
import numpy as np

# import libraries for data visualization
import matplotlib.pyplot as plt
import seaborn as sns
from statsmodels.graphics.gofplots import ProbPlot

# import libraries for building linear regression model
from statsmodels.formula.api import ols
import statsmodels.api as sm
from sklearn.linear_model import LinearRegression

# import library for preparing data
from sklearn.model_selection import train_test_split

# import library for data preprocessing
from sklearn.preprocessing import MinMaxScaler

import warnings
warnings.filterwarnings("ignore")
"""
### Reading the dataset
"""
df = pd.read_csv('../input/the-boston-houseprice-data/boston.csv')
df.head()
"""
**Observations:**

* The price of the house indicated by the variable MEDV is the target variable and the rest are the independent variables based on which we will predict house price.
"""
df.info()
"""
**Observations**
* There are a total of 506 non-null observations in each of the columns. This indicates that there are no missing values in the data.

* Every column in this dataset is numeric in nature.
"""
df.describe()
"""
**Observations:**

* There is an average of around 6.2 rooms per dwelling across all the towns. 

* There is a very large variance for crime rate, as we can see that the lowest crime rate is 0.006 whereas the highest one is 88.97. It could be an outlier since it is very different from the median and even 75% quantile value. 
"""
"""
Before performing the modeling, it is important to check the univariate distribution of the variables.
"""
"""
### Univariate Analysis
"""
"""
#### Checking the distribution of the variables
"""
# let's plot all the columns to look at their distributions
for i in df.columns:
    plt.figure(figsize=(7, 4))
    sns.histplot(data=df, x=i, kde = True)
    plt.show()
"""
**Observations**
* **The variables CRIM and ZN are positively skewed.** This suggests that most of the areas have lower crime rates and most residential plots are under the area of 25,000 sq. ft.
* **The variable CHAS, with only 2 possible values 0 and 1, follows a binomial distribution**, and the majority of the houses are away from Charles river (CHAS = 0).
* The distribution of the variable AGE suggests that many of the owner-occupied houses were built before 1940. 
* **The variable DIS** (average distances to five Boston employment centers) **has a nearly exponential distribution**, which indicates that most of the houses are closer to these employment centers.
* **The variables TAX and RAD have a bimodal distribution.**, indicating that the tax rate is possibly higher for some properties which have a high index of accessibility to radial highways.  
* The dependent variable MEDV seems to be slightly right skewed.
"""
"""
As the dependent variable is sightly skewed, we will apply a **log transformation on the 'MEDV' column** and check the distribution of the transformed column.
"""
df['MEDV_log'] = np.log(df['MEDV'])
sns.histplot(data=df,x='MEDV_log',kde=True)
"""
**Observations**
* The log-transformed variable (**MEDV_log**) appears to have a **nearly normal distribution without skew**, and hence we can proceed.
"""
"""
Before creating the linear regression model, it is important to check the bivariate relationship between the variables. Let's check the same using the heatmap and scatterplot.
"""
"""
## Bivariate Analysis
"""
"""
Let's check the correlation using a heatmap
"""
plt.figure(figsize = (12,8))
cmap = sns.diverging_palette(230,20,as_cmap=True)
sns.heatmap(df.corr(),annot=True,fmt='.2f',cmap=cmap)
plt.show()
"""
**Observations:**

* We see a significantly correlated relationship between NOX and INDUS. 

* NOX also has a high positive correlation value with Age, as well as a significantly negatively correlated value with DIS.

* MEDV has a high correlation value with RM, and also significantly negatively correlated with LSTAT.

* INDUS has a high correlation value with TAX
"""
"""
Now, we will visualize the relationship between the pairs of features having significant correlations.
"""
# scatterplot to visualize the relationship between NOX and INDUS
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'INDUS', y = 'NOX', data = df) #write you code here
plt.show()
"""
**Observations:**

* There seems to be a somewhat positive relationship between the nitric oxides concentration and the proportion of non-retail business acres per town. When the proportion gets higher, there also seems to be a higher concentration of nitric oxides. 

* There are a number of data points that are on a vertical line at some point between the 15 and 20 values for INDUS.  
"""
# scatterplot to visualize the relationship between AGE and NOX
plt.figure(figsize=(6, 6))
sns.scatterplot(x='AGE', y = 'NOX', data = df)   #Write your code here
plt.show()
"""
**Observations:**

* When the proportion of owner occupied units built before 1940 gets higher, there also seems to be a higher concentration of Nitric oxides, thus, we can see a positive relationship. It is not the strongest relationship however, as we can see a good number of higher end AGE data points that are associated with low nitric oxides concentrations. 
"""
# scatterplot to visualize the relationship between DIS and NOX
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'DIS', y = 'NOX', data = df) #Write your code here
plt.show()
"""
**Observations:**

* We can see a negative, and somewhat curved relationship between nitric oxide concentration levels and weighted distances to five Boston employment centers. 

* Lower weighted distances seem to be associated with higher nitric oxide concentration levels, and higher weighted distances seem to be associated with lower nitric oxide concentration levels. 
"""
# scatterplot to visualize the relationship between AGE and DIS
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'AGE', y = 'DIS', data = df)
plt.show()
"""
**Observations:**
* The distance of the houses to the Boston employment centers appears to decrease moderately as the the proportion of the old houses increase in the town. It is possible that the Boston employment centers are located in the established towns where proportion of owner-occupied units built prior to 1940 is comparatively high.
"""
# scatterplot to visualize the relationship between AGE and INDUS
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'AGE', y = 'INDUS', data = df)
plt.show()
"""
**Observations:**
* No trend between the two variables is visible in the above plot.
"""
# scatterplot to visulaize the relationship between RAD and TAX
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'RAD', y = 'TAX', data = df)
plt.show()
"""
**Observations:**
* The correlation between RAD and TAX is very high. But, no trend is visible between the two variables. 
This might be due to outliers. 
"""
"""
Let's check the correlation after removing the outliers.
"""
# removing the data corresponding to high tax rate
df1 = df[df['TAX'] < 600]
# importing the required function
from scipy.stats import pearsonr
# calculating the correlation
print('The correlation between TAX and RAD is', pearsonr(df1['TAX'], df1['RAD'])[0])
"""
So the high correlation between TAX and RAD is due to the outliers. The tax rate for some properties might be higher due to some other reason.
"""
# scatterplot to visualize the relationship between INDUS and TAX
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'INDUS', y = 'TAX', data = df)
plt.show()
"""
**Observations:**
* The tax rate appears to increase with an increase in the proportion of non-retail business acres per town. This might be due to the reason that the variables TAX and INDUS are related with a third variable.
"""
# scatterplot to visulaize the relationship between RM and MEDV
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'RM', y = 'MEDV', data = df)
plt.show()
"""
**Observations:**
* The price of the house seems to increase as the value of RM increases. This is expected as the price is generally higher for more rooms.

* There are a few outliers in a horizontal line as the MEDV value seems to be capped at 50.
"""
# scatterplot to visulaize the relationship between LSTAT and MEDV
plt.figure(figsize=(6, 6))
sns.scatterplot(x = 'LSTAT', y = 'MEDV', data = df)
plt.show()
"""
**Observations:**
* The price of the house tends to decrease with an increase in LSTAT. This is also possible as the house price is lower in areas where lower status people live.
* There are few outliers and the data seems to be capped at 50.

We have seen that the variables LSTAT and RM have a linear relationship with the dependent variable MEDV. Also, there are significant relationships among a few independent variables, which is not desirable for a linear regression model. Let's first split the dataset.
"""
"""
### Splitting the Dataset

Let's split the dataset into the dependent and independent variables and further split it into train and test set in a ratio of 70:30 for train and test set.
"""
# separate the dependent and independent variable
Y = df['MEDV_log']
X = df.drop(columns = {'MEDV', 'MEDV_log'})

# add the intercept term
X = sm.add_constant(X)
# splitting the data in 70:30 ratio of train to test data
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.30 , random_state=1)
"""
Next, we will check the multicollinearity in the train dataset.
"""
"""
### Checking for Multicollinearity

We will use the Variance Inflation Factor (VIF), to check if there is multicollinearity in the data.

Features having a VIF score > 5 will be dropped/treated till all the features have a VIF score < 5
"""
from statsmodels.stats.outliers_influence import variance_inflation_factor

# function to check VIF
def checking_vif(train):
    vif = pd.DataFrame()
    vif["feature"] = train.columns

    # calculating VIF for each feature
    vif["VIF"] = [
        variance_inflation_factor(train.values, i) for i in range(len(train.columns))
    ]
    return vif


print(checking_vif(X_train))
"""
**Observations:**
* There are two variables with a high VIF - RAD and TAX. Let's remove TAX as it has the highest VIF values and check the multicollinearity again.
"""
# creating the model after dropping TAX
X_train = X_train.drop(columns = {'TAX'})

# checking for VIF
print(checking_vif(X_train))
"""
Now, we will create the linear regression model as the VIF is less than 5 for all the independent variables, and we can assume that multicollinearity has been removed between the variables.
"""
"""
## Model Building
"""
# create the model
model1 = sm.OLS(y_train, X_train).fit()


# get the model summary
model1.summary()
"""
**Observations:**

* The R-Squared valued value is 0.771, and the adjusted R_Squared value is 0.763

* The P-Values for CRIM, CHAS, NOX, RM, DIS, RAD, PTRATIO, and LSTAT all look statistically significant as they are either zero or close to it. 
"""
"""
### Examining the significance of the model

It is not enough to fit a multiple regression model to the data, it is necessary to check whether all the regression coefficients are significant or not. Significance here means whether the population regression parameters are significantly different from zero. 

From the above it may be noted that the regression coefficients corresponding to ZN, AGE, B, and INDUS are not statistically significant at level α = 0.05. In other words, the regression coefficients corresponding to these three are not significantly different from 0 in the population. Hence, we will eliminate the three features and create a new model.
"""
# creating the model after dropping columns 'MEDV', 'MEDV_log', 'TAX', 'ZN', 'AGE', 'B' 'INDUS' from df dataframe
Y = df['MEDV_log']
X = df.drop(columns = {'MEDV', 'MEDV_log', 'TAX', 'ZN', 'AGE', 'B', 'INDUS'}) 
X = sm.add_constant(X)

#splitting the data in 70:30 ratio of train to test data
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.30 , random_state=1)

# create the model
model2 = sm.OLS(y_train, X_train).fit() #write your code here
# get the model summary
model2.summary()
"""
**Observations:**
* We can see that the **R-squared value has decreased by 0.004**, since we have removed variables from the model, whereas the **adjusted R-squared value has decreased by 0.001.**
"""
"""
Now, we will check the linear regression assumptions.
"""
"""
### Checking the below linear regression assumptions

1. **Mean of residuals should be 0**
2. **No Heteroscedasticity**
3. **Linearity of variables**
4. **Normality of error terms**
"""
"""
**Checking for mean residuals**
"""
residuals = model2.resid

residuals.mean()
"""
**Observations:**

* The mean of the residuals is very close to 0, therefore, the assumption for residuals is satisfied
"""
"""
#### Checking for homoscedasticity

* Homoscedasticity - If the residuals are symmetrically distributed across the regression line, then the data is said to homoscedastic.

* Heteroscedasticity - If the residuals are not symmetrically distributed across the regression line, then the data is said to be heteroscedastic. In this case, the residuals can form a funnel shape or any other non-symmetrical shape.

* We'll use `Goldfeldquandt Test` to test the following hypothesis with alpha = 0.05:

    - Null hypothesis: Residuals are homoscedastic
    - Alternate hypothesis: Residuals have heteroscedastic
"""
from statsmodels.stats.diagnostic import het_white
from statsmodels.compat import lzip
import statsmodels.stats.api as sms
name = ["F statistic", "p-value"]
test = sms.het_goldfeldquandt(y_train, X_train)
lzip(name, test)
"""
**Observations:**

* The above test shows that the p-value is greater than 0.05 which is not stastitically significant and therefore means that we fail to reject the null-hypothesis which says that residuals are homoscedastic, therefore- the assumption for no heteroscedasticity is satisfied. 
"""
"""
#### Linearity of variables

It states that the predictor variables must have a linear relation with the dependent variable.

To test the assumption, we'll plot residuals and fitted values on a plot and ensure that residuals do not form a strong pattern. They should be randomly and uniformly scattered on the x-axis.
"""
# predicted values
fitted = model2.fittedvalues

# sns.set_style("whitegrid")
sns.residplot(x = fitted, y = residuals, color="lightblue", lowess=True) #write your code here
plt.xlabel("Fitted Values")
plt.ylabel("Residual")
plt.title("Residual PLOT")
plt.show()
"""
**Observations:**

* The above plot shows that there is no clear pattern, and that the points are randomly and uniformly scattered on the x axist, and therefore the linearity assumption is satisfied. 
"""
"""
#### Normality of error terms
The residuals should be normally distributed.
"""
# Plot histogram of residuals
sns.histplot(residuals, kde = True)
# Plot q-q plot of residuals
import pylab
import scipy.stats as stats

stats.probplot(residuals, dist="norm", plot=pylab)
plt.show()
"""
**Observations:**

The above two graphs show that the residuals are normally distributed, therefore the assumption for normality is satisfied. 
"""
"""
### Checking the performance of the model on the train and test data set
"""
# RMSE
def rmse(predictions, targets):
    return np.sqrt(((targets - predictions) ** 2).mean())


# MAPE
def mape(predictions, targets):
    return np.mean(np.abs((targets - predictions)) / targets) * 100


# MAE
def mae(predictions, targets):
    return np.mean(np.abs((targets - predictions)))


# Model Performance on test and train data
def model_pref(olsmodel, x_train, x_test):

    # In-sample Prediction
    y_pred_train = olsmodel.predict(x_train)
    y_observed_train = y_train

    # Prediction on test data
    y_pred_test = olsmodel.predict(x_test)
    y_observed_test = y_test

    print(
        pd.DataFrame(
            {
                "Data": ["Train", "Test"],
                "RMSE": [
                    rmse(y_pred_train, y_observed_train),
                    rmse(y_pred_test, y_observed_test),
                ],
                "MAE": [
                    mae(y_pred_train, y_observed_train),
                    mae(y_pred_test, y_observed_test),
                ],
                "MAPE": [
                    mape(y_pred_train, y_observed_train),
                    mape(y_pred_test, y_observed_test),
                ],
            }
        )
    )


# Checking model performance
model_pref(model2, X_train, X_test)  
"""
**Observations:**

* The mean absolute percentage error is 4.98 on the training data and 5.26 on the test data

* The mean absolute error is 0.14 on the training data and 0.15 on the test data. 

* The Root Mean Squared Error is 0.196 on the training data and 0.198 on the test data. We use this value to assess how accurately the model predicts the response.  
"""
"""
### Applying cross validation to improve the model and evaluating it using different evaluation metrics
"""
# import the required function

from sklearn.model_selection import cross_val_score

# build the regression model and cross-validate
linearregression = LinearRegression()                                    

cv_Score11 = cross_val_score(linearregression, X_train, y_train, cv = 10)
cv_Score12 = cross_val_score(linearregression, X_train, y_train, cv = 10, 
                             scoring = 'neg_mean_squared_error')                              


print("RSquared: %0.3f (+/- %0.3f)" % (cv_Score11.mean(), cv_Score11.std() * 2))
print("Mean Squared Error: %0.3f (+/- %0.3f)" % (-1*cv_Score12.mean(), cv_Score12.std() * 2))
"""
**Observations**
- The R-squared on the cross validation is 0.729, whereas on the training dataset it was 0.769
- And the MSE on cross validation is 0.041, whereas on the training dataset it was 0.038
"""
"""
We may want to reiterate the model building process again with new features or better feature engineering to increase the R-squared and decrease the MSE on cross validation.
"""
"""
### Let's get model coefficients in a pandas dataframe with column 'Feature' having all the features and column 'Coefs' with all the corresponding Coefs. We will then write the regression equation.
"""
coef1 = pd.DataFrame({"Features":model2.params, "Coefs":np.transpose(model2.params.values)})

Equation = "log (Price) ="
print(Equation, end='\t')
for i in range(len(coef1)):
    print('(', coef1['Features'][i], ') * ', coef1.index[i], '+', end = ' ')
"""
**Conclusions** 

* First we did EDA on the data set and did initial visual analysis on the variables including both univariate and bivariate analysis. 

* We then checked for multicollinearity, and dropped the TAX column as a result. 

* Then we ran our linear regression model on the data and observed the results. Following that, we dropped variables that were not significant and examined the changes in the R-Squared variable. 

* Checked to make sure our model satisfies the assumptions of linear regression. 

* We then checked the performance of our model using various evaluation metrics including cross validation. 

* We then finalized our regression equation
"""