import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
insurance= pd.read_csv("/kaggle/input/insurance/insurance.csv")
insurance.head()
insurance.info()
insurance.describe()
insurance.shape
"""
## Point to note

- Dataset has 1338 rows and 7 columns.
- Looking at the data, there seems to be some fields that are categorical in nature, but in integer/float type.

- We will analyse and finalize whether to convert them to categorical or treat as integer.
"""
"""
# DATA QUALITY CHECK
"""
"""
## Check for NULL/MISSING values
"""
# percentage of missing values in each column
round(100*(insurance.isnull().sum()/len(insurance)),2).sort_values(ascending = False)
# percentage of missing values in each row
round(100*(insurance.isnull().sum(axis=1)/len(insurance)),2).sort_values(ascending = False)[:5]
"""
### Finding

- There are no missing / Null values either in columns or rows
"""
"""
## Duplicate Check
"""
insurance_dub=insurance.copy()
# Checking for duplicates and dropping the entire duplicate row if any
insurance_dub.drop_duplicates(subset=None, inplace=True)
insurance_dub.shape
insurance.shape
"""
### Insights

-The shape after running the drop duplicate command is same as the original dataframe.

-Hence we can conclude that there were zero duplicate values in the dataset.
"""
"""
## Data Cleaning
- Checking value_counts() for entire dataframe.

- This will help to identify any Unknow/Junk values present in the dataset.
"""
for col in insurance:
    print(insurance[col].value_counts(ascending=False), '\n\n\n')
"""
### Insights
- There seems to be no Junk/Unknown values in the entire dataset.
"""
"""
## Creating Dummy Variables

- We will create DUMMY variables for 4 categorical variables 'sex','children','smoker' & 'region'.

- Before creating dummy variables, we will have to convert them into 'category' data types.
"""
# Check the datatypes before convertion
insurance.info()
# Convert to 'category' data type

insurance['sex']=insurance['sex'].astype('category')
insurance['smoker']=insurance['smoker'].astype('category')
insurance['children']=insurance['children'].astype('category')
insurance['region']=insurance['region'].astype('category')
# This code does 3 things:
# 1) Create Dummy variable
# 2) Drop original variable for which the dummy was created
# 3) Drop first dummy variable for each set of dummies created.

insurance_new = pd.get_dummies(insurance, drop_first=True)
insurance_new.info()
insurance_new.shape
"""
# SPLITTING THE DATA
- Splitting the data to Train and Test: - We will now split the data into TRAIN and TEST (70:30 ratio)
- We will use train_test_split method from sklearn package for this
"""

# Check the shape before spliting

insurance_new.shape
# Check the info before spliting

insurance_new.info()
from sklearn.model_selection import train_test_split

# We should specify 'random_state' so that the train and test data set always have the same rows, respectively

np.random.seed(0)
df_train, df_test = train_test_split(insurance_new, train_size = 0.70, test_size = 0.30, random_state = 100)
"""
- Verify the info and shape of the dataframes after split:
"""
df_train.info()
df_train.shape
df_test.info()
df_test.shape
"""
# EXPLORATORY DATA ANALYSIS
-We need to perform the EDA on TRAINING (df_train) Dataset.
## Visualising Numeric Variables
- Let's make a pairplot of all the numeric variables.
"""
df_train.info()
df_train.columns
# Create a new dataframe of only numeric variables:

insurance_num=df_train[[ 'age', 'bmi', 'charges']]

sns.pairplot(insurance_num, diag_kind='kde')
plt.show()
"""
## Insights
- The above Pair-Plot tells us that there is a LINEAR RELATION between 'age','bmi' and 'charges'
"""
"""
## Visualising Catagorical Variables
"""
df_train.info()
# Build boxplot of all categorical variables (before creating dummies) againt the target variable 'cnt' 
# to see how each of the predictor variable stackup against the target variable.

plt.figure(figsize=(25, 10))
plt.subplot(2,2,1)
sns.boxplot(x = 'sex', y = 'charges', data = insurance)
plt.subplot(2,2,2)
sns.boxplot(x = 'children', y = 'charges', data = insurance)
plt.subplot(2,2,3)
sns.boxplot(x = 'smoker', y = 'charges', data = insurance)
plt.subplot(2,2,4)
sns.boxplot(x = 'region', y = 'charges', data = insurance)
plt.show()
"""
# Insights
There were  4 categorical variables in the dataset.

We used Box plot (refer the fig above) to study their effect on the dependent variable (‘charges’) .

The inference that We could derive were:

- **sex**: data is bit baised towards male , data is represented as 50.5 : 49.5 for male: female . Interquartile range for male is higher compared  to interquartile range for female.

- **smoker**: 20.5 % of our data are smoker. Median for Smokers are quite high compared to people who don't smoke.

- **Children** : People have 2 children are having highest data  distribution  between 50 percentile to 75 percentile among all others.
- **Region** : Region as Souteast are having highest data  distribution  between 50 percentile to 75 percentile among all others.
"""
"""
## Correlation Matrix
"""
# Let's check the correlation coefficients to see which variables are highly correlated. Note:
# here we are considering only those variables (dataframe: insurance_new) that were chosen for analysis

plt.figure(figsize = (25,10))
sns.heatmap(insurance_new.corr(), annot = True, cmap="RdBu")
plt.show()
"""
## Insights:
- The heatmap clearly shows which all variable are multicollinear in nature, and which variable have high collinearity with the target variable.
- We will refer this map back-and-forth while building the linear model so as to validate different correlated values along with VIF & p-value, for identifying the correct variable to select/eliminate from the model.
"""
"""
# RESCALING THE FEATURES
"""
from sklearn.preprocessing import MinMaxScaler
scaler= MinMaxScaler()
df_train.head()
df_train.columns
# Apply scaler() to all the numeric variables

num_vars = ['age', 'bmi', 'charges']

df_train[num_vars] = scaler.fit_transform(df_train[num_vars])
# Checking values after scaling
df_train.head()
df_train.describe()
"""
# BUILDING A LINEAR MODEL
"""
"""
## Dividing into X and Y sets for the model building
"""
y_train = df_train.pop('charges')
X_train = df_train
"""
### RFE
Recursive feature elimination: We will be using the **LinearRegression function from SciKit Learn** for its compatibility with RFE (which is a utility from sklearn)
"""
#Importing RFE and LinearRegression
from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression
# Running RFE with the output number of the variable equal to 6
lm= LinearRegression()
lm.fit(X_train,y_train)
rfe= RFE(lm,6)
rfe=rfe.fit(X_train,y_train)
list(zip(X_train.columns,rfe.support_,rfe.ranking_))
col=X_train.columns[rfe.support_]
col
X_train.columns[~rfe.support_]
# Creating X_test dataframe with RFE selected variables
X_train_rfe = X_train[col]

"""
# Building Linear Model using 'STATS MODEL'
"""
"""
## Model 1
"""
"""
### VIF
"""
# Check for the VIF values of the feature variables. 
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train_rfe.columns
vif['VIF'] = [variance_inflation_factor(X_train_rfe.values, i) for i in range(X_train_rfe.shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif
import statsmodels.api as sm

# Add a constant
X_train_lm1 = sm.add_constant(X_train_rfe)

# Create a first fitted model
lr1 = sm.OLS(y_train, X_train_lm1).fit()
# Check the parameters obtained

lr1.params
# Print a summary of the linear regression model obtained
print(lr1.summary())
"""
## Model 2
- Removing the variable 'children_5' based on its High p-value 
"""
X_train_new = X_train_rfe.drop(["children_5"], axis = 1)
"""
### VIF
"""
# Check for the VIF values of the feature variables. 
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train_new.columns
vif['VIF'] = [variance_inflation_factor(X_train_new.values, i) for i in range(X_train_new.shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif

# Add a constant
X_train_lm2 = sm.add_constant(X_train_new)

# Create a first fitted model
lr2 = sm.OLS(y_train, X_train_lm2).fit()
# Check the parameters obtained

lr2.params
# Print a summary of the linear regression model obtained
print(lr2.summary())
"""
## Model 3

- Removing the variable 'children_4' based on its High p-value
"""
X_train_new = X_train_new.drop(["children_4"], axis = 1)
"""
### VIF
"""
# Check for the VIF values of the feature variables. 
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train_new.columns
vif['VIF'] = [variance_inflation_factor(X_train_new.values, i) for i in range(X_train_new.shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif
# Add a constant
X_train_lm3 = sm.add_constant(X_train_new)

# Create a first fitted model
lr3 = sm.OLS(y_train, X_train_lm3).fit()
# Check the parameters obtained

lr3.params
# Print a summary of the linear regression model obtained
print(lr3.summary())
"""
## Model 4
- Removing the variable 'children_2' based on its High p-value
"""
X_train_new = X_train_new.drop(["children_2"], axis = 1)
"""
### VIF
"""
# Check for the VIF values of the feature variables. 
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train_new.columns
vif['VIF'] = [variance_inflation_factor(X_train_new.values, i) for i in range(X_train_new.shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif
# Add a constant
X_train_lm4 = sm.add_constant(X_train_new)

# Create a first fitted model
lr4 = sm.OLS(y_train, X_train_lm4).fit()
# Check the parameters obtained

lr4.params
# Print a summary of the linear regression model obtained
print(lr4.summary())
"""
### Insights
- This model looks good, as there seems to be VERY LOW Multicollinearity between the predictors and the p-values for all the predictors seems to be significant. For now, we will consider this as our final model (unless the Test data metrics are not significantly close to this number).
"""
"""
# Final Model Interpretation

## Hypothesis Testing:

### Hypothesis testing states that:

H0:B1=B2=...=Bn=0
H1: at least one Bi!=0

### lr4 model coefficient values
- const -0.0415

- bmi            0.1779     
- smoker_yes     0.3833     
- age  0.1950
"""
"""
## Insights
- From the lr4 model summary, it is evident that all our coefficients are not equal to zero which means We REJECT the NULL HYPOTHESIS
"""
"""
### F Statistics

F-Statistics is used for testing the overall significance of the Model: Higher the F-Statistics, more significant the Model is.

- **F-statistic**:                     895.7
- **Prob (F-statistic)**:          7.80e-274
The F-Statistics value of 2895.7 (which is greater than 1) and the p-value of '~0.0000' states that the overall model is significant
"""
"""
### The equation of best fitted surface based on model lr4:
charges = -0.0415  + (**age**  × 0.1950) + (**bmi** × 0.1779) + (**smoker_yes** × 0.3833 ) 
"""
"""
### Interpretation of Coefficients:
- **age**: A coefficient value of ‘0.1950’ indicated that a unit increase in age variable, increases the charge numbers by 0.1950 units.
- **bmi**: A coefficient value of ‘0.1779’ indicated that a unit increase in bmi variable, increases the charge numbers by 0.1779 units.
- **smoker_yes**: A coefficient value of ‘0.3833’ indicated that a unit increase in smoker_yes variable, increases the charge numbers by 0.3833 units.


"""
"""
# ASSUMPTIONS
"""
"""
## Error terms are normally distributed with mean zero (not X, Y)
- Residual Analysis Of Training Data
"""
y_train_pred = lr4.predict(X_train_lm4)
res = y_train-y_train_pred
# Plot the histogram of the error terms
fig = plt.figure()
sns.distplot((res), bins = 20)
fig.suptitle('Error Terms', fontsize = 20)                  # Plot heading 
plt.xlabel('Errors', fontsize = 18)  
"""
### Insights
- From the above histogram, we could see that the Residuals are normally distributed. Hence our assumption for Linear Regression is valid.
"""
"""
## There is a linear relationship between X and Y
"""
insurance_new=insurance_new[[ 'bmi', 'age', 'charges']]

sns.pairplot(insurance_new, diag_kind='kde')
plt.show()
"""
### Insight
- Using the pair plot, we could see there is a linear relation between bmi and age variable with the predictor ‘charges’.
"""
"""
## There is No Multicollinearity between the predictor variables
"""
# Check for the VIF values of the feature variables. 
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train_new.columns
vif['VIF'] = [variance_inflation_factor(X_train_new.values, i) for i in range(X_train_new.shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif
"""
### Insight
- From the VIF calculation we could find that there is no multicollinearity existing between the predictor variables, as all the values are within permissible range of below 
"""
"""
# MAKING PREDICTION USING FINAL MODEL
"""
"""
Now that we have fitted the model and checked the assumptions, it's time to go ahead and make predictions using the final model (lr4)

### Applying the scaling on the test sets
"""
# Apply scaler() to all numeric variables in test dataset. Note: we will only use scaler.transform, 
# as we want to use the metrics that the model learned from the training data to be applied on the test data. 
# In other words, we want to prevent the information leak from train to test dataset.

num_vars = ['age', 'bmi', 'charges']

df_test[num_vars] = scaler.transform(df_test[num_vars])
df_test.head()
df_test.describe()
"""
### Dividing into X_test and y_test
"""
y_test = df_test.pop('charges')
X_test = df_test
X_test.info()
#Selecting the variables that were part of final model.
col1=X_train_new.columns
X_test=X_test[col1]
# Adding constant variable to test dataframe
X_test_lm4 = sm.add_constant(X_test)
X_test_lm4.info()
# Making predictions using the final model (lr4)

y_pred = lr4.predict(X_test_lm4)

"""
# MODEL EVALUATION
"""
# Plotting y_test and y_pred to understand the spread

fig = plt.figure()
plt.scatter(y_test, y_pred, alpha=.5)
fig.suptitle('y_test vs y_pred', fontsize = 20)              # Plot heading 
plt.xlabel('y_test', fontsize = 18)                          # X-label
plt.ylabel('y_pred', fontsize = 16) 
plt.show()
"""
## R^2 Value for TEST
"""
from sklearn.metrics import r2_score
r2_score(y_test, y_pred)
"""
## Adjusted R^2 Value for TEST
"""
# We already have the value of R^2 (calculated in above step)

r2=0.75783003115855
# Get the shape of X_test
X_test.shape
# n is number of rows in X

n = X_test.shape[0]


# Number of features (predictors, p) is the shape along axis 1
p = X_test.shape[1]

# We find the Adjusted R-squared using the formula

adjusted_r2 = 1-(1-r2)*(n-1)/(n-p-1)
adjusted_r2
"""
## Final Result Comparison

- **Train R^2** :0.743
- **Train Adjusted R^2** : 0.742
- **Test R^2** :0.757
- **Test Adjusted R^2** :0.756

This seems to be a really good model that can very well 'Generalize' various datasets.
"""
"""
# FINAL REPORT
"""
"""
As per our final Model, the top predictor variables that influences the Charges are:
- **age**: A coefficient value of ‘0.1950’ indicated that a unit increase in age variable, increases the charge numbers by 0.1950 units.
- **bmi**: A coefficient value of ‘0.1779’ indicated that a unit increase in bmi variable, increases the charge numbers by 0.1779 units.
- **smoker_yes**: A coefficient value of ‘0.3833’ indicated that a unit increase in smoker_yes variable, increases the charge numbers by 0.3833 units.


"""