"""
# Regression Definition 
**Regression is a statistical method used to determine the strength of the relationship between one dependent variable (Y) and a series of other changing variables (X).**

**If the relationship is linear between Y and X, then it is Linear Regression**
"""
"""
# Assumptions of Linear Regression
1. X and Y are linearly related
2. No Multicollinearity between X variables
3. No Auto-correlation between residuals
4. Residuals follow Normal distribution
5. Residuals follow homoscedastic
"""
# Import the file
import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))
import pandas as pd
from patsy import dmatrices
# Read the data
df = pd.read_csv('/kaggle/input/dataset/data.csv')
df.head()
"""
# Assumption 1

**Relationship between X variable and Y variable is Linear**

1. Plot the scatter plot for each X with Y
2. Look for approximate linear relationship
3. if it's not linear, then try to "Drop X variable" or "Change the technique" or "apply transformation log(X)".


"""
from matplotlib import pyplot as plt 
# Scatter plot between X1 and Output
plt.scatter(df.X1, df.Output)
# Scatter plot between X2 and Output
plt.scatter(df.X2, df.Output)
# Scatter plot between X3 and Output
plt.scatter(df.X3, df.Output)
# Scatter plot between X4 and Output
plt.scatter(df.X4, df.Output)
"""
**from above plots it is clear that, X1 variable and X2 variable are almost Linear to Y variable**
"""
"""
# Assumption 2
"""
"""
No Multicollinearity between X variables
* There should not any significant relationship between X variables
* VIF (Variance Inflation factor) can be used to find out the relationship between X variables
* In VIF, each X variable is picked and regress with other X variables.
* VIF = 1/(1-R2)
* Higher the R2, greater the VIF.
* In general, VIF > 5 or VIF > 10 indicates high multicollinearity.
"""
df.head(2)
df.columns
# Consider only X variables
df1 = df[['X1', 'X2', 'X3', 'X4']]
df1.head(2)
from statsmodels.stats.outliers_influence import variance_inflation_factor
VIF = pd.DataFrame()
VIF['feature'] = df1.columns
VIF['VIF'] = [variance_inflation_factor(df1.values, i) 
                          for i in range(len(df1.columns))] 
VIF
"""
**From above it is clear that, VIF values are > 10. It means X variables are highly correlated.**
"""
"""
# Assumption 3
"""
"""
**No Auto correlation between Residuals**
* Residuals are random and independent.
"""
"""
**Residuals are Random**

1. For a given sample (X_train, y_train) from a Dataset, run the linear regression model.
2. predict the output = y(pred)
3. difference between Y and Y(pred) is Residual.
4. If we pick another sample of (X_train, y_train), model will generate the output = y(pred) which is different from previous one. 
5. Now we have different set of Residuals. 
6. So, Residuals are Random.
"""
"""
**Residuals are Independent**

* Two random variables are independent if the probability of one of them taking up some value doesn’t depend on what value the other variable has taken

   **Example:**

* When you roll a die twice, the probability of its coming up as (1,2,3,4,5,6) in the second throw does not depend on the value it came up on the first throw. So the two throws are independent random variables that can each take a value from (1,6), independent of the other throw
"""
"""
**Check if residuals are independent**

* sometimes patterns can be detected in the plot of 'residual errors' vs 'predicted values' or         'residual errors' vs 'actual values'.
* Durbin-Watson test is used to check if residuals are independent or not.
* It measures degree of correlation of each residual error with the ‘previous’ residual error
"""
model = 'Output ~  X1 + X2 + X3 + X4'
# in Above expression, Output is Y varialble and remaning are X variables.
y, X = dmatrices(model, df, return_type='dataframe')
y.head(2)
X.head(2)
# Now split the (train/test) data into (70/30)
import numpy as np
split_num = np.random.rand(len(X)) < 0.7
X_train = X[split_num]
y_train = y[split_num]
X_test = X[~split_num]
y_test = y[~split_num]
X_train.shape
# Build the model
import statsmodels.api as sm
LR = sm.OLS(y_train, X_train).fit()
LR.summary()
# Predict 'Y' for the test data
y_test_pred = LR.predict(X_test)
y_test_pred = pd.DataFrame(y_test_pred)
y_test_pred.head()
# Rename the column
y_test_pred = y_test_pred.rename(columns = {0:'Output'})
y_test_pred.head()
# Calculate the residual
Residual = y_test - y_test_pred
Residual
# Residual vs Predicted value
plt.scatter(x = y_test_pred, y = Residual)
plt.xlabel('Predicted values')
plt.ylabel('Residuals')
"""
**Above graph illustrates a linear pattern at the end**

**If Durbin Watson value is around 2, then there is no auto correlation between residuals**
"""
"""
# Assumption 4
"""
"""
**Residuals are Normally Distributed**
1. 'Skewness' and 'Kurtosis' is used to measure the normality of Residuals.
2. Skewness of a perfectly normal distribution is 0 and its kurtosis is 3.0
3. Statistical tests like 'Jarque Bera' or Omnibus can be used.
4. If p value <= 0.05, then we can say distribution is normal with >= 95% cofidence.
"""
from statsmodels.compat import lzip
import statsmodels.stats.api as sms
# 'Jarque Bera test' for finding the normality of Residuals
name = ['Jarque-Bera test', 'p-value', 'Skewness', 'Kurtosis']
# Perform 'Jarque-Bera' test
JB_test = sms.jarque_bera(Residual)
# Print
lzip(name, JB_test)
"""
**From above it is clear that, Skewness is close to 0 and Kurtosis is close to 3**

**We can confirm by plotting frequency of Residuals**
"""
Residual.hist(bins = 40)
plt.show()
"""
**Above graph illustrates that Residuals are almost normally distributed**
"""
"""
# Assumption 5
"""
"""
**Residuals follow homoscedastic**
* Homo means Same, scedastic means variance - so homoscedastic means same variance (Constant variance)
* Heteroscedastic means - Different Variance
"""
"""
**Below are few tests to find out homoscedasticity**
1. Park test
2. Glejser test
3. Breusch–Pagan test
4. White test
5. Goldfeld–Quandt test
"""
# White test 
from statsmodels.stats.diagnostic import het_white
keys = ['Lagrange Multiplier statistic:', 'LM test\'s p-value:', 'F-statistic:', 'F-test\'s p-value:']
result = het_white(Residual, X_test)
lzip(keys, result)
"""
**F test Hypothesis**
* Null Hypothesis :     Residuals are homoscedasticity
* Alternate hypothesis: Residuals are heteroscedasticity
"""
"""
**From above, it is clear that p-value <=0.05. Hence null hypothesis is rejected.** 

**So Residuals follow heteroscedasticity**
"""