# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# +
# Imports
import numpy as np
import pandas as pd
import altair as alt
from sklearn.model_selection import train_test_split
import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Allow Altair to make plots using more than 5000 rows
alt.data_transformers.disable_max_rows()
# -

# ## Description of the Dataset
#
# This data was collected as part of California's Census in 1990. An analysis first emerged in 1997 in a paper titled Sparse Spatial Autoregressions. It was collected by <NAME> and <NAME>. This dataset was improved upon by <NAME>, who added a categorical column of proximity to the ocean. Each row in the dataset contains details for a single census block group in California. The data collected contains information about the houses for each block group and also details about the population in the city. 
#
# ## Load the Data

# +
train = pd.read_csv('https://github.com/UBC-MDS/DSCI_522_GROUP_312/blob/master/data/train.csv?raw=true')
test = pd.read_csv('https://github.com/UBC-MDS/DSCI_522_GROUP_312/blob/master/data/test.csv?raw=true')


X_train = train.iloc[:,:9]
y_train = train['median_house_value']
X_test = test.iloc[:,:9]
y_test = test['median_house_value']
# -

# ## Explore the Dataset

train.describe(include='all')

# +
print("There are {0} records in the training portion of the dataset. Each record is a census block group.\n".format(
    X_train.count().min()))

print("The median age of houses/complexes in census blocks ranges from {0} to {1} with a mean of {2} years old and a median of {3} years old.\n".format(
    X_train['housing_median_age'].min(),
    X_train['housing_median_age'].max(),
    round(X_train['housing_median_age'].mean(), 2),
    X_train['housing_median_age'].median()))

print("The total number of rooms in a census block ranges from {0} to {1} with a mean of {2} rooms and a median of {3} rooms.\n".format(
    X_train['total_rooms'].min(),
    X_train['total_rooms'].max(),
    round(X_train['total_rooms'].mean(), 2),
    X_train['total_rooms'].median()))

print("The number of bedrooms in a census block ranges from {0} to {1} with a mean of {2} bedrooms and a median of {3} bedrooms.\n".format(
    X_train['total_bedrooms'].min(),
    X_train['total_bedrooms'].max(),
    round(X_train['total_bedrooms'].mean(), 2),
    X_train['total_bedrooms'].median()))

print("The population of a census block ranges from {0} to {1} with a mean of {2} and a median of {3}.\n".format(
    X_train['population'].min(),
    X_train['population'].max(),
    round(X_train['population'].mean(), 2),
    X_train['population'].median()))

print("The number of households in a census block ranges from {0} to {1} with a mean of {2} and a median of {3}.\n".format(
    X_train['households'].min(),
    X_train['households'].max(),
    round(X_train['households'].mean(), 2),
    X_train['households'].median()))

print("The median annual income in a census block ranges from ${0} to ${1} with a mean of ${2} and a median of ${3}.\n".format(
    X_train['median_income'].min(),
    X_train['median_income'].max(),
    round(X_train['median_income'].mean(), 2),
    X_train['median_income'].median()))

print("Ocean Proximity is a categorical value with one of the values:\n\t{0}, \n\t{1} (very close to the ocean), \n\t{2}, \n\t{3}, \n\t{4}.\n".format(
    X_train['ocean_proximity'].unique()[0].lower(),
    X_train['ocean_proximity'].unique()[1].lower(),
    X_train['ocean_proximity'].unique()[2].lower(),
    X_train['ocean_proximity'].unique()[3].lower(),
    X_train['ocean_proximity'].unique()[4].lower()))

print("The median house value in a census block ranges from ${0} to ${1} with a mean of ${2} and a median of ${3}.\n".format(
    y_train.min(),
    y_train.max(),
    round(y_train.mean(), 2),
    y_train.median()))
# -

# ## Initial Thoughts
#
# - Blocks vary drastically in the number of bedrooms and total number of rooms.
# - Lowest median income blocks differ in the number of households (some highly populated blocks have very low income).
# - New houses vary in their value (interesting to study the relationship between age and value).
# - 7 out of the 10 lowest-value houses are INLAND. (proximity to ocean appears to heavily influence value).
# - Median House Value and Median income appear to be exactly capped at 500000 and 150000 exactly.
#
# ## Research Question
#
# Our research will focus on predicting median housing value given the independent variables about location and population.
#
# ## Analysis and Visualizations
#
# The following plots visualize the relationship between various independent variables and the dependent variable, median house value. It is clear that some variables appear to have linear relationships, while others have high variance.
#
# ![Image](total-rooms_scatterplot.png) ![Image](total-bedrooms_scatterplot.png)
# ![Image](households_scatterplot.png) ![Image](population_scatterplot.png)
# ![Image](median-age_scatterplot.png) ![Image](median-income_scatterplot.png)
#
# This raises a question of whether the variables are correlated. The following heatmap shows correlation values between all variables. The negative correlation result for latitude and longitude is purely related to the location of California.
#
# ![Image](correlation_heatmap.png)
#
# Building on this, we can obtain the Variance Inflation Factors (VIFs) to identify multicollinearity between variables. Generally, a VIF of 5 is extremely high.

pd.read_csv('vif_table.csv')

# As we can see, VIFs are extremely high, so there are relationships between the independent variables.
#
# The highest VIF is for total_bedrooms, and it appears to be linearly related to total_rooms (as bedrooms are included in a room count).
#
# ![Image](total-rooms_total-bedrooms.png)

# ## Summary and Conclusions
#
# The California Housing Dataset contains nine variables that we identified as explanatory, for the response variable of the median housing value per housing block in California. Previous analyses have been completed, and the general conclusion is that this data can be predicted linearly. Some variables have stronger linear relationships with the response than others, but there is also very strong multicollinearity, which may cause unwanted effects. To predict the median housing value, we will compare Linear Regression, KNN, and Random Forest Regressors.



StarcoderdataJupyterScriptsDedupFiltered