# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 2
#     language: python
#     name: python2
# ---

# # Use Cases for Linear and Logistic Regression with Scikit-Learn
#
# * **Continuous Predictors**
#     * Linear Regression 
#     * Tree Regression
# * **Binary Classifiers**
#     * Logistic Regression
#     * Naive Bayes (if Time)
#     
# We'll use my good friend Scikit-Learn: http://scikit-learn.org/stable/

# ## Continuous Variables & Linear Regression
#
# Outcomes can take on a real-value in a range between $-\inf$ and $\inf$. It's still good if your predictors only take on a range between certain values. 
#
# Formula in the form $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + ... + \beta_n X_n + \epsilon$, where $\epsilon$ is the error term. Includes things we forgot to include, things we can't measure, measurement error, etc. 
#
# Each $\beta$ is the value we expect per a one-unit change in the corresponding $X$. So if $\beta_3 = 4.3$, we expect $Y$ to go up $4.3$ units whenever $X_3$ increase by one-unit. 
#
# Fit in the new data into the formula, multiply by the $\beta$'s and that's your prediction
#
# We use the form $\hat{Y} = \hat{\beta_0} + \hat{\beta_1} X_1 + \hat{\beta_2} X_2 + ... + \hat{\beta_n} X_n$ where the 'hats' show those are the calculated values, nor the error. 

# ## Calculating the X's 
#
# We use a series of matrix operations to calculate the values of the $\beta$'s. 
#
# $$ X = 
# \begin{matrix}
# 1 & X_{1,1} & X_{1,2} & X_{1,3} &...& X_{1,n}\\
# 1 & X_{2,1} & X_{2,2} & X_{2,3} &...& X_{2,n}\\
# \vdots & \vdots & \vdots & \vdots& \ddots & \vdots \\
# 1 & X_{m,1} & X_{m,2} & X_{m,3} &...& X_{m,n}
# \end{matrix}$$
#
# Where the $X$'s are our data points. 
#
# We do some matrix operations: $(X^TX)^{-1}X^T y = \hat{\underline{\beta}}$ to generate our coefficients. Then we use the formula $\hat{Y} = \hat{\underline{\beta}}X$ to get our predictions. 
#
# We don't actually do this. Matrix operations are insanely expensive for computers to perform. Matrix multiplication is $O(n^{2.373})$ under the fastest matrix multiplication algorithms. So computers use a gradient descent algorithm that tries to minimize the squared distance between the predicted outcomes and the actual outcomes.
#
# We define the residuals, $r$, as $r = \hat{Y} - Y$. We want those residuals to look like random noise, and ideally normally distributed with mean 0. 
# <img src = "good_resids1.png">
#
# Residuals with a pattern suggest we missed something 
# <img src = "bad_resids.png">
#
#

# ## Let's Look at Some Data
#
# We'll use the Boston Housing Data for this. 

# +
import numpy as np
import pandas as pd 
boston = pd.read_csv("/Users/evancolvin/dropbox/projects/PyData/Boston.csv", sep = '\t')
cancer = pd.read_csv("/Users/evancolvin/dropbox/projects/PyData/cancer.csv")

import matplotlib.pyplot as plt
# %matplotlib inline
# -

boston.head(20)

#
#
# * **crim:** Per Capita Crime per Town
# * **zn:** proportion of residential land zoned for lots over 25,000 sq.ft
# * **indus:** proportion of non-retail business acres per town
# * **chas:** Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).
# * **nox:** nitrogen oxides concentration (parts per 10 million)
# * **rm:** average number of rooms per dwelling
# * **age:** proportion of owner-occupied units built prior to 1940
# * **dis:** weighted mean of distances to five Boston employment centres
# * **rad:** index of accessibility to radial highways
# * **tax:** full-value property-tax rate per <span>$</span>10,000
# * **ptratio:** pupil-teacher ratio by town
# * **black:** 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
# * **lstat:** lower status of the population (percent)
# * **medv:** median value of owner-occupied homes in 1000s. <--What we Want to Predict
#

boston.describe()

cancer.describe()

# ## Splitting the data sets for cross-validation 
#
# Simple training and test sets with 70% of the day and 30% of the data respectively. 

from sklearn.cross_validation import train_test_split as split
boston_train, boston_test = split(boston, test_size = .30)
cancer_train, cancer_test = split(cancer, test_size = .30)

# ## Linear Regression 

features = ['crim', 'zn', 'indus', 'chas', 'nox', 'rm', 
            'age', 'dis', 'rad', 'tax', 'ptratio', 'black', 'lstat']
target = 'medv'
predictor_data = boston_train[features].values
median_price = boston_train[target].values

# +
from sklearn.linear_model import LinearRegression
linear_regression = LinearRegression()
linear_regression.fit(predictor_data, median_price)


y_hat = linear_regression.predict(predictor_data)
resids = y_hat - median_price
plt.figure(figsize = (10.5, 6))
plt.scatter(range(len(median_price)), resids)


print "R2 " + str(linear_regression.score(predictor_data, median_price)) + "\n\n"
coefficients = linear_regression.coef_.tolist()
print "intercept: " + str(linear_regression.intercept_)
for predictor in range(len(features)):
    print features[predictor] + ": " + str(coefficients[predictor])

# +
testing_data = boston_test[features].values
testing_targets = boston_test[target].values
new_predictions = linear_regression.predict(predictor_data)
print "The first predictions for prices are: "
for i in range(10):
    print "\t" + str(new_predictions[i])
    
#print linear_regression.predict(testing_data)[:10]
R2 = linear_regression.score(testing_data, testing_targets)
print "\nAnd the R-Squared value for the unseen data is: " + str(R2)
# -

# ## Regression Trees 
#
# "Prediction trees use the tree to represent the recursive partition. Each of the terminal nodes, or leaves, of the tree represents a cell of the partition, and has attached to it a simple model which applies in that cell only. A point x belongs to a leaf if x falls in the corresponding cell of the partition."--<NAME> 
#
# * Data "splits" to make the predictions
# * Greedy algorithm: makes best choice at each stage; doesn't look far ahead
# * May miss a really good split down the line 
# * Bagging: Combining similar trees and averaging outcome
# * Random Forest: Combining vastly different trees and average outcome
#     * Does this by hiding data from the trees at different time so optimum's look different
#     * <NAME>: Can't Overfit 
#
# <img src="https://camo.githubusercontent.com/4dad481e246c73548fb37cb3dc26ea106a2494dc/687474703a2f2f65636f6c6f67792e6d73752e6d6f6e74616e612e6564752f6c61626473762f522f6c6162732f6c6162362f64656d6f5f332e706e67">

from sklearn.tree import DecisionTreeRegressor
tree = DecisionTreeRegressor(max_depth = 10)
tree.fit(predictor_data, median_price)
from sklearn.metrics import r2_score
print "The R2 for the Tree Regression is " + str(r2_score(median_price, tree.predict(predictor_data)))

new_preds = tree.predict(testing_data)
print "The R2 for the new data is " + str(r2_score(testing_targets, new_preds))

# ## Classification & Logistic Regression 
#
# ### Logistic Regression 
# * Feed linear regression into the sigmoid function $S(t) = \frac{1}{1 + e^{-t}}$ where $t$ is the linear regression
# * Used for binary classification
#     * $ \le .5 \Rightarrow$ class 0
#     * $ > .5 \Rightarrow$ class 1
# * Need new metrics since $R^2$ doesn't make sense here 
#     * Log-Loss if you predict probabilities (big for Kaggle)
#     * Accuracy score that looks at the proportion of the correctly classified 
#
# <img src="sigmoid.png">

cancer.head(20)

cancer.describe()

features = ['Clump Thickness', 'Uniformity of Cell Shape', 'Marginal Adhesion', 
            'Single Epithelial Cell Size', 'Brand Chromatin', 'Normal Nucleoli', 'Mitosis']
target = 'Class'
predictor_data = cancer_train[features].values
cell_class = cancer_train[target].values

from sklearn.linear_model import LogisticRegression
logistic_regression = LogisticRegression()
logistic_regression.fit(predictor_data, cell_class)
from sklearn.metrics import accuracy_score
print "The model predicts " + str(accuracy_score(cell_class, logistic_regression.predict(predictor_data))*100) + "% of the cells accurately"

validation_data = cancer_test[features].values
validation_target = cancer_test[target].values

new_preds = logistic_regression.predict(validation_data)
print "Also predicts " + str(accuracy_score(validation_target, new_preds)*100) + "% of new observations"

# ## Naive Bayes
#
# New Belief = Old Belief $\times$ Likelihood of Evidence / Evidence $\leftarrow$ Bayes' Rule
#
# * "Naive" Because the algorithm assumes all predictors are conditionally independent of other predictors, which is dumb
# * Uses the top half of Bayes rule and compares the two classes; Selects the larger value for classes
# * Probabilities tend towards 0 and 1 
#     * Principle components vote more than once, skewing the results
# * Used for spam classifiers 
# * Converges into Logistic Regression as the predictors become more and more independent
# * Gaussian Naive Bayes assumes underlying probabilities are normally distributed 
# * You can start with "priors" and weight for data that's skewed towards one classifier or the other 

# +
from sklearn.naive_bayes import GaussianNB
gaussian_NB = GaussianNB()
old_preds = gaussian_NB.fit(cancer_train[features].values, cell_class)
new_preds = gaussian_NB.predict(validation_data)
print "Predicts " + str(accuracy_score(validation_target, new_preds) * 100) + "% of unseen observations"

#print accuracy_score(cell_class, old_preds)
# -



StarcoderdataJupyterScriptsDedupFiltered