# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/standroidbeta/DS-Unit-1-Sprint-4-Statistical-Tests-and-Experiments/blob/master/DS-Unit-1-Sprint-Challenge-4/DS_Unit_1_Sprint_Challenge_4.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="NooAiTdnafkz" colab_type="text"
# # Data Science Unit 1 Sprint Challenge 4
#
# ## Exploring Data, Testing Hypotheses
#
# In this sprint challenge you will look at a dataset of people being approved or rejected for credit.
#
# https://archive.ics.uci.edu/ml/datasets/Credit+Approval
#
# Data Set Information: This file concerns credit card applications. All attribute names and values have been changed to meaningless symbols to protect confidentiality of the data. This dataset is interesting because there is a good mix of attributes -- continuous, nominal with small numbers of values, and nominal with larger numbers of values. There are also a few missing values.
#
# Attribute Information:
# - A1: b, a.
# - A2: continuous.
# - A3: continuous.
# - A4: u, y, l, t.
# - A5: g, p, gg.
# - A6: c, d, cc, i, j, k, m, r, q, w, x, e, aa, ff.
# - A7: v, h, bb, j, n, z, dd, ff, o.
# - A8: continuous.
# - A9: t, f.
# - A10: t, f.
# - A11: continuous.
# - A12: t, f.
# - A13: g, p, s.
# - A14: continuous.
# - A15: continuous.
# - A16: +,- (class attribute)
#
# Yes, most of that doesn't mean anything. A16 (the class attribute) is the most interesting, as it separates the 307 approved cases from the 383 rejected cases. The remaining variables have been obfuscated for privacy - a challenge you may have to deal with in your data science career.
#
# Sprint challenges are evaluated based on satisfactory completion of each part. It is suggested you work through it in order, getting each aspect reasonably working, before trying to deeply explore, iterate, or refine any given step. Once you get to the end, if you want to go back and improve things, go for it!

# + [markdown] id="5wch6ksCbJtZ" colab_type="text"
# ## Part 1 - Load and validate the data
#
# - Load the data as a `pandas` data frame.
# - Validate that it has the appropriate number of observations (you can check the raw file, and also read the dataset description from UCI).
# - UCI says there should be missing data - check, and if necessary change the data so pandas recognizes it as na
# - Make sure that the loaded features are of the types described above (continuous values should be treated as float), and correct as necessary
#
# This is review, but skills that you'll use at the start of any data exploration. Further, you may have to do some investigation to figure out which file to load from - that is part of the puzzle.

# + id="Q79xDLckzibS" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="282f9eb0-d5c7-4d26-966a-0a60fdf67142"
# TODO
import pandas as pd
from scipy import stats
import numpy as np

credit = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/credit-screening/crx.data', 
                     header=None, names=['A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7', 'A8','A9',
                                         'A10', 'A11', 'A12', 'A13', 'A14', 'A15', 'A16'])

credit.shape

# + id="axRR6rH7KMXI" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 87} outputId="605076a5-001d-40a4-f851-dc1828f0aa1f"
credit['A1'].value_counts()

# + id="6_fpiB_iHFLy" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 202} outputId="2db50cb1-29f1-47d2-c1e7-0889fa6ad9d1"
credit.head()

# + id="1bXMYdrpMdUK" colab_type="code" colab={}
credit = credit.replace({'?': np.nan, '+': 1, '-': 0})

# + id="Mwvm8EFtJMGU" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 329} outputId="f8f4c403-13b7-4998-92c7-9e9efe47db59"
credit.isnull().sum()

# + id="txOG27ifM_84" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 329} outputId="d98d16c1-13ec-42c2-cb7f-a8090dd63b10"
credit.dtypes

# + id="llHe8NosOmSM" colab_type="code" colab={}
credit['A2'] = credit['A2'].astype(float)
credit['A11'] = credit['A11'].astype(float)
credit['A14'] = credit['A14'].astype(float)
credit['A15'] = credit['A15'].astype(float)

# + id="Gzj9kKLmOzsH" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 312} outputId="f1bb1baf-cb96-493f-ba4f-93bba8960ce3"
credit.dtypes

# + id="yW6MEbwSQXBa" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 386} outputId="0491ec8d-2cf9-44ff-ab94-9ff5403608aa"
credit.head(11)

# + [markdown] id="G7rLytbrO38L" colab_type="text"
# ## Part 2 - Exploring data, Testing hypotheses
#
# The only thing we really know about this data is that A16 is the class label. Besides that, we have 6 continuous (float) features and 9 categorical features.
#
# 1. Explore the data: you can use whatever approach (tables, utility functions, visualizations) to get an impression of the distributions and relationships of the variables. In general, your goal is to understand how the features are different when grouped by the two class labels (`+` and `-`).
#
# 2. For the 6 continuous features, how are they different when split between the two class labels? Choose two features to run t-tests (again split by class label) - specifically, select one feature that is *extremely* different between the classes, and another feature that is notably less different (though perhaps still "statistically significantly" different). You may have to explore more than two features to do this.
#
# 3. For the categorical features, explore by creating "cross tabs" (aka [contingency tables](https://en.wikipedia.org/wiki/Contingency_table)) between them and the class label, and apply the Chi-squared test to them. [pandas.crosstab](http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.crosstab.html) can create contingency tables, and [scipy.stats.chi2_contingency](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2_contingency.html) can calculate the Chi-squared statistic for them.
#
# There are 9 categorical features - as with the t-test, try to find one where the Chi-squared test returns an extreme result (rejecting the null that the data are independent), and one where it is less extreme.
#
# **NOTE** - "less extreme" just means smaller test statistic/larger p-value. Even the least extreme differences may be strongly statistically significant.
#
# Your *main* goal is the hypothesis tests, so don't spend too much time on the exploration/visualization piece. That is just a means to an end - use simple visualizations, such as boxplots or a scatter matrix (both built in to pandas), to get a feel for the overall distribution of the variables.
#
# This is challenging, so manage your time and aim for a baseline of at least running two t-tests and two Chi-squared tests before polishing. And don't forget to answer the questions in part 3, even if your results in this part aren't what you want them to be.

# + id="_nqcgc0yzm68" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 171} outputId="aaa65dbe-5b26-42ce-dda5-e69d5ef1a844"
# TODO
credit.describe(exclude=np.number)

# + id="dpAfWISIYlwM" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 202} outputId="d01d1828-455a-4517-c6f5-be042286583e"
credit_1 =  credit[credit['A16'] == 1]



credit_1.head()

# + id="Sb_4VRrydXOw" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="39388527-8d7b-4040-bc42-c97ca6953ad8"
credit_1.shape

# + id="g1n2xBiSeucc" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="89285c64-d3b2-4d67-d5a2-2fff4d9ec9f4"
stats.ttest_1samp(credit_1['A2'], 0, nan_policy='omit')

# + id="cC2aPWcVgE-6" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 693} outputId="64797e90-0622-435e-de8f-cee7a2076361"
credit_1.apply(pd.Series.value_counts).T

# + id="XOCdc6A3hVDR" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="3091c26a-2b52-42e3-af0c-cb878210626c"
credit_1_mean = credit_1['A2'].mean()
print("Credit_1's Mean =", credit_1_mean)

# + id="VSJSUAF2ckf6" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 202} outputId="febbcdda-76dd-45be-85fd-75ec5141f009"
credit_0 = credit[credit['A16'] == 0]

credit_0.head()

# + id="N3IYeSw7fWoD" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="ba14468a-fb81-4122-bbf7-23cafa5031f1"
credit_0.shape

# + id="JcY7vbMHffo_" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="f395b17a-97c5-41c7-951a-133a2c1560cc"
stats.ttest_1samp(credit_0['A2'], 0, nan_policy='omit')

# + id="LDSoOnJBgQVq" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 693} outputId="e337bfdd-9e9d-4cd8-9ceb-d823e03a4544"
credit_0.apply(pd.Series.value_counts).T

# + id="fJYRzd8ch9V-" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="3e0c49cd-cce8-410e-d7a0-dbe2a5fdb344"
credit_0_mean = credit_0['A2'].mean()
print("Credit_0's Mean =", credit_0_mean)

# + id="KYgCyyliifiO" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="6e6f3ad1-b235-431c-93e7-dcba3fe90694"
# credit_0 and credit_1 2 sample T-test

stats.ttest_ind(credit_0['A2'], credit_1['A2'], nan_policy='omit')

# reject

# + id="C8_o56qalOFa" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 202} outputId="6fc410a2-966c-4d49-f03c-a4fd08b8afad"
cr_contingency_table =  pd.crosstab(credit['A13'], credit['A16'], margins=True)
cr_contingency_table

# + id="cx3U0kWP9vxx" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 1092} outputId="db4740ec-7908-4027-c9b9-ab84e7df08c5"
credit['A2'].sort_values()


# + id="EZ_QNzg088Nl" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 202} outputId="1f5264f5-63ad-4bd8-c71e-54324efc1975"
def process_age(credit):
  cut_points = [12,24,36,48,60,72,100]
  label_names = ["0-14","15-23","24-35","35-47","48-59","60-71, 72+"]
  credit['A2_age'] = pd.cut(credit['A2'],
                                           cut_points,labels=label_names)
  return credit

data = process_age(credit)
credit_approval_age = data[['A16', 'A2_age']]
credit_approval_age.head()

# + id="E_NL3LuzB8st" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 202} outputId="dfabe85b-2425-4697-ca21-e915d7a1aade"
credit_approval_age = credit_approval_age.rename(columns = {'A16':'Credit Approval', 'A2_age': 'Age'})

credit_approval_age.head()

# + id="rDhav9NICxjX" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 69} outputId="581fcb70-8eb0-44da-da04-e90ba9a3a57e"
credit_approval_age['Credit Approval'].value_counts()

# + id="wPV7LUYCDLPV" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 139} outputId="8a1a4715-d708-462f-c750-ed98d844fe77"
credit_approval_age['Age'].value_counts()

# + id="ssdrggVKrZ70" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 171} outputId="35271d74-fa1c-414a-a31a-e64ed2c3c43d"
cr_contingency =  pd.crosstab(credit_approval_age['Credit Approval'], 
                              credit_approval_age['Age'], margins=True)


cr_contingency

# + id="ONcuuk95WCDC" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 371} outputId="03ae6362-462f-47b2-e8f2-4fa2ca218edb"
import matplotlib.pyplot as plt

cr_contingency.plot.bar(stacked=True)
plt.legend(title='Credit Approval Across Ages');


# + id="ku1SP2j9uzxS" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="6df8e26f-558e-4302-d96c-7997b0f60fa4"
row_sums = cr_contingency.iloc[0:2,6].values
print(row_sums)

# + id="xypLx2jSvIik" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="5b0891b6-dae8-4e07-f514-96631c0ff451"
col_sums = cr_contingency.iloc[2,0:6].values
print(col_sums)

# + id="UpizWPK6tnTe" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="d8507a89-3ea2-4bda-9a40-174842ad1fd5"
total = cr_contingency_table.loc['All', 'All']
print("Total number of observations:", total)

expected = []
for i in range(len(row_sums)):
  expected_row = []
  for column in col_sums:
    expected_val = column*row_sums[i]/total
    expected_row.append(expected_val)
  expected.append(expected_row)
  
# print(np.array(expected))

# + id="N5Vn8zKhSDjg" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 52} outputId="456c0d82-a1a7-4f54-e1ad-4a8a302067bc"
cr_contingency =  pd.crosstab(credit_approval_age['Credit Approval'], 
                              credit_approval_age['Age'])


cr_values = cr_contingency.values

cr_values

# + id="ysNVLZ7Kp9i9" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="c386a545-782b-4581-9876-da34aa29e91c"
chi_squared = ((cr_values - expected)**2/(expected)).sum()
print(f"Chi-Squared: {chi_squared}")

# + colab_type="code" outputId="1b2a846b-c4ea-467c-9210-15b0ac5f7ea8" id="EkJermun3o7v" colab={"base_uri": "https://localhost:8080/", "height": 35}
dof = (len(row_sums)-1)*(len(col_sums)-1)
print(f"Degrees of Freedom: {dof}") 

# + id="TeUey24L3x9X" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="1f93ae34-73aa-47c9-8c42-c9b5a26c2a85"
p_value = stats.chi2.sf(chi_squared, dof)
print(f"P-value: {p_value}")

# + id="PzQSPqExy35r" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 156} outputId="8438c147-aa41-4c1b-9992-b91feaffa56d"
chi_squared, p_value, dof, expected = stats.chi2_contingency(cr_contingency)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}") 
print("Expected: \n", np.array(expected))

# + id="TsQwa7HXyxFd" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 35} outputId="04c6d7e7-b6c4-4122-d656-53e828c9a68a"
p_value = stats.chi2.sf(chi_squared, dof)
print(f"P-value: {p_value}")

# + [markdown] id="ZM8JckA2bgnp" colab_type="text"
# ## Part 3 - Analysis and Interpretation
#
# Now that you've looked at the data, answer the following questions:
#
# - Interpret and explain the two t-tests you ran - what do they tell you about the relationships between the continuous features you selected and the class labels?
# - Interpret and explain the two Chi-squared tests you ran - what do they tell you about the relationships between the categorical features you selected and the class labels?
# - What was the most challenging part of this sprint challenge?
#
# Answer with text, but feel free to intersperse example code/results or refer to it from earlier.

# + [markdown] id="LIozLDNG2Uhu" colab_type="text"
# *1. All of the T-tests came back as a rejection of the null hypothesis due to all of the P-values being <.05.* 
#
# *2. The Chi-squared test determined that there is a dependant relationship of the categorical values that I processed. *
#
# *3. Ensuring that all of the elements of the Sprint expectations were met.*

StarcoderdataJupyterScriptsDedupFiltered