# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] colab_type="text" id="egXb7YpqEcZF"
# ## Congressional Voting Assignment - Apply the t-test to real data
#
# Your assignment is to determine which issues have "statistically significant" differences between political parties in this [1980s congressional voting data](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records). The data consists of 435 instances (one for each congressperson), a class (democrat or republican), and 16 binary attributes (yes or no for voting for or against certain issues). Be aware - there are missing values!
#
# Your goals:
#
# 1. Load and clean the data (or determine the best method to drop observations when running tests)
# 2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.01
# 3. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.01
# 4. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)
#
# Note that this data will involve *2 sample* t-tests, because you're comparing averages across two groups (republicans and democrats) rather than a single group against a null hypothesis.

# + colab={"base_uri": "https://localhost:8080/", "height": 263} colab_type="code" id="nstrmCG-Ecyk" outputId="e24f5399-69a2-42c8-dba9-f9cffb2dd834"
# Grab the file from UCI
# !wget https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data

# + colab={} colab_type="code" id="rBdcYs-o9MH7"
# Imports
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import seaborn as sns

# + colab={"base_uri": "https://localhost:8080/", "height": 280} colab_type="code" id="ksR75_YZ9ZR4" outputId="acb9a84f-468d-49db-d13a-e857df6724bc"
# Load Data
df = pd.read_csv('house-votes-84.data', 
                 header=None,
                 names=['party','handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa'])
print(df.shape)
df.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 263} colab_type="code" id="C5VUpcut9f2b" outputId="9815c1bf-be46-4c54-c76c-04c2c3f98da9"
# Replace '?' with np.NaN, 'n' with 0, and 'y' with 1
df = df.replace({'?':np.NaN, 'n':0, 'y':1})
df.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 323} colab_type="code" id="tcHR0EoV_9rO" outputId="69da2536-eed9-4d04-f1bd-16920795efce"
# How many abstentions? (NaNs)
df.isnull().sum()

# + colab={"base_uri": "https://localhost:8080/", "height": 280} colab_type="code" id="RWfKVbSvArDy" outputId="4e792a47-10de-44cf-d586-e26a2bcdbc55"
# Create Republicans Dataframe
rep = df[df.party == "republican"]
print(rep.shape)
rep.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 280} colab_type="code" id="wvS9kXRDA4Rm" outputId="48d1b509-4a5d-44b2-a1ba-8bbc4e0d2026"
# Create Democrats Dataframe
dem = df[df.party == "democrat"]
print(dem.shape)
dem.head()

# + [markdown] colab_type="text" id="-TbWrjOsXVPg"
# # 1 Sample T-Tests
#
# In a 1-sample T-test we are testing the mean of one sample against a null hypothesis of our choosing.
#
# The null hypothesis that we designate depends on how we have encoded our data and the kind of questions that we want to test. 
#
# If I have encoded votes as 0 for no and 1 for yes, I want to test Democratic support for an issue, and I use a null hypothesis of 0, then I am comparing Democrat voting support against a null hypothesis of no Democrat support at all for a given issue.
#
# If I use a null hypothesis of .5 then I am comparing the democrat voting support against a null hypothesis of democrats being neither in favor or against a particular issue. 
#
#
# If I use a null hypothesis of 1 then I am comparing the democrat voting support against a null hypothesis of all democrats being favor of a particular issue.
#
# Lets use the 0 and .5 null-hypotheses to test the significance of those particular claims. They're all valid questions to be asking, they're just posing a slightly different question --testing something different.

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="tMOWdS-jXY8s" outputId="f6b3a5a3-8217-468d-94d8-45c913ad7cc6"
# Lets test this out on the handicapped-infants issue since it's the first one.
# I am just going to omit NaN values from my tests.
# Null Hypothesis that Democratic support is 0.
stats.ttest_1samp(dem['handicapped-infants'], 0, nan_policy='omit')

# + [markdown] colab_type="text" id="gN5r_MUTceNM"
# Given the results of the above test I would REJECT the null hypothesis that there is no Democrat support for the handicapped-infants bill at the 95% significance level.
#
# In Layman's terms It would be a FALSE statment to declare that there is no democratic support for this bill. That's something that you might hear a political pundit declare, but you'll notice that they don't report their alpha value or p-value when they make such claims, they just spew them. --*Tell us how you really feel!*

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="4qqZcb04b4ec" outputId="5246b6de-fb2e-47a9-b98a-b9a85a3df472"
# Null Hypothesis that Democrats neither support not oppose the issue
stats.ttest_1samp(dem['handicapped-infants'], .5, nan_policy='omit')

# + colab={"base_uri": "https://localhost:8080/", "height": 638} colab_type="code" id="1CBhze5ycmcX" outputId="91c98a3f-b41c-4504-c8b1-4db25ee15e58"
# Look at vote counts by party and by issue
# It's very easy to perform some Interocular Traumatic Tests (IOT) on this data
# https://www.r-bloggers.com/inter-ocular-trauma-test/
# We can eyeball the outcomes of some of these before we perform any T-tests.
# But which of the differences is statistically significant?
dem.apply(pd.Series.value_counts).T

# + [markdown] colab_type="text" id="Wnvni1GGe-Ah"
# As we look at the above table we see our findings corroborated in the raw voting counts of democrats. They are definitely not all against the handicapped-infants bill hence the very high p-value. We also see that democrats on this issue are not evenly split but that there is a significant margin of support for the handicapped-infants bill. However this margin is not as extreme for our second hypothesis as for our first, and hence our secont 1-sample t-test has a lower t statistic and a higher p-value although it is still significant. 

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="0EtYrjgUfuvc" outputId="626dcf92-7de1-4639-d3f2-97ebec089aea"
# Null hypothesis that there is no democratic support for the bill.
stats.ttest_1samp(dem['physician-fee-freeze'], 0, nan_policy='omit')

# + [markdown] colab_type="text" id="Rf6QmMw1gMT0"
# We see that even though this issue has the most extreme Democrat opposition, we cannot conclude that there is no Democrat support for this issue. We will again REJECT the null hypothesis that the mean of democratic support is 0 --or that there is no Democrat support for this issue. 

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="237TY_jTf3mj" outputId="2067fa70-50fd-4170-90fc-270c962526d1"
# Null hypothesis that there is even support for the bill among democrats.
stats.ttest_1samp(dem['physician-fee-freeze'], .5, nan_policy='omit')

# + [markdown] colab_type="text" id="c9rR-KJsglzG"
# Here we are again testing the null hypothesis of Democrats being neither for nor against the issue, but being split in their voting. This time we see a strong negative t-statistic and low p-value. The negative sign on this t-statistic suggests that democratic support is much further to the left of .5 (our null hypothesis of even yes/no voting in the party) in other words, this t-statistic says that no only are democrats not split on the issue, they are highly against this policy. Again this is very clear to us as we consult the output of raw vote counts for the democrats. 
#
# As a final test and example of 1-sample T-tests lets try and get something that's not statistically significant. Lets test our split-support hypothesis on the "synfuels" policy. 

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="eq7X8gdRhxRM" outputId="7a1e07be-f413-4e36-ca96-b168b6c02a57"
stats.ttest_1samp(dem['synfuels'], .5, nan_policy='omit')

# + [markdown] colab_type="text" id="2CGe5scFiRoh"
# Here we FAIL TO REJECT the null hypothesis that democrats voting is even on this issue because our t-statistic is small resulting in a p-value greater than .05. So we would fail to reject this null hypothesis at the 95% level. Remember that we never "accept" the null hypothesis, we only "fail to reject" it. We're not claiming that the null hypothesis is true, we're just stating that our test doesn't have enough statistical power to show otherwise.  

# + [markdown] colab_type="text" id="o2AtRg6di7wk"
# ### We could do the same thing with the Republicans dataframe, but the result would be extremely similar.

# + colab={"base_uri": "https://localhost:8080/", "height": 635} colab_type="code" id="gruyk8jojFXp" outputId="565b0519-10d1-44e7-8ef9-9026ed92bf31"
# Look at republican voting patterns
rep.apply(pd.Series.value_counts).T

# + [markdown] colab_type="text" id="VtwHw3c-jN88"
# ## What if we didn't split our dataframe up by Republicans and Democrats, then what would that be testing?
#
# The contents of the overall dataframe that we're working with determines the GENERALIZABILITY of our results. If we're running tests on Democrat voting behavior then our hypothesis tests can only make claims about Democrat voting, they say nothing about Republican support or opposition for an issue, we would have to run those tests on the Republican dataframe.
#
# But then what do T-tests on the entire dataframe of both Republicans and Democrats say? They're testing the same thing but generalized to all congresspeople and the results are not specific to one party or another. 

# + colab={"base_uri": "https://localhost:8080/", "height": 635} colab_type="code" id="Z6CU5zb5jZYg" outputId="629d49ae-a36b-421d-f9a9-e82d8c40b3df"
# Look at congressional voting patterns
df.apply(pd.Series.value_counts).T

# + [markdown] colab_type="text" id="4C8o5hBYmCDq"
# We'll give one example of this, but it's the same as the T-tests above, just with the context being congress as a whole rather than a specific party.

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="FaMnflMtlg-M" outputId="f949fc3b-e3c8-45ec-c124-2f00fedc5cf1"
stats.ttest_1samp(df['mx-missile'], .5, nan_policy='omit')

# + [markdown] colab_type="text" id="wnxJUIvllxCL"
# We FAIL TO REJECT the null hypothesis that congressional support for the 'mx-missile' policy is split (even) among congresspeople. 

# + [markdown] colab_type="text" id="4jpBRb2Vmexi"
# # 2-sample T-tests
#
# Two-sample T-tests are very similar to 1-sample t-tests, except that instead of providing a raw value as a null hypothesis, we will be comparing the mean of a second sample as the alternate hypothesis. 

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="uowS49VLmu6W" outputId="fd6aeca0-e683-4036-b90a-d181715fc050"
stats.ttest_ind(dem['mx-missile'], rep['mx-missile'],nan_policy='omit')

# + [markdown] colab_type="text" id="Vm8V54Zqnp4Y"
# ## Don't make this mistake!
#
# Notice that a 2-sample T-test does not give the same results as a 1-sample t-test where the null hypothesis is the mean of a second sample. The test below is representative of a t-test comparing the mean of the first sample to a null-hypothesis value of .115, but it is not representative of comparing the mean of the sample to the mean of a second sample. This is because passing in the mean as a single value does not account for the variance around the mean of the second sample, so the 1-sample t-test against the mean value of another sample has a much higher t-statistic and significance than a test where you pass in the full data of both samples. 
#
# You can avoid this mistake by making sure that you provide a raw null-hypothesis value (like 0, .5, 1) as the null-hypothesis value when performing a 1-sample t-test, and only two separate samples in their entirety to the function when performing a two-sample t-test.

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="f7oZcq7wlvgq" outputId="43318bfd-6df4-4900-da4a-63e9c95d1f26"
rep_mx_missile_mean = rep['mx-missile'].mean()
print(rep_mx_missile_mean)

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="ZSpJva95nHEc" outputId="450eed41-e050-4cfe-ed09-3ebcca031d54"
stats.ttest_1samp(dem['mx-missile'], rep_mx_missile_mean, nan_policy='omit')

# + [markdown] colab_type="text" id="5vIu8rIKr2BS"
# Also notice that the order in which you pass the two samples to the 2-sample test will reverse the direction of the t-statistic.
#
# A t-statistic with a positive sign indicates that the first sample mean is higher than the second sample mean, and the significance level indicates whether or not the means are different at the 95% confidence level.

# + colab={"base_uri": "https://localhost:8080/", "height": 67} colab_type="code" id="WGn7p3zZr-Uv" outputId="4ef32bcd-f169-4721-ede9-6439631c9c6b"
print(dem['mx-missile'].mean())
print(rep['mx-missile'].mean())

stats.ttest_ind(dem['mx-missile'], rep['mx-missile'], nan_policy='omit')

# + [markdown] colab_type="text" id="XNw-CZod7hmF"
# Here we have reversed the order of which sample we pass in first to demonstrate how `ttest_ind` might result in a negative t-statistic.
#

# + colab={"base_uri": "https://localhost:8080/", "height": 67} colab_type="code" id="kvaZq-qesODH" outputId="16bb8eff-8301-4cf4-87cd-9f0bdc73a1ee"
print(dem['mx-missile'].mean())
print(rep['mx-missile'].mean())

stats.ttest_ind(rep['mx-missile'], dem['mx-missile'], nan_policy='omit')

# + [markdown] colab_type="text" id="1DZ3K_x5sboG"
# Because of this, in order to ensure consistency of the signs of the t-statistic during 2-sample tests, I suggest sticking with passing in one party as the first argument and the other party as the second argument and sticking with that pattern throughout your testing so that you don't confuse yourself.

# + [markdown] colab_type="text" id="YOJ5ahfgpcRY"
# ## Two-sample T-tests for Democrat Support, Republican Support, and no singnificant difference in support between parties:
#

# + [markdown] colab_type="text" id="K74YMzbIqD4h"
# ## Significant Democrat Support

# + [markdown] colab_type="text" id="3ObXLZZt6bXZ"
# The outcome of this tests indicates that we should reject the null hypothesis that the mean of Democrat votes is equal to the mean of Republican votes for this issue. We would conclude due to the positive t-statistic here that Democrat support is significantly higher than republican support for this issue.

# + colab={"base_uri": "https://localhost:8080/", "height": 67} colab_type="code" id="gG06dA3XqDAx" outputId="66ba07be-e6bb-4002-8b06-471c6e10e2fb"
stats.ttest_ind(dem['synfuels'], rep['synfuels'], nan_policy='omit')

# + [markdown] colab_type="text" id="OfhdLA3ZqGcE"
# ## Significant Republican Support

# + [markdown] colab_type="text" id="HpiuhNS06pbH"
# The outcome of this tests indicates that we should reject the null hypothesis that the mean of democrat votes is equal to the mean of republican votes for this issue. We would conclude due to the positive t-statistic here that Republican support is significantly higher than Democrat support for this issue.

# + colab={"base_uri": "https://localhost:8080/", "height": 67} colab_type="code" id="esSjOulYqCh3" outputId="eee6357f-d8a5-4d81-d6b8-3ff9a5e34cb9"
print(dem['south-africa'].mean())
print(rep['south-africa'].mean())


stats.ttest_ind(rep['south-africa'], dem['south-africa'], nan_policy='omit')

# + [markdown] colab_type="text" id="zN91yLt9qJVq"
# ## No Significant Difference in Support
#
# Due to the insignificant p-value we would FAIL TO REJECT the null hypothesis that the means of Democrat support and Republican support are equal. This means that regardless of the sign on the t-statistic, we cannot make statistically significant claims about the difference between Republican and Democrat support of this issue.

# + colab={"base_uri": "https://localhost:8080/", "height": 67} colab_type="code" id="lvRgxQZ4nOpE" outputId="707bd180-f398-4d1b-9dc7-0c131a17560e"
print(dem['water-project'].mean())
print(rep['water-project'].mean())

stats.ttest_ind(dem['water-project'], rep['water-project'], nan_policy='omit')

# + [markdown] colab_type="text" id="GtrGLl8u-NMB"
# # $\chi^2$ Hypothesis Tests
#
# Chi-square hypothesis tests can only be performed on categorical data to test independence (lack of association) between two variables. 
#
# You can tabulate the different cross-sections of your categorical data by creating a "contingency table" also known as a "cross-tab". If we see substantial differences between categories in the cross-tab, this might indicate a dependence between variables (possibly indicating some level of correlation although correlation and dependence are not perfectly synonymous),however, we must perform a chi-square test in order to test this hypothesis. 
#
# Correlation implies dependence (association), but dependence does not necessarily imply correlation. (Although it is correlated with correlation - you see what I did there?)

# + [markdown] colab_type="text" id="BRGcpta8_Fvm"
# ## First, a Numpy vs Scipy implementation
#
# Lets demonstrate a chi-squared test on the "Adult" dataset (1994 census data) from UCI.
#
# Lets compare gender and binned working hours per week (both are categorical).
#
# <http://archive.ics.uci.edu/ml/datasets/adult>

# + colab={"base_uri": "https://localhost:8080/", "height": 365} colab_type="code" id="1QXjZvRREDoJ" outputId="34e0b88b-b54a-4c99-c7d2-f0c3ef08e0fe"
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/datasets/master/adult.csv')
print(df.shape)
df.head()


# + [markdown] colab_type="text" id="p2jshGRLCE2J"
# Notice that for chi-squared tests, I will not have to categorically encode my data since I will perform my calculations on the contingency table (cross-tab) and not on the raw dataframe.

# + colab={"base_uri": "https://localhost:8080/", "height": 204} colab_type="code" id="7X-ZV8kWIRlI" outputId="5232f287-c060-440e-acb4-37de78207c7f"
def process_hours(df):
  cut_points = [0,9,19,29,39,49,1000]
  label_names = ["0-9","10-19","20-29","30-39","40-49","50+"]
  df["hours_per_week_categories"] = pd.cut(df["hours-per-week"],
                                           cut_points,labels=label_names)
  return df

data = process_hours(df)
workhour_by_sex = data[['sex', 'hours_per_week_categories']]
workhour_by_sex.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 68} colab_type="code" id="UWPW5V7pInq8" outputId="3055aac2-11a7-47c6-f910-9a88b6edcf51"
workhour_by_sex['sex'].value_counts()

# + colab={"base_uri": "https://localhost:8080/", "height": 136} colab_type="code" id="9PkgOVwSIute" outputId="e33efd22-0664-400b-9ea0-8d94a8d670c8"
workhour_by_sex['hours_per_week_categories'].value_counts()

# + [markdown] colab_type="text" id="Ubv7UOknCq6-"
# Before we calculate our contingency table, lets make very clear what our null and alternative hypotheses are in this situation. 
#
# $H_{0}$ : There is *no* statistically significant relationship between gender and working hours per week.
#
# $H_{a}$ : There *is* a statistically significant relationship between gender and working hours per week.

# + colab={"base_uri": "https://localhost:8080/", "height": 173} colab_type="code" id="94QdK0noI7fV" outputId="437dc3e5-e31d-4bf4-f476-971faec1ff0d"
# Calculate our contingency table with margins
contingency_table = pd.crosstab(
    workhour_by_sex['sex'],
    workhour_by_sex['hours_per_week_categories'],
    margins = True)

contingency_table

# + [markdown] colab_type="text" id="NJaC1rN0DVNa"
# Using the contingency table with margins included will make our from-scratch implementation a little bit easier.
#
# Our code would be more reusable if we calculated totals as we went instead of pulling them from the table, but I wanted to be really clear about what they represented, so we're going to grab the row and column totals from the margins of the contingency table directly.
#

# + [markdown] colab_type="text" id="h1wRLvskEEpW"
# ### Visualizing preferences with a stacked bar-chart.

# + colab={"base_uri": "https://localhost:8080/", "height": 350} colab_type="code" id="YEzlcsPeDLUx" outputId="94988d11-1f93-47cd-de5d-fdc8167be348"
#Assigns the frequency values
malecount = contingency_table.iloc[0][0:6].values
femalecount = contingency_table.iloc[1][0:6].values

#Plots the bar chart
fig = plt.figure(figsize=(10, 5))
sns.set(font_scale=1.8)
categories = ["0-9","10-19","20-29","30-39","40-49","50+"]
p1 = plt.bar(categories, malecount, 0.55, color='#d62728')
p2 = plt.bar(categories, femalecount, 0.55, bottom=malecount)
plt.legend((p2[0], p1[0]), ('Male', 'Female'))
plt.xlabel('Hours per Week Worked')
plt.ylabel('Count')
plt.show()

# + [markdown] colab_type="text" id="yI7mDQz5Zicb"
# It's harder to eyeball if these variables might be dependent. I would look at the ratios between 30-39, 40-49 and 50+. As we do this we see that males working 40-49 hours per week experiences something like 5x jump from the amount working 30-39 whereas for women it's something like a 2.5x jump. Similarly for men we see a decrease to about 1/3 the size as we move from >40 to the >50 category, but for women the final category is about 1/5 the size. This suggests to me that gender and working hours are not independent. Hopefully this passes the smell test for you as it matches your intuition that has been trained from a lifetime of training data. 

# + [markdown] colab_type="text" id="17rpjP9MF8QG"
# ### Expected Value Calculation
# \begin{align}
# expected_{i,j} =\frac{(row_{i} \text{total})(column_{j} \text{total}) }{(\text{total observations})}  
# \end{align}

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="5qJUDgLhJR5Q" outputId="0c1cad1d-5e49-4401-9910-0eb8b69eace4"
# Get Row Sums
row_sums = contingency_table.iloc[0:2,6].values
print(row_sums)

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="PcR_YTU3KhmH" outputId="3c4dd004-d3af-4fa3-a7d0-89ebfd7e53d7"
# Get Column Sums
col_sums = contingency_table.iloc[2,0:6].values
print(col_sums)

# + colab={"base_uri": "https://localhost:8080/", "height": 101} colab_type="code" id="7NhHCkSJMT_p" outputId="501c19c9-e8c9-40d8-cda9-9b18514c804a"
# Calculate Expected Values for each cell
total = contingency_table.loc['All', 'All']
print("Total number of observations:", total)

expected = []
for i in range(len(row_sums)):
  expected_row = []
  for column in col_sums:
    expected_val = column*row_sums[i]/total
    expected_row.append(expected_val)
  expected.append(expected_row)
  
print(np.array(expected))

# + [markdown] colab_type="text" id="5l656fkhFHMI"
# ## Chi-Squared Statistic with Numpy
#
# \begin{align}
# \chi^2 = \sum \frac{(observed_{i}-expected_{i})^2}{(expected_{i})}
# \end{align}
#
# For the $observed$ values we will just use a version of our contingency table without the margins as a numpy array. In this way, if our observed values array and our expected values array are the same shape, then we can subtract them and divide them directly which makes the calculations a lot cleaner. No for loops!

# + colab={"base_uri": "https://localhost:8080/", "height": 67} colab_type="code" id="NDr72RcGI0kE" outputId="4023dbfd-a5a2-42cb-8e9e-f409ff17f0d0"
# Get contingency table without margins
contingency = pd.crosstab(workhour_by_sex['sex'], 
                          workhour_by_sex['hours_per_week_categories'])
contingency = contingency.values

print(contingency.shape)
print(contingency)

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="62rXAnuhM7xs" outputId="a3595c18-9d0c-4f1a-8aa9-ff27e7877d4a"
chi_squared = ((contingency - expected)**2/(expected)).sum()
print(f"Chi-Squared: {chi_squared}")

# + [markdown] colab_type="text" id="kJQHlyZVc_PX"
# ### Degrees of Freedom
#
# \begin{align}
# DoF = (\text{Number of Rows} -1)\times(\text{Number of Columns}-1)
# \end{align}

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="ZUdSOrL3PaPe" outputId="d04dff2b-431d-45bd-d28f-4955f23333ce"
dof = (len(row_sums)-1)*(len(col_sums)-1)
print(f"Degrees of Freedom: {dof}")                        

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="Hv0dIDbDQI0b" outputId="687e95db-986b-4347-dda5-d917d9173c75"
# Calculate the p-value from the chi_squared and dof
p_value = stats.chi2.sf(chi_squared, dof)
print(f"P-value: {p_value}")

# + colab={"base_uri": "https://localhost:8080/", "height": 151} colab_type="code" id="yRy-4OpZQmNe" outputId="bd3216ed-407d-4779-ff1a-aa6176ff06ab"
chi_squared, p_value, dof, expected = stats.chi2_contingency(contingency)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}") 
print("Expected: \n", np.array(expected))

# + [markdown] colab_type="text" id="NG89JCDLURyT"
# # Can we perform a Chi2 test on our congressional voting data?
#
# Is it categorical? Then yes. Lets do it! 
#
# Are political party and voting behavior on the "budget" independent? Lets test it!

# + colab={"base_uri": "https://localhost:8080/", "height": 266} colab_type="code" id="BJFHyjnDV0c-" outputId="4ee91f9c-296d-456e-aeed-9f1e200f3c0b"
# Load the data again to be safe:
df = pd.read_csv('house-votes-84.data', 
                 header=None,
                 names=['party','handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa'])
df = df.replace({'?':np.NaN, 'n':0, 'y':1})
print(df.shape)
df.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 136} colab_type="code" id="ZtT_nrl_VoJV" outputId="21928d83-e4a9-474e-ce45-a04bafc86a51"
contingency_table =  pd.crosstab(df['party'], df['budget'])
contingency_table

# + colab={"base_uri": "https://localhost:8080/", "height": 118} colab_type="code" id="LkdayAPVWSA-" outputId="676150be-c947-41ea-f41b-1fb5c6c8fe10"
chi_squared, p_value, dof, expected = stats.chi2_contingency(contingency_table)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}") 
print("Expected: \n", np.array(expected))

# + colab={} colab_type="code" id="QIq2MFTp3bxJ"
# stats.chi2.sf?

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="UgEck9IDWnYC" outputId="db592848-d7f4-424d-b85d-2edf35b3f043"
# Calculate the p-value from the chi_squared and dof
p_value = stats.chi2.sf(chi_squared, dof)
print(f"P-value: {p_value}")

# + [markdown] colab_type="text" id="_dwKXzZIk1WS"
# We REJECT the null hypothesis that political party and voting on the budget are independent. They must be associated. Which hopefully also passes the smell test. 

StarcoderdataJupyterScriptsDedupFiltered