# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/npgeorge/DS-Unit-1-Sprint-1-Dealing-With-Data/blob/master/Nicholas_George_Copy_of_LS_DS_113_Making_Data_backed_Assertions_Assignment.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="Okfr_uhwhS1X" colab_type="text"
# # Lambda School Data Science - Making Data-backed Assertions
#
# This is, for many, the main point of data science - to create and support reasoned arguments based on evidence. It's not a topic to master in a day, but it is worth some focused time thinking about and structuring your approach to it.

# + [markdown] id="lOqaPds9huME" colab_type="text"
# ## Assignment - what's going on here?
#
# Consider the data in `persons.csv` (already prepared for you, in the repo for the week). It has four columns - a unique id, followed by age (in years), weight (in lbs), and exercise time (in minutes/week) of 1200 (hypothetical) people.
#
# Try to figure out which variables are possibly related to each other, and which may be confounding relationships.
#
# Try and isolate the main relationships and then communicate them using crosstabs and graphs. Share any cool graphs that you make with the rest of the class in Slack!

# + id="F1_ORL6jt9Nj" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 105} outputId="a7e89095-0dc9-467c-e4cb-3711a279e31b"
# !pip install pandas==0.23.4
#make sure to "restart runtime" to ensure this will run, even if it install and says its satisfied. Runtime-->Restart Runtime and it will work
#loading this at the top to ensure it works anytime i want to take a look at the crosstabs

# + id="TGUS79cOhPWj" colab_type="code" outputId="4b60cb4a-7b93-4e2a-9fd5-0dbe084401c5" colab={"base_uri": "https://localhost:8080/", "height": 197}
# TODO - your code here
# Use what we did live in lecture as an example

# HINT - you can find the raw URL on GitHub and potentially use that
# to load the data with read_csv, or you can upload it yourself

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.read_csv('https://raw.githubusercontent.com/LambdaSchool/DS-Unit-1-Sprint-1-Dealing-With-Data/master/module3-databackedassertions/persons.csv')

df.head()

# + id="NmLiFz7Azorp" colab_type="code" outputId="c9d74eb8-730d-471f-abc2-0bb3825f10c0" colab={"base_uri": "https://localhost:8080/", "height": 34}
df.shape

# + id="UxqlEKzDzr_Z" colab_type="code" outputId="7bb89c2c-53db-46f4-86a5-10c02244acb3" colab={"base_uri": "https://localhost:8080/", "height": 105}
df.dtypes

# + id="Cg4DUl_Iz2RV" colab_type="code" outputId="c79dcc92-4da1-4476-b643-34a2a392620b" colab={"base_uri": "https://localhost:8080/", "height": 287}
#all integers so lets check some summary statistics
df.describe()

# + id="kvde5uS90DtN" colab_type="code" outputId="0562eb10-969e-4fd8-810f-4f6a3f0ba793" colab={"base_uri": "https://localhost:8080/", "height": 283}
df.plot(x='age', y='weight', kind='scatter') ;

# + id="LDlMmJx_1LNW" colab_type="code" outputId="39dccae7-ee7b-4220-d7e7-23e9665fb58c" colab={"base_uri": "https://localhost:8080/", "height": 284}
df.plot(x='exercise_time', y='weight', kind='scatter', color='r') ;

# + id="x9EAEp12199g" colab_type="code" outputId="d3e334a3-4ffd-438c-f632-b153d52b9683" colab={"base_uri": "https://localhost:8080/", "height": 284}
df.plot(x='exercise_time', y='age', kind='scatter', color='r') ;

# + id="7GuNkTPOGbBb" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 257} outputId="29517807-24b1-4665-aa6e-eb432f0f3889"
df.plot.hexbin(x='exercise_time', y='age', gridsize=20); #trying out different plots


# + id="9x6t6c0p5C3G" colab_type="code" colab={}
#both weight and age are related to exercise time, thus making exercise time the confounding variable

# + id="loAhA93nrPck" colab_type="code" outputId="a84a8c14-e7bd-4844-ba04-d9f8f82dcfd4" colab={"base_uri": "https://localhost:8080/", "height": 227}
#bin the weights, and ages using the crosstab to see if you can pull a more meaningful relationship
#need to bin all three of the parameters
#it makes sense to bin all the variables so you only have a few categories

#pd.crosstab(df['age'], df['exercise_time'])
exercise_time_bins = pd.cut(df['exercise_time'], 5) 
age_bins = pd.cut(df['age'], 5)
weight_bins = pd.cut(df['weight'],5)

#pd.crosstab(df['exercise_time'], exercise_time_bins, normalize='columns')
#pd.crosstab(df['age'], age_bins, normalize='columns')
#pd.crosstab(df['weight'], weight_bins, normalize='columns')

#pd.crosstab(exercise_time_bins, df['age'])
pd.crosstab(exercise_time_bins, age_bins, normalize='columns') #still gives error after downgrading to older version
#print('This is a better way to split the data up. We can see five different age groups and the bins of the percentage of exercise time for each age group.')


# + id="j7ovfJPZsm7y" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 227} outputId="9f593f97-ea3f-4b06-a05f-74b85f54058b"
pd.crosstab(exercise_time_bins, weight_bins, normalize='columns')

# + id="5DVDA3O2sm3o" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 227} outputId="90fdf9bd-466b-4c76-b205-b969be3e4e3f"
wa = pd.crosstab(weight_bins, age_bins, normalize='columns')
wa

# + id="m5GinvL0smu4" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 227} outputId="22e0d82b-569a-4e64-f19c-fad9b07af697"
wa_subset = wa.iloc[:, [0,1,2]] #the ":" is specifying ALL rows, and the list is specifying which columns
wa_subset

# + id="L58voteICd3H" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 273} outputId="b9daa0af-4120-476b-e24f-3c7d6f414f87"
wa_subset.plot();

# + id="kJUAWg-gC_QC" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 227} outputId="7f6e1043-31e9-462d-c07b-aeb956fe2544"
a_t = pd.crosstab(exercise_time_bins, age_bins, normalize='columns')
a_t

# + id="k4BZS0fJDoJ1" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 344} outputId="ffea2150-b30c-438f-f2e9-9aa647f73031"
#plotting the age bins to exercise time. This shows some interesting relationships. You can clearly see the higher age bracket drops off considerably
a_t.plot(kind='bar',style='ggplot');

# + id="YctPD_xxJSwc" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 453} outputId="033b995b-afae-4593-e4b1-bdad1273b5b8"
a_t.plot(kind='bar',stacked=True, figsize=(8, 6));

# + id="cDtTPphLDO50" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 227} outputId="7ab6c566-5289-4e30-9b64-959d446048cd"
#using the iloc feature to segment the data to visualize it better
a_t_subset = a_t.iloc[:,[0,1,2]]
a_t_subset

# + id="v3-XW9yMDcRm" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 344} outputId="a22b8305-bc7a-467c-c366-cba278e0ca2d"
a_t_subset.plot(kind='bar'); #playing with different age groups

# + [markdown] id="BT9gdS7viJZa" colab_type="text"
# ### Assignment questions
#
# After you've worked on some code, answer the following questions in this text block:
#
# 1.  What are the variable types in the data?
# 2.  What are the relationships between the variables?
# 3.  Which relationships are "real", and which spurious?
#

# + [markdown] id="nhmrJGapoM7k" colab_type="text"
# 1. The variable types in the data are all integers.
#
# numerical --> ordinal - set order, discrete - uses whole numbers like age
# categorical -->

# + [markdown] id="syCMCXoJoSi9" colab_type="text"
# 2. The relationships between the variables lead to a conclusion that both weight and age are related to exercise time, making exercise time the confounding variable and obfuscating the possible relationship between age and weight. One possible conclusion you can draw from a persons age as it relates to their weight is that it is evenly distributed.  

# + [markdown] id="qkieeiiUpisJ" colab_type="text"
# 3.
# Real: 
# *   The amount of time spent exercising as a person ages starts to drop considerably after age 60.
# *   The amount of time spent exercising as a person grows past a weight of around 150-160 lbs drops considerably as well.
#
# Spurious:
# *   The relationship between age and weight in this data set.
#
#

# + [markdown] id="_XXg2crAipwP" colab_type="text"
# ## Stretch goals and resources
#
# Following are *optional* things for you to take a look at. Focus on the above assignment first, and make sure to commit and push your changes to GitHub.
#
# - [Spurious Correlations](http://tylervigen.com/spurious-correlations)
# - [NIH on controlling for confounding variables](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4017459/)
#
# Stretch goals:
#
# - Produce your own plot inspired by the Spurious Correlation visualizations (and consider writing a blog post about it - both the content and how you made it)
# - Pick one of the techniques that NIH highlights for confounding variables - we'll be going into many of them later, but see if you can find which Python modules may help (hint - check scikit-learn)

# + id="MDwqeoIdxlSv" colab_type="code" colab={}
#reminder after looking at this data: correlation does not equal causation

StarcoderdataJupyterScriptsDedupFiltered