# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/Scott-Huston/DS-Unit-1-Sprint-1-Dealing-With-Data/blob/master/LS_DS_112_Loading_Data.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="-c0vWATuQ_Dn" colab_type="text"
# # Lambda School Data Science - Loading, Cleaning and Visualizing Data
#
# Objectives for today:
# - Load data from multiple sources into a Python notebook 
#  - !curl method
#  - CSV upload method
# - Create basic plots appropriate for different data types
#  - Scatter Plot
#  - Histogram
#  - Density Plot
#  - Pairplot
# - "Clean" a dataset using common Python libraries
#  - Removing NaN values "Interpolation"

# + [markdown] id="grUNOP8RwWWt" colab_type="text"
# # Part 1 - Loading Data
#
# Data comes in many shapes and sizes - we'll start by loading tabular data, usually in csv format.
#
# Data set sources:
#
# - https://archive.ics.uci.edu/ml/datasets.html
# - https://github.com/awesomedata/awesome-public-datasets
# - https://registry.opendata.aws/ (beyond scope for now, but good to be aware of)
#
# Let's start with an example - [data about flags](https://archive.ics.uci.edu/ml/datasets/Flags).

# + [markdown] id="wxxBTeHUYs5a" colab_type="text"
# ## Lecture example - flag data

# + id="nc-iamjyRWwe" colab_type="code" outputId="bae43e1c-3cdb-4ab7-fefd-0d7942470b1b" colab={"base_uri": "https://localhost:8080/", "height": 1000}
# Step 1 - find the actual file to download

# From navigating the page, clicking "Data Folder"
flag_data_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data'

# You can "shell out" in a notebook for more powerful tools
# https://jakevdp.github.io/PythonDataScienceHandbook/01.05-ipython-and-shell-commands.html

# Funny extension, but on inspection looks like a csv
# !curl https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data

# Extensions are just a norm! You have to inspect to be sure what something is

# + id="UKfOq1tlUvbZ" colab_type="code" colab={}
# Step 2 - load the data

# How to deal with a csv? 🐼
import pandas as pd
column_headers = #finish this
flag_data = pd.read_csv(flag_data_url, header = None, names = column_headers)



# + id="uXotUzMTtJ1y" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 122} outputId="3915b1c6-de0e-4a89-b345-a18b3d90add8"
from google.colab import drive
drive.mount('/content/drive')

# + id="exKPtcJyUyCX" colab_type="code" colab={}
# Step 3 - verify we've got *something*
flag_data.head()
# ?pd.read_csv

# + id="rNmkv2g8VfAm" colab_type="code" outputId="055bcf09-ed6b-4dea-957b-3a911481536b" colab={"base_uri": "https://localhost:8080/", "height": 555}
# Step 4 - Looks a bit odd - verify that it is what we want
flag_data.count()

# + id="iqPEwx3aWBDR" colab_type="code" outputId="6acf3192-dac1-4007-debc-7c5e820b6aa5" colab={"base_uri": "https://localhost:8080/", "height": 87}
# !curl https://archive.ics.uci.edu/ml/machine-learning-databases/flags/flag.data | wc

# + id="5R1d1Ka2WHAY" colab_type="code" outputId="0b4539d9-52ba-4d4e-edd1-6c540d93b55f" colab={"base_uri": "https://localhost:8080/", "height": 5081}
# So we have 193 observations with funny names, file has 194 rows
# Looks like the file has no header row, but read_csv assumes it does
help(pd.read_csv)

# + id="EiNiR6vExQUt" colab_type="code" colab={}
# ?pd.read_csv

# + id="oQP_BuKExQWE" colab_type="code" colab={}
??pd.read_csv

# + id="o-thnccIWTvc" colab_type="code" outputId="57799072-a851-4d4f-fd37-e24f6997492b" colab={"base_uri": "https://localhost:8080/", "height": 273}
# Alright, we can pass header=None to fix this
flag_data = pd.read_csv(flag_data_url, header=None)
flag_data.head()

# + id="iG9ZOkSMWZ6D" colab_type="code" outputId="d8b6f16a-6964-41f4-b438-9ddbff9eb08a" colab={"base_uri": "https://localhost:8080/", "height": 555}
flag_data.count()

# + id="gMcxnWbkWla1" colab_type="code" outputId="52fc01aa-00bd-4420-8539-0fe7dba9e263" colab={"base_uri": "https://localhost:8080/", "height": 555}
flag_data.isna().sum()

# + [markdown] id="AihdUkaDT8We" colab_type="text"
# ### Yes, but what does it *mean*?
#
# This data is fairly nice - it was "donated" and is already "clean" (no missing values). But there are no variable names - so we have to look at the codebook (also from the site).
#
# ```
# 1. name: Name of the country concerned
# 2. landmass: 1=N.America, 2=S.America, 3=Europe, 4=Africa, 4=Asia, 6=Oceania
# 3. zone: Geographic quadrant, based on Greenwich and the Equator; 1=NE, 2=SE, 3=SW, 4=NW
# 4. area: in thousands of square km
# 5. population: in round millions
# 6. language: 1=English, 2=Spanish, 3=French, 4=German, 5=Slavic, 6=Other Indo-European, 7=Chinese, 8=Arabic, 9=Japanese/Turkish/Finnish/Magyar, 10=Others
# 7. religion: 0=Catholic, 1=Other Christian, 2=Muslim, 3=Buddhist, 4=Hindu, 5=Ethnic, 6=Marxist, 7=Others
# 8. bars: Number of vertical bars in the flag
# 9. stripes: Number of horizontal stripes in the flag
# 10. colours: Number of different colours in the flag
# 11. red: 0 if red absent, 1 if red present in the flag
# 12. green: same for green
# 13. blue: same for blue
# 14. gold: same for gold (also yellow)
# 15. white: same for white
# 16. black: same for black
# 17. orange: same for orange (also brown)
# 18. mainhue: predominant colour in the flag (tie-breaks decided by taking the topmost hue, if that fails then the most central hue, and if that fails the leftmost hue)
# 19. circles: Number of circles in the flag
# 20. crosses: Number of (upright) crosses
# 21. saltires: Number of diagonal crosses
# 22. quarters: Number of quartered sections
# 23. sunstars: Number of sun or star symbols
# 24. crescent: 1 if a crescent moon symbol present, else 0
# 25. triangle: 1 if any triangles present, 0 otherwise
# 26. icon: 1 if an inanimate image present (e.g., a boat), otherwise 0
# 27. animate: 1 if an animate image (e.g., an eagle, a tree, a human hand) present, 0 otherwise
# 28. text: 1 if any letters or writing on the flag (e.g., a motto or slogan), 0 otherwise
# 29. topleft: colour in the top-left corner (moving right to decide tie-breaks)
# 30. botright: Colour in the bottom-left corner (moving left to decide tie-breaks)
# ```
#
# Exercise - read the help for `read_csv` and figure out how to load the data with the above variable names. One pitfall to note - with `header=None` pandas generated variable names starting from 0, but the above list starts from 1...

# + id="okEjAUHwEZtE" colab_type="code" colab={}


# + [markdown] id="XUgOnmc_0kCL" colab_type="text"
# ## Loading from a local CSV to Google Colab

# + id="-4LA4cNO0ofq" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 153} outputId="f1bf35f7-8233-4e9e-fe28-377115cfa4b4"


# + [markdown] id="aI2oN4kj1uVQ" colab_type="text"
# # Part 2 - Basic Visualizations

# + [markdown] id="INqBGKRl88YD" colab_type="text"
# ## Basic Data Visualizations Using Matplotlib

# + id="6FsdkKuh8_Rz" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 283} outputId="0816886b-a4c1-4810-a3af-a04e1b4d947b"
import matplotlib.pyplot as plt



# + id="huwUQ7zE9gkD" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 521} outputId="32bde518-e2c3-4000-aae9-31e83564aa67"


# + id="CSmpwXQN9o8o" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 531} outputId="43c5b431-9e52-44ae-945a-33bba65013e7"
# Seaborn Density Plot



# + id="TMMJG5rQ-g_8" colab_type="code" colab={}


# + [markdown] id="ipBQKbrl76gE" colab_type="text"
# ## Create the same basic Visualizations using Pandas

# + id="qWIO8zuhArEr" colab_type="code" colab={}
# Pandas Histogram - Look familiar?

# + id="zxEajNvjAvfB" colab_type="code" colab={}
# Pandas Scatterplot

# + id="XjR5i6A5A-kp" colab_type="code" colab={}
# Pandas Scatter Matrix - Usually doesn't look too great.

# + [markdown] id="tmJSfyXJ1x6f" colab_type="text"
# # Part 3 - Deal with Missing Values

# + [markdown] id="bH46YMHEDzpD" colab_type="text"
# ## Diagnose Missing Values
#
# Lets use the Adult Dataset from UCI. <https://github.com/ryanleeallred/datasets>

# + id="NyeZPpxRD1BA" colab_type="code" colab={}


# + [markdown] id="SYK5vXqt7zp1" colab_type="text"
# ## Fill Missing Values

# + id="32ltklnQ71A6" colab_type="code" colab={}


# + [markdown] id="nPbUK_cLY15U" colab_type="text"
# ## Your assignment - pick a dataset and do something like the above
#
# This is purposely open-ended - you can pick any data set you wish. It is highly advised you pick a dataset from UCI or a similar semi-clean source. You don't want the data that you're working with for this assignment to have any bigger issues than maybe not having headers or including missing values, etc.
#
# After you have chosen your dataset, do the following:
#
# - Import the dataset using the method that you are least comfortable with (!curl or CSV upload).
#  - Make sure that your dataset has the number of rows and columns that you expect.
#  - Make sure that your dataset has appropriate column names, rename them if necessary.
#  - If your dataset uses markers like "?" to indicate missing values, replace them with NaNs during import.
# - Identify and fill missing values in your dataset (if any)
#  - Don't worry about using methods more advanced than the `.fillna()` function for today.
# - Create one of each of the following plots using your dataset
#  - Scatterplot
#  - Histogram
#  - Density Plot
#  - Pairplot (note that pairplots will take a long time to load with large datasets or datasets with many columns)
#
# If you get that done and want to try more challenging or exotic things, go for it! Use documentation as illustrated above, and follow the 20-minute rule (that is - ask for help if you're stuck!).
#
# If you have loaded a few traditional datasets, see the following section for suggested stretch goals.

# + [markdown] id="tc1yRfS7dMzw" colab_type="text"
# **Importing and cleaning data:**

# + id="NJdISe69ZT7E" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 153} outputId="cc9de980-2350-4303-dbc9-7f977624902b"
# importing libraries
import pandas as pd
import numpy as np

# creating column_headers list to create header row. The info is in the names file on UCI.

column_headers = ['is-democrat', 'handicapped-infants', 'water-project-cost-sharing', 'adoption-of-the-budget-resolution', 'physician-fee-freeze',
                 'el-salvador-aid', 'religions-groups-in-schools', 'anti-satellite-test-ban', 'aid-to-nicaraguan-contras', 
                 'mx-missile', 'immigration', 'synfuels-corporation-cutback', 'education-spending', 'superfund-right-to-sue', 
                 'crime', 'duty-free-exports', 'export-administration-act-south-africa']

# downloading data using read_csv
df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data', names = column_headers)

# Exploratory Data Analysis -----------

# print(df.head());
# df.isnull().sum()

# This says there is no null values, but the UCI site says there is missing data.
# Looking at the head, I can see that there are question marks in the data so I am going to replace them with null values
df.replace("?", np.NaN, inplace = True)

# Checking to make sure the question marks were replaced with null values correctly
# print(df.head());

# print(df.isnull().sum())

# Reading the names file from the UCI database, it says that question marks do not necessarily indicate missing data. Question markse mean the data was not yea or nea. The only other type of vote I can think of would be abstaining so I am going to replace the question marks with 'a' for abstain.
df.fillna('a', inplace = True)
# Checking to make sure that filled correctly
print(df.head())

# I now see that there is no header row, so I am changing the read_csv call in the beginning to handle that

# + [markdown] id="Ybsi4VRjdYES" colab_type="text"
# **Scatterplot:**

# + id="x1BcD2YfdmgY" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 328} outputId="24830adc-1fe9-4fdd-a248-fe6261c1632d"
# Scatter Plot

# Converting the y/n/a strings to numbers
df.replace("n", 0, inplace = True)
df.replace("a", 1, inplace = True)
df.replace("y", 2, inplace = True)

# Converting Class Name values to boolean. I also renamed the column header in the column_headers list from 'Class Name' to 'is-democrat' to be more descriptive
df.replace("republican", 0, inplace = True)
df.replace("democrat", 1, inplace = True)
df.head()

# Created a scatterplot, but it isn't very useful because you cant see the amount of data at each point. I'm sure there's a way to make bigger dots or something if there's more data.
df.plot.scatter('is-democrat', 'immigration');

# + [markdown] id="xqgIQaL4dz6t" colab_type="text"
# **Histogram:**

# + id="rEXHnjiFd3hQ" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 531} outputId="9ab87f44-08f1-4e7d-9dbe-8416dbd81e5f"
# Histogram

df['is-democrat'].hist()
plt.show()

df['immigration'].hist()
plt.show()

# + [markdown] id="C_OkR9oGd9G-" colab_type="text"
# **Density Plot:**

# + id="ZKO5dbhleAGE" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 531} outputId="d9a57d1e-fc91-4b2d-80ac-30ae576d8ede"
# Density Plot

df.immigration.plot.density()
plt.show()

df['handicapped-infants'].plot.density()
plt.show()

# + [markdown] id="-E33pq0neK7P" colab_type="text"
# **Pairplot**

# + id="V_NDsYqqeMz9" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 1000} outputId="312a35d7-a9e4-4ade-c2f0-8010d752a10d"
# Seaborn Pairplot

import seaborn as sns

sns.set(style = "ticks", color_codes = True)
g = sns.pairplot(df)

# + [markdown] id="MZCxTwKuReV9" colab_type="text"
# ## Stretch Goals - Other types and sources of data
#
# Not all data comes in a nice single file - for example, image classification involves handling lots of image files. You still will probably want labels for them, so you may have tabular data in addition to the image blobs - and the images may be reduced in resolution and even fit in a regular csv as a bunch of numbers.
#
# If you're interested in natural language processing and analyzing text, that is another example where, while it can be put in a csv, you may end up loading much larger raw data and generating features that can then be thought of in a more standard tabular fashion.
#
# Overall you will in the course of learning data science deal with loading data in a variety of ways. Another common way to get data is from a database - most modern applications are backed by one or more databases, which you can query to get data to analyze. We'll cover this more in our data engineering unit.
#
# How does data get in the database? Most applications generate logs - text files with lots and lots of records of each use of the application. Databases are often populated based on these files, but in some situations you may directly analyze log files. The usual way to do this is with command line (Unix) tools - command lines are intimidating, so don't expect to learn them all at once, but depending on your interests it can be useful to practice.
#
# One last major source of data is APIs: https://github.com/toddmotto/public-apis
#
# API stands for Application Programming Interface, and while originally meant e.g. the way an application interfaced with the GUI or other aspects of an operating system, now it largely refers to online services that let you query and retrieve data. You can essentially think of most of them as "somebody else's database" - you have (usually limited) access.
#
# *Stretch goal* - research one of the above extended forms of data/data loading. See if you can get a basic example working in a notebook. Image, text, or (public) APIs are probably more tractable - databases are interesting, but there aren't many publicly accessible and they require a great deal of setup.

# + [markdown] id="tHwGhLTg1Ai7" colab_type="text"
# **Getting basic API calls to work**

# + id="f4QP6--JBXNK" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 190} outputId="c8f8a5ef-334d-4e37-c03f-ad0cbdd50d9b"
# installing stuff I need for API requests
# !pip install requests
import requests

# ----This was code I copied from a Medium post to get a feel for how the process works
# request = requests.get('http://api.open-notify.org/astros.json')
# print(request.text)

# people = request.json()
# print(people)

# print("\nNumber of people in space: " + str(people['number']))
# ----End of code I copied from the Medium post

# Creating request from XKCD's comic today
request2 = requests.get('https://xkcd.com/info.0.json')

# checking status_code and printing text to make sure there's something there
print(request2.status_code)
print(request2.text)

# formatting the data pulled from the API
xkcd_data = request2.json()
print("\nToday's xkcd comic title is '" + str(xkcd_data['safe_title']+"'."))



# + [markdown] id="WKo-WNCX1KY-" colab_type="text"
# **Creating a dataframe from API calls**

# + id="tuwHyOuei1Cc" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 85} outputId="ea087014-b915-4d0b-bc29-0eec52d7fe45"
import json
example_for_df = requests.get('https://xkcd.com/1/info.0.json')
formatted_example = example_for_df.json()
df = pd.DataFrame([formatted_example], columns=formatted_example.keys())


for i in range(2,10):
  xkcd_call = requests.get('https://xkcd.com/' + str(i) + '/info.0.json')
  formatted_call = xkcd_call.json()
  df.append(formatted_call, ignore_index=True)
  
print(df)
  
                        


StarcoderdataJupyterScriptsDedupFiltered