# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/spentaur/DS-Unit-1-Sprint-1-Dealing-With-Data/blob/master/module1-afirstlookatdata/LS_DS_111_A_First_Look_at_Data.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="Okfr_uhwhS1X" colab_type="text"
# # Lambda School Data Science - A First Look at Data
#
#

# + [markdown] id="9dtJETFRhnOG" colab_type="text"
# ## Lecture - let's explore Python DS libraries and examples!
#
# The Python Data Science ecosystem is huge. You've seen some of the big pieces - pandas, scikit-learn, matplotlib. What parts do you want to see more of?

# + id="WiBkgmPJhmhE" colab_type="code" outputId="8e564c02-c8d4-482c-f837-eca9403bf41e" colab={"base_uri": "https://localhost:8080/", "height": 36}
# TODO - we'll be doing this live, taking requests
# and reproducing what it is to look up and learn things
2 + 2

# + id="oRULtcvouo8Y" colab_type="code" colab={}
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# %matplotlib inline

# + id="-rmZBFpTuscF" colab_type="code" colab={}
drinks = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/alcohol-consumption/drinks.csv')

# + id="i-Ybt6vFvnXH" colab_type="code" outputId="ef6b7759-5676-4337-fe44-5d6f55cbd4c2" colab={"base_uri": "https://localhost:8080/", "height": 363}
drinks.sort_values(by='beer_servings', ascending=False).head(10)

# + id="cUAAj3-PyvTx" colab_type="code" colab={}
drinks['drinks_alcohol'] = np.where(drinks['total_litres_of_pure_alcohol'] > 0, 'Yes', 'No')

# + id="u1ms_cofzzYE" colab_type="code" outputId="c37e04d0-e899-4728-873a-8923c8abcbb2" colab={"base_uri": "https://localhost:8080/", "height": 206}
drinks.head()

# + id="Ormn2rTh0CFl" colab_type="code" outputId="715d48b9-6ad2-40d9-bf54-c78908fbcad5" colab={"base_uri": "https://localhost:8080/", "height": 441}
drinks.total_litres_of_pure_alcohol.hist()
drinks['total_litres_of_pure_alcohol'].describe()

# + id="FSfBeSbs1Xj0" colab_type="code" colab={}
drinks['drinks_alcohol'] = np.where(drinks['total_litres_of_pure_alcohol'] > 7.2, 'high',
                           np.where(drinks['total_litres_of_pure_alcohol'] > 4.2, 'medium',
                           np.where(drinks['total_litres_of_pure_alcohol'] == 0, 'none', 'low')))

# + id="EKnQISGC2BVW" colab_type="code" outputId="6b749d2c-d898-4b02-be72-eeb07d6f1592" colab={"base_uri": "https://localhost:8080/", "height": 113}
drinks['drinks_alcohol'].value_counts()

# + id="GqJUEDrU33HW" colab_type="code" outputId="35ae2234-9cc4-45c8-c93e-534281b77567" colab={"base_uri": "https://localhost:8080/", "height": 312}
countries = pd.read_csv('https://raw.githubusercontent.com/lukes/ISO-3166-Countries-with-Regional-Codes/master/all/all.csv')
print(countries.shape)
countries.head()

# + id="gDB17JqK5xgr" colab_type="code" outputId="193ad956-07a6-4eb2-93a4-e1e59ff8859d" colab={"base_uri": "https://localhost:8080/", "height": 1000}
countries[['name', 'region', 'sub-region']]

# + id="rJ7dT5Ki86qG" colab_type="code" colab={}
drinks.at[184, 'country'] = "United States of America"

# + id="Z6u-fdj04jo5" colab_type="code" colab={}
df = pd.merge(drinks, countries[['name', 'region', 'sub-region']], how='left', left_on='country', right_on='name')

# + id="Y6VuvHEJ6_br" colab_type="code" outputId="3369f6df-a5f3-4495-98ed-5a3509040441" colab={"base_uri": "https://localhost:8080/", "height": 208}
df.isnull().sum()

# + id="pFc3s0fd8Kp3" colab_type="code" outputId="29911669-6195-45e1-f466-3e05434cd81d" colab={"base_uri": "https://localhost:8080/", "height": 834}
df[df['name'].isna()].head(30)

# + id="UBpOQ263_sKu" colab_type="code" outputId="63f08107-aa77-48a0-9369-d319dbd49a71" colab={"base_uri": "https://localhost:8080/", "height": 206}
df.head()

# + id="5Dp6GBHM-YBz" colab_type="code" outputId="3fd3b821-9119-49b7-cd15-c791cb87abdd" colab={"base_uri": "https://localhost:8080/", "height": 361}
df['sub-region'].value_counts()

# + id="HCqGDBI0_aZF" colab_type="code" outputId="e9ce722a-0b6c-43f2-f3c5-f4816d1ea2df" colab={"base_uri": "https://localhost:8080/", "height": 441}
df.groupby('region')['beer_servings'].mean().plot(kind="bar", figsize=(10,6))
plt.title('Average Beer Servings by Region')
plt.ylabel('Beer Servings per Year')
plt.show()

# + id="Tq3CNFmlAa4s" colab_type="code" outputId="5a048ec0-fb25-4099-cab8-2c279126b0d6" colab={"base_uri": "https://localhost:8080/", "height": 420}
df.boxplot(column="beer_servings", by="region", figsize=(10,6));

# + id="uoew5Z_RCS4E" colab_type="code" colab={}
df = df.dropna()

# + id="0iRyFPPxCfte" colab_type="code" outputId="9246e676-2f90-4267-9007-0a001ea19fc2" colab={"base_uri": "https://localhost:8080/", "height": 36}
df.shape

# + id="NR55Rjz6B8bz" colab_type="code" outputId="fa50756c-f581-4083-e419-d805a4311139" colab={"base_uri": "https://localhost:8080/", "height": 647}
from bokeh.io import output_file, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, CategoricalColorMapper
from bokeh.palettes import d3
from bokeh.plotting import figure
from bokeh.transform import transform
output_notebook()
x = df.beer_servings.values
y = df.wine_servings.values
country = df.country
region = df.region
source = ColumnDataSource(data=dict(x=x, y=y, country=country, region=region))
hover = HoverTool(tooltips=[
    ("index", "$index"),
    ("(x,y)", "(@x, @y)"),
    ('country', '@country'),
])
palette = d3['Category10'][len(df['region'].unique())]
mapper = CategoricalColorMapper(factors = df['region'].unique(), palette = palette)
# create figure and plot
p = figure(plot_width=600, plot_height=600, 
           tools=[hover, "wheel_zoom", "box_zoom", "reset"], 
           title="Beer and Wine Servings per Year by Country")
# create plot
p.scatter(x='x', y='y', size=10, alpha=0.8,
          color={'field': 'region', 'transform': mapper},
          legend='region', 
          source=source)
# add axis labels
p.xaxis.axis_label = "Beer Servings"
p.yaxis.axis_label = "Wine Servings"
output_file('interactive_bokeh_plot.html')
show(p)

# + [markdown] id="lOqaPds9huME" colab_type="text"
# ## Assignment - now it's your turn
#
# Pick at least one Python DS library, and using documentation/examples reproduce in this notebook something cool. It's OK if you don't fully understand it or get it 100% working, but do put in effort and look things up.

# + id="2lFAgZVv1Biu" colab_type="code" colab={}
import pandas as pd
import matplotlib.pyplot as plt

# + id="hWWLx0Ev1VIf" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 56} outputId="196092e6-8c51-448b-942a-a20d442e3233"
ratings = pd.read_csv("title.ratings.tsv", sep='\t')
basics = pd.read_csv("title.basics.tsv", sep='\t')

# + id="UUqG2j-w2Ght" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 303} outputId="8199ee84-0c76-43c2-ff2a-83f9006ee835"
print(ratings.shape)
print(ratings.isnull().sum())
ratings.head()

# + id="vacW12EN2NTd" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 419} outputId="d170b643-0f96-488c-e6d2-ff69a6ec009e"
print(basics.shape)
print(basics.isnull().sum())
basics.head()

# + id="UOL12t-028-r" colab_type="code" colab={}
basics = basics.dropna()

# + id="fDVr5HLo2Z_d" colab_type="code" colab={}
df = pd.merge(ratings, basics, how='left', left_on='tconst', right_on='tconst')

# + id="l1jlY1Sv2jVQ" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 226} outputId="b83f950f-40c0-4f14-fe05-6727776aea8f"
print(df.shape)
df.head()

# + id="RQ1zIPyp2zH8" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 249} outputId="29d9995c-4d13-4e75-a84a-36b8c9aa7314"
df.isnull().sum()

# + id="Dhr3mf1O3uqe" colab_type="code" colab={}
df = df.drop(columns=['tconst', 'titleType'])

# + id="g8_f3fCMffa-" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 36} outputId="aca87bab-53d8-4500-d4be-9b36bfe0263d"
df.shape

# + id="8eP0RjkHnFxx" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 300} outputId="e952624f-e3e1-40f0-a507-bd5f084c346c"
df.describe()

# + id="ieaUjvH6fr1M" colab_type="code" colab={}
df = df[df['runtimeMinutes'] != '\\N']
df = df[df['genres'] != '\\N']
df = df[df['startYear'] != '\\N']

# + id="8qCLOXlEgxuj" colab_type="code" colab={}
df['runtimeMinutes'] = df['runtimeMinutes'].astype(str).astype(int)
df['startYear'] = df['startYear'].astype(str).astype(int)

# + id="59ZwpsKlfxO8" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 206} outputId="d6bd1c4d-2408-4123-8596-547c57c50dc4"
df.head()

# + id="PJnZ86d4f2Zr" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 211} outputId="b5aec55e-3a19-4a0c-da2c-54d77b8e0ef6"
df.dtypes

# + id="2fnnkPFvpQDa" colab_type="code" colab={}
df = df.loc[(df['numVotes'] > 1000) & (df['runtimeMinutes'] > 10) & (df['endYear'] == '\\N') & (df['isAdult'] == 0)]

# + id="MwOG-pZa06Oh" colab_type="code" colab={}
df = df.drop(columns=['isAdult'])

# + id="vFLp-4Luw95-" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 36} outputId="c78f7ac0-08f6-4e27-eda2-16381a7ec1ff"
df.shape

# + id="bbFVuDez3FQU" colab_type="code" colab={}
df_year_count = df.groupby('startYear').size().to_frame()

# + id="mUvXdKkb4GgQ" colab_type="code" colab={}
df_year_count.columns = ['numMovies']

# + id="cOIWfnMM53U7" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 36} outputId="05549315-4a80-4f13-84a7-70e869b129bc"
df_year_count.shape

# + id="KYUal5IO3Zim" colab_type="code" colab={}
df_by_year = df.groupby('startYear').mean()

# + id="aA64F7K76PIF" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 36} outputId="eb838c68-ae2d-41c0-903a-53f746a447eb"
df_by_year.shape

# + id="fsPJ800Z4SeK" colab_type="code" colab={}
df_by_year = df_by_year.join(df_year_count, how='left')

# + id="yuJ7kj6A3lvs" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 238} outputId="73feaa61-3a97-4062-e48b-178b9ae5f441"
df_by_year.head()

# + id="kjLhzjsQzZSg" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 404} outputId="0807d9b9-b0dd-443f-c1a0-68d6b05555c7"
df_by_year['averageRating'].plot(figsize=(10,6), title="Are Movies Getting Worse?")
plt.xlabel('Year')
plt.ylabel('Average Rating')
plt.show()

# + id="qU9YtV-J1Lla" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 404} outputId="6c83199a-3018-410a-b19f-6f5c8040ee56"
df_by_year['numVotes'].plot(figsize=(10,6), title="Are Movies Getting More Popular?")
plt.xlabel('Year')
plt.ylabel('Average Number of Votes')
plt.show()

# + id="XnTBKTna1fIS" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 404} outputId="227d7e36-a585-490d-962d-a2786a7c3fe5"
df_by_year['runtimeMinutes'].plot(figsize=(10,6), title="Are Movies Getting Longer?")
plt.xlabel('Year')
plt.ylabel('Average Runtime in Minutes')
plt.show()

# + id="EKnnRoaD6T80" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 404} outputId="48259487-30c1-414b-9cd2-ff0307b5ee28"
df_by_year['numMovies'].plot(figsize=(10,6), title="Are There Too Many Movies?")
plt.xlabel('Year')
plt.ylabel('Number of Movies')
plt.show()

# + [markdown] id="CPe6T0jb8Fib" colab_type="text"
# ### Assignment questions
#
# After you've worked on some code, answer the following questions in this text block:
#
# 1.  Describe in a paragraph of text what you did and why, as if you were writing an email to somebody interested but nontechnical.
#
# 2.  What was the most challenging part of what you did?
#
# 3.  What was the most interesting thing you learned?
#
# 4.  What area would you like to explore with more time?
#
#
#
#
# ---
#
# 1. I've always been a fan of bad 80's movies, so I wanted to see if movies in the 80's were acutally worse then other decades. In order to do this, I first downloaded data from IMdb. Once I had the data I looked at it and saw that I was going to have to "clean" it up a little bit. Bascially this just means getting rid of any movies or tv shows that don't have all the information I'm interested in. I also wanted to eliminate anything that doesn't have a certain number of ratings, let's say above 1,000 votes. Now that I've got the data how I want, I group all the movies by year, and then get the mean number of votes per movie per year, the mean rating per movie per year, and the mean length per movie per year. I also decided I wanted to see which years had the most movies, so I counted them up and added that to the data set as well. Finally I plotted everything up and confirmed that yes, the 80's did infact have the worst movies. But I love them anyway!
# 2. The most challenging part was deciding what data to use, what data to get rid of and what data to change.
# 3. I learned that movies have been getting less reviewed, higher rated, shorter, and much more frequent in the last ~15 years.
# 4. I'd like to explore more about specific genres, like have horror movies kept up with this trend.

StarcoderdataJupyterScriptsDedupFiltered