# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Web Data Scraping
#
#
#
# ## Acknowledgements
#
# These notebooks are adaptations from a 5 session mini course at the University of Colorado. The github repo can be found [here](https://github.com/CU-ITSS/Web-Data-Scraping-S2019) [Spring 2019 ITSS Mini-Course] The course is taught by [<NAME>, Ph.D.](http://brianckeegan.com/) [Assistant Professor, Department of Information Science](https://www.colorado.edu/cmci/people/information-science/brian-c-keegan). They have been adapted for relevant content and integration with Docker so that we all have the same environment. Professor Keegan suggests using a most recent version of Python (3.7) which is set in the `requirements.txt` file.
#
# The Spring ITSS Mini-Course was adapted from a number of sources including [Allison Morgan](https://allisonmorgan.github.io/) for the [2018 Summer Institute for Computational Social Science](https://github.com/allisonmorgan/sicss_boulder), which were in turn derived from [other resources](https://github.com/simonmunzert/web-scraping-with-r-extended-edition) developed by [<NAME>](http://simonmunzert.github.io/) and [<NAME>](http://www.chrisbail.net/). 
#
# This notebook is adapted from excellent notebooks in Dr. [<NAME>](http://cody.bunta.in/)'s seminar on [Social Media and Crisis Informatics](http://cody.bunta.in/teaching/2018_winter_umd_inst728e/) as well as the [PRAW documentation](https://praw.readthedocs.io/en/latest/).
#
#

# ## Parsing HTML data into tabular data
#
# The overall goal we have as researchers in scraping data from the web is converting data from one structured format (HTML's tree-like structures) into another structured format (probably a tabular structure with rows and columns). 
# This could involve simply reading tables out of a webpage all the way up to taking irregularly-structured HTML elements into a tabular format. 
#
# We are going to make some use of the [`pandas`](https://pandas.pydata.org/) library ("**pan**el **da**ta", not the cute animal), which is Python's implementation of a data frame concept. This is a very powerful and complex library that I typically spend more than 12 hours of lecture teaching in intermediate programming classes. I hope to convey some important elements as we work through material, but it is far beyond the scope of this class to be able to cover all the fundamentals and syntax. 
#
# Let's begin by importing the libraries we'll need in this notebook: requests, BeautifulSoup, and pandas

# +
# Most straight-forward way to import a librayr in Python
import requests

# BeautifulSoup is a module inside the "bs4" library, we only import the BeautifulSoup module
from bs4 import BeautifulSoup

# We import pandas but give the library a shortcut alias "pd" since we will call its functions so much
import pandas as pd
# -

# ### Reading an HTML table into Python
#
# [The Numbers](http://www.the-numbers.com) is a popular source of data about movies' box office revenue numbers. Their daily domestic charts are HTML tables with the top-grossing movies for each day of the year, going back for several years. This [table](https://www.the-numbers.com/box-office-chart/daily/2018/12/25) for Christmas day in 2018 has coluns for the current week's ranking, previous week's ranking, name of movie, distributor, gross, change over the previous week, number of theaters, revenue per theater, total gross, and number of days since release. This looks like a fairly straightforward table that could be read directly into data frame-like structure.
#
# Using the Inspect tool, we can see the table exists as a `<table border="0" ... align="CENTER">` element with child tags like `<tbody>` and `<tr>` (table row). Each `<tr>` has `<td>` which defines each of the cells and their content. For more on how HTML defines tables, check out [this tutoral](https://www.w3schools.com/html/html_tables.asp).
#
# Using `requests` and `BeautifulSoup` we would get this webpage's HTML, turn it into soup, and then find the table (`<table>`) or the table rows (`<tr>`) and pull out their content.

# +
# Make the request
xmas_bo_raw = requests.get('https://www.the-numbers.com/box-office-chart/daily/2018/12/25').text

# Turn into soup, specify the HTML parser
xmas_bo_soup = BeautifulSoup(xmas_bo_raw,'html.parser')

# Use .find_all to retrieve all the tables in the page
xmas_bo_tables = xmas_bo_soup.find_all('table')
# -

# It turns out there are two tables on the page, the first is a baby table consisting of the "Previous Chart", "Chart Index", and "Next Chart" at the top. We want the second table with all the data: `xmas_bo_tables[1]` returns the second chart (remember that Python is 0-indexed, so the first chart is at `xmas_bo_tables[0]`). With this table identified, we can do a second `find_all` to get the table rows inside it and we save it as `xmas_bo_trs`.

xmas_bo_trs = xmas_bo_tables[1].find_all('tr')

# Let's inspect a few of these rows. The first row in our list of rows under `xmas_bo_trs` should be the header with the names of the columns.

xmas_bo_trs[0]

# The next table row should be for Aquaman.

xmas_bo_trs[1]

# If we wanted to access the contents of this table row, we could use the `.contents` method to get a list of each of the `<td>` table cells, which (frustratingly) intersperses newline characters.

xmas_bo_trs[1].contents

# Another alternative is to use the `.text` method to get the text content of all the cells in this row.

xmas_bo_trs[1].text

# The `\n` characters re-appear here, but if we `print` out this statement, we see their newline functionality.

print(xmas_bo_trs[1].text)

# We could use string processing to take this text string and convert it into a simple list of data. `.split('\n')` will split the string on the newline characters and return a list of what exists in between.

xmas_bo_trs[1].text.split('\n')

# We'll write a `for` loop to go through all the table rows in `xmas_bo_trs`, get the list of data from the row, and add it back to a list of all the rows.

# +
cleaned_xmas_bo_rows = []

# Loop through all the non-header (first row) table rows
for row in xmas_bo_trs[1:]:
    
    # Get the text of the row and split on the newlines (like above)
    cleaned_row = row.text.split('\n')
    
    # Add this cleaned row back to the external list of row data
    cleaned_xmas_bo_rows.append(cleaned_row)
    
# Inspect the first few rows of data
cleaned_xmas_bo_rows[:2]
# -

# Now we can pass this list of lists in `cleaned_xmas_bo_rows` to pandas's `DataFrame` function and hopefully get a nice table out.

# +
xmas_bo_df = pd.DataFrame(cleaned_xmas_bo_rows)

# Inspect
xmas_bo_df.head()
# -

# We need to do a bit of cleanup on this data:
#
# * Columns 0 and 11 are all empty
# * Add column names

# +
# Drop columns 0 and 11 and overwrite the xmas_box_df variable
xmas_bo_df = xmas_bo_df.drop(columns=[0,11])

# Rename the columns
xmas_bo_df.columns = ['Rank','Last rank','Movie','Distributor','Gross',
                      'Change','Theaters','Per theater','Total gross',
                      'Days']

# Write to disk
# xmas_bo_df.to_csv('christmas_2018_box_office.csv',encoding='utf8')

# Inspect
xmas_bo_df.head()
# -

# ### `pandas`'s `read_html`
# That was a good amount of work just to get this simple HTML table into Python. But it was important to cover how table elements moved from a string in `requests`, into a soup object from `BeautifulSoup`. into a list of data, and finally into `pandas`. 
#
# `pandas` also has powerful functionality for reading tables directly from HTML. If we convert the soup of the first table (`xmas_bo_tables[1]`) back into a string, `pandas` can read it directly into a table. 
#
# There are a few ideosyncracies here, the result is a list of dataframes—even if there's only a single table/dataframe—so we need to return the first (and only) element of this list. This is why there's a `[0]` at the end and the `.head()` is just to show the first five rows.

# +
xmas_bo_table_as_string = str(xmas_bo_tables[1])

pd.read_html(xmas_bo_table_as_string)[0].head()
# -

# The column names got lumped in as rows, but we can fix this as well with the `read_html` function by passing the row index where the column lives. In this case, it is the first row, so we pass `header=0`.

pd.read_html(xmas_bo_table_as_string,header=0)[0].head()

# Finally, you can point `read_html` at a URL without any `requests` or `BeautifulSoup` and get all the tables on the page as a list of DataFrames. `pandas` is simply doing the `requests` and `BeautifulSoup` on the inside. Interestingly, I'm getting a [HTTP 403](https://en.wikipedia.org/wiki/HTTP_403) error indicating the server (The Numbers) is forbidding the client (us) from accessing their data using this strategy. We will discuss next week whether and how to handle situations where web servers refuse connections from non-human clients. In this case, you cannot use the off-the-shelf `read_html` approach and would need to revert to using the `requests`+`BeautifulSoup` approach above.

simple_tables = pd.read_html('https://www.the-numbers.com/box-office-chart/daily/2018/12/25')
simple_tables

# If we point it at Wikipedia's [2018 in film](https://en.wikipedia.org/wiki/2018_in_film), it will pull all of the tables present on the page.

simple_tables = pd.read_html('https://en.wikipedia.org/wiki/2018_in_film')

# The first three correspond to the "Year in film" navigation box on the side and are poorly-formatted by default.

simple_tables[0]

# The fourth table in the `simple_tables` list we got from parsing the Wikipedia page with `read_html` is the table under the "Highest-grossing films" section.

simple_tables[3]

# You can pass the "header" option in `read_html` to make sure the column names from a particular row (in this case the first row) do not accidentally become rows of data. 

wiki_top_grossing_t = pd.read_html('https://en.wikipedia.org/wiki/2018_in_film',header=0)[3]
wiki_top_grossing_t

# Note that there are still a few errors in this table because the "Disney" value in the Wikipedia table spans two rows and `read_html` thus skips the "Distributor" value for Black Panther. 

# +
# Copy the value at index position 1, column position Distributor to Wordwide gross
wiki_top_grossing_t.loc[1,'Worldwide gross'] = wiki_top_grossing_t.loc[1,'Distributor']

# Change the value at 1, Distributor to Disney
wiki_top_grossing_t.loc[1,'Distributor'] = 'Disney'

wiki_top_grossing_t
# -

# ## Writing your own parser
#
# We will return to the historical Oscars data. Even though data as prominent as this is likely to already exist in tabular format somewhere, we will maintain the illusion that we are the first to both scrape it and parse it into a tabular format. Our goal here is to write a parser that will (ideally) work across multiple pages; in this case, each of the award years.
#
# One of the first things we should do before writing any code is come up with a model of what we want our data to look like at the end of this. This is an intuitive and "tidy" format, but you might come up with alternatives based on your analysis and modeling needs.
#
# | *Year* | *Category* | *Nominee* | *Movie* | *Won* |
# | --- | --- | --- | --- | --- |
# | 2019 | Actor in a leading role | <NAME> | Vice | NA |
# | 2019 | Actor in a leading role | <NAME> | A Star Is Born | NA |
# | 2019 | Actor in a leading role | <NAME> | At Eternity's Gate | NA |
# | 2019 | Actor in a leading role | <NAME> | <NAME> | NA |
# | 2019 | Actor in a leading role | <NAME> | Green Book | NA |
#
# We will begin with writing a parser for a (hopefully!) representative year, then scrape the data for all the years, then apply the scraper to each of those years, and finally combine all the years' data together into a large data set. 
#
# Let's begin with writing a parser for a (hopefully!) representative year: in this case, 2019 is actually not a great case because it is missing information about who won and lost since (at the time of my writing this notebook) the winners had not been announced. We will use 2018 instead and make the profoundly naïve assumption it should work the same going back in time.
#
# Start off with using `requests` to get the data and then use `BeautifulSoup` to turn it into soup we can parse through.

# +
oscars2018_raw = requests.get('https://www.oscars.org/oscars/ceremonies/2018').text

oscars2018_soup = BeautifulSoup(oscars2018_raw)
# -

# Using the Inspect tool exercise from Class 1, the `<div class="view-grouping">` seems to be the most promising tag for us to extract. Use `.find_all('div',{'class':'view-grouping'})` to (hopefully!) get all of these award groups. Inspect the first and last ones to make sure they looks coherent.

# +
# Get all the groups that have a <div class="view-grouping"> tag
oscars2018_groups = oscars2018_soup.find_all('div',{'class':'view-grouping'})

# Inspect the first one
oscars2018_groups[0]
# -

# The last group is something besides "Writing (Original Screenplay)" and it's not clear to me where this tag's content renders on the page.

# Inspect the last one
oscars2018_groups[-1]

# This puts us into something of a bind going forward: if the `.find_all` returns more groupings than we expected, then it's not sufficiently precise to identify *only* groupings of nominees.  However, there do not appear to be any child tags in the `oscars2018_groups[0]` grouping that uniquely differentiate them from the child tags present in the `oscars2018_groups[-1]` grouping. Another alternative is to simple parse the first 24 groupings, but this is a very brittle solution since other years' awards might have more or fewer groupings.

len(oscars2018_groups)

# ### Navigating the HTML tree to find more specific parent elements
# A third alternative is to leverage the tree structure of HTML and get the parent element in the hopes it is more unique than its children. In this case something like `<div id="quicktabs-tabpage-honorees-0"...>` is a promising lead. Use `find_all` to search for this tag and confirm there is only one the one `<div>` element (with its children) rather than multiple `<div>` elements matching "quicktabs-container-honorees".

# +
# Get the new tag group
oscars2018_parent_group = oscars2018_soup.find_all('div',{'id':'quicktabs-tabpage-honorees-0'})

# Hopefully there is only one group matching this pattern
len(oscars2018_parent_group)

# -

# So far so good, now we can use `find_all` on the soup for this `<div class="view-grouping">` to search *within* this specific parent group and hopefully there should be the 24 awards groupings. Nope, still 105. This is because 

# +
# Note the addition of the [0] since the _parent_group is a list with 1 element in it
# We just extract that single element (which is a soup) and then we can use find_all on it

oscars2018_true_groups = oscars2018_parent_group[0].find_all('div',{'class':'view-grouping'})

len(oscars2018_true_groups)
# -

# Hallelujah! The award names for each group live inside a `<div class="view-grouping-header">`, so we can `find_all` for those, loop through each, and print out the name.

for group in oscars2018_parent_group[0].find_all('div',{'class':'view-grouping-header'}):
    print(group.text)

# It turns out that the Oscars site loads a bunch of extra data that it does not render that lives underneath a `<div id="quicktabs-tabpage-honorees-1">` which is where the 81 extra "awards" come from. This appears to be an attempt to organize the page by film, rather than by category.
#
# ### Navigating the HTML tree from a specific child to find specific generic parents
# Now bear with me through some additional and presently unnecessary pain. Above, we were able to isolate the 24 category groupings we wanted through finding an appropriate *parent* tag and then working *down*. But I also want to show how we could identify the same 24 category groups by finding an appropriate *child* tag and working back up. This could be helpful in other situations where the elements are hard to disambiguate.
#
# Let's start by finding the tag for the "Actor in a Leading Role" from the soup containing all the tags.

oscars2018_groups[0]

# Rather than finding *all* the `<div class="view-grouping">` present in the page, we only want the 23 *siblings* of this specific tag. We can use the `find_next_siblings()` to get these 23 siblings. I do not like this method very much because you have to find the "eldest" sibling and then combine it with its siblings later on if you want all the children. In this case, you'd need to keep track of the `<div class="view-grouping">` corresponding to Best Actor and then combine it with its 23 siblings, rather than an approach that simply returns all 24 in a single list.

# +
oscars2018_group0_next_siblings = oscars2018_groups[0].find_next_siblings()

len(oscars2018_group0_next_siblings)
# -

# We could also go up to get the parent and then find all 24 of the `<div class='view-grouping'>` among the children.

# +
# From the child we like, get its parent
oscars2018_group0_parent = oscars2018_groups[0].parent

# Now with the parent, find all the relevant children
oscars2018_group0_parent_children = oscars2018_group0_parent.find_all('div',{'class':'view-grouping'})

# Confirm
len(oscars2018_group0_parent_children)
# -

# ### Checking the relevant fields
#
# That seemed like a major digression away from the core task of writing a parser, but it is critical that we write a parser that parses *only* the data we want and nothing else. Now that we have our 24 awards groups in `oscars2018_true_groups`, let's break one open and extract all the yummy data waiting inside.
#
# There are a few `<div>` sub-classes that are helpfully named that should make extracting this data a bit easier.
#
# * `<div class="view-grouping-header">` - name of the category
# * `<span class="golden-text">` - winner
# * `<div class="views-field views-field-field-actor-name">` - name of actor
# * `<div class="views-field views-field-title">` - title of movie

oscars2018_true_groups[0]

# "Zoom in" to the `views-field-field-actor-name`.

oscars2018_true_groups[0].find_all('div',{'class':"views-field views-field-field-actor-name"})

# These `<h4>` tags may be more specific and helpful.

oscars2018_true_groups[0].find_all('h4')

# Zoom into the `views-field-title`.

oscars2018_true_groups[0].find_all('div',{'class':"views-field views-field-title"})

# These `<span>` tags may be more specific and helpful, but there are also empty tags here clogging things up.

oscars2018_true_groups[0].find_all('span')

# As a battle-scarred web scraper, let me continue to emphasize the importance of quick-checking your assumptions before commiting to writing code. Are these fields still appropriate for other awards categories? Let's check the last category for original screenplay. Are the `<div>`s for "field-actor-name" still people and for "field-title" still movies? Nope. 
#
# Looking back at the web page, it's now obvious that the movie title and person who gets the award are flipped between actors/actresses and the other awards categories. We're going to have to keep this in mind going forward!

oscars2018_true_groups[-1].find_all('div',{'class':"views-field views-field-field-actor-name"})

oscars2018_true_groups[-1].find_all('div',{'class':"views-field views-field-title"})

# ### Writing the core parser functionality
#
# How will we map the contents of the HTML to the 
#
# * **Year**: All the awards are from the same year, also in the URL
# * **Category**: `<h2>`
# * **Nominee**: `<h4>` for actors, `<span>` for non-actors
# * **Movie**: `<span>` for actors, `<h4>` for non-actors
# * **Won**: `<h3>` for sibling, 0 for everyone else; alternatively just the top nominee

oscars2018_true_groups[0]

category = oscars2018_true_groups[0].find_all('h2')[0].text
print("The name of the category is:",category)

names = []
for _nominee in oscars2018_true_groups[0].find_all('h4'):
    nominee_name = _nominee.text
    names.append(nominee_name)
    print("The name of a nominee is:",nominee_name)

# +
movies = []

for _movie in oscars2018_true_groups[0].find_all('span'):
    if len(_movie.text) > 0:
        movie_name = _movie.text.strip()
        movies.append(movie_name)
        print("The name of a movie is:",movie_name)
# -

# One strategy is to use Python's [`zip`](https://docs.python.org/3.7/library/functions.html#zip) library to combine elements from different lists together. But `zip` is a bit too slick and abstract for my tastes.

# The elements of each list being combined need to be the same size
# So we make a list of the category name and multiply it by 5 to make it the same size as the others
list(zip([category]*5,names,movies))

# Another strategy is to use the `<div class="views-row">`s for each nominee and extract the relevant information from its subdivs. This is a bit more intuitive in the sense of reading from top to bottom and also makes it easier to capture the winner and losers based on position.

# +
actor_nominees = oscars2018_true_groups[0].find_all('div',{'class':'views-row'})

for i,nominee in enumerate(actor_nominees):
    
    # If in the first position, the nominee won
    if i == 0:
        winner = 'Won'
    # Otherwise, the nominee lost
    else:
        winner = 'Lost'
    
    # Get a list of all the sub-divs
    subdivs = nominee.find_all('div')
    
    # The first subdiv (for an actor) is the name
    name = subdivs[0].text.strip()
    
    # The second subdiv (for an actor) is the movie name
    movie = subdivs[1].text.strip()
    
    print("{0} was nominated for \"{1}\" and {2}.".format(name,movie,winner))
# -

# Check that reversing "movie" and "name" works for another award category like original screenplay (`oscars2018_true_groups[-1]`). There's some weirdness with "Written by" and "Story by" filtering in here rather than simply names that may need to get fixed in the final calculation, but I would want to talk to a domain expert about the differences between these labels.

# +
original_screenplay_nominees = oscars2018_true_groups[-1].find_all('div',{'class':'views-row'})

for i,nominee in enumerate(original_screenplay_nominees):
    if i == 0:
        winner = 'Won'
    else:
        winner = 'Lost'
        
    subdivs = nominee.find_all('div')
    
    # movie and name reversed
    movie = subdivs[0].text.strip()
    name = subdivs[1].text.strip()
    
    print("{0} was nominated for \"{1}\" and {2}.".format(name,movie,winner))
# -

# This was just for Best Actors, now lets add another layer for all the different awards categories. We can see the movie name and awardee switch is important now since most of the categories are reversed.

for group in oscars2018_true_groups:
    category = group.find_all('h2')[0].text
    
    for i,nominee in enumerate(group.find_all('div',{'class':'views-row'})):
        if i == 0:
            winner = 'Won'
        else:
            winner = 'Lost'

        subdivs = nominee.find_all('div')
        name = subdivs[0].text.strip()
        movie = subdivs[1].text.strip()

        print("{0} was nominated in {1} for {2}\" and {3}.".format(name,category,movie,winner))

# Include some flow control, if the name "actor" or "actree" appears in the category title, then  do nominee name first and movie name second, otherwise do movie name first and nominee name second.

for group in oscars2018_true_groups:
    category = group.find_all('h2')[0].text
    
    if 'Actor' in category or 'Actress' in category:
    
        for i,nominee in enumerate(group.find_all('div',{'class':'views-row'})):
            if i == 0:
                winner = 'Won'
            else:
                winner = 'Lost'

            subdivs = nominee.find_all('div')
            name = subdivs[0].text.strip()
            movie = subdivs[1].text.strip()

            print("{0} was nominated in {1} for \"{2}\" and {3}.".format(name,category,movie,winner))
            
    else:
        
        for i,nominee in enumerate(group.find_all('div',{'class':'views-row'})):
            if i == 0:
                winner = 'Won'
            else:
                winner = 'Lost'

            subdivs = nominee.find_all('div')
            movie = subdivs[0].text.strip()
            name = subdivs[1].text.strip()

            print("\"{0}\" was nominated in {1} for {2} and {3}.".format(name,category,movie,winner))
            

# Rather than printing out the information, store it in `nominees_2018` so that we can turn it into a DataFrame.

# +
nominees_2018 = []

for group in oscars2018_true_groups:
    category = group.find_all('h2')[0].text
    
    if 'Actor' in category or 'Actress' in category:
    
        for i,nominee in enumerate(group.find_all('div',{'class':'views-row'})):
            if i == 0:
                winner = 'Won'
            else:
                winner = 'Lost'

            subdivs = nominee.find_all('div')
            name = subdivs[0].text.strip()
            movie = subdivs[1].text.strip()
            
            # Swap out the print
            # Make a payload for each nominee
            nominee_payload = {'Category':category,
                               'Name':name,
                               'Movie':movie,
                               'Year':2018, # We're only looking at 2018 right now
                               'Winner':winner}
            
            # Add the payload to the list of nominees at top
            nominees_2018.append(nominee_payload)
            
    else:
        
        for i,nominee in enumerate(group.find_all('div',{'class':'views-row'})):
            if i == 0:
                winner = 'Won'
            else:
                winner = 'Lost'

            subdivs = nominee.find_all('div')
            movie = subdivs[0].text.strip()
            name = subdivs[1].text.strip()

            # Swap out the print
            nominee_payload = {'Category':category,
                               'Name':name,
                               'Movie':movie,
                               'Year':2018,
                               'Winner':winner}
            
            nominees_2018.append(nominee_payload)
            
# -

# Moment of truth!

nominees_df = pd.DataFrame(nominees_2018)
nominees_df


# Now let's turn this hulking beast of a parser into a function so we can apply it to other years' nominees in the next step.

def parse_nominees(true_groups,year):
    nominees_list = []

    for group in true_groups:
        category = group.find_all('h2')[0].text

        if 'Actor' in category or 'Actress' in category:

            for i,nominee in enumerate(group.find_all('div',{'class':'views-row'})):
                if i == 0:
                    winner = 'Won'
                else:
                    winner = 'Lost'

                subdivs = nominee.find_all('div')
                name = subdivs[0].text.strip()
                movie = subdivs[1].text.strip()
                
                nominee_payload = {'Category':category,
                                   'Name':name,
                                   'Movie':movie,
                                   'Year':year, # We may look at other years
                                   'Winner':winner}

                nominees_list.append(nominee_payload)

        else:

            for i,nominee in enumerate(group.find_all('div',{'class':'views-row'})):
                if i == 0:
                    winner = 'Won'
                else:
                    winner = 'Lost'

                subdivs = nominee.find_all('div')
                movie = subdivs[0].text.strip()
                name = subdivs[1].text.strip()

                nominee_payload = {'Category':category,
                                   'Name':name,
                                   'Movie':movie,
                                   'Year':year,
                                   'Winner':winner}

                nominees_list.append(nominee_payload)
    
    return nominees_list


# ## Iterating vs. parsing to retrieve data
#
# Often the data you are interested in is spread across multiple web pages. In an ideal world, the naming conventions would let you retrieve the data from these pages systematically. In the case of the Oscars, the URLs appear to be consistently formatted: `https://www.oscars.org/oscars/ceremonies/2019` suggests that we could change the 2019 to any other date going back to the start of the Oscars and get that year as well: `https://www.oscars.org/oscars/ceremonies/2018` should get us the page for 2018, and so on. Let's demonstrate each of these strategies with the Oscars data: iterating from 2019 back to 1929 in the URL versus parsing the list of links from the header.
#
# ### Iterating strategies for retrieving data
#
# The fundamental assumption with this strategy is that the data are stored at URLs in a consistent way that we can access sequentially. In the case of the Oscars, we *should* be able to simply pass each year to the URL in requests. Here we want to practice responsible data scraping by including a sleep between each request so that we do not overwhelm the Oscars server with requests. We can use the `sleep` function within `time`.

from time import sleep

# The `sleep(3)` below prevents any more code from progressing for 3 seconds.

print("The start of something.")
sleep(3)
print("The end of something.")

# The core part of the iterating strategy is simply using Python's [`range`](https://docs.python.org/3.7/library/functions.html#func-range) function to generate a sequence of values. Here, we can use `range` to print out a sequence of URLs that should correspond to awards pages from 2010 through 2019. We can also incorporate the `sleep` functionality and wait a second between each `print` statement—it should now take 10 seconds for this code to finish printing. This simulates how we can use `sleep` to slow down and spread out requests so that we do not overwhelm the servers whose data we are trying to scrape.

for year in range(2010,2020):
    sleep(1)
    print('https://www.oscars.org/oscars/ceremonies/{0}'.format(year))

# We defined a function `parse_nominees` above that takes the "true groups" of nominees. Let's try to tie these pieces together for all the nominees in the 2010s.

# +
# Create an empty list to store the data we get
all_years_nominees = dict()

# For each year starting in 2010 until 2019
for year in range(2010,2020):
    
    # Pause for a second between each request
    sleep(1)
    
    # Get the raw HTML
    year_raw_html = requests.get('https://www.oscars.org/oscars/ceremonies/{0}'.format(year)).text

    # Soup-ify
    year_souped_html = BeautifulSoup(year_raw_html)
    
    # Get the parent group
    year_parent_group = year_souped_html.find_all('div',{'id':'quicktabs-tabpage-honorees-0'})
    
    # Get the true groups under the parent group
    year_true_groups = year_parent_group[0].find_all('div',{'class':'view-grouping'})
    
    # Use our parsing function, passing the year from above
    year_nominees = parse_nominees(year_true_groups,year)
    
    # Convert the year_nominees to a DataFrame and add them to all_years_nominees
    all_years_nominees[year] = pd.DataFrame(year_nominees)
# -

# Combine each of the DataFrames in `all_years_nominees` into a giant DataFrame of all the nominees from 2010-2019.

all_years_nominees_df = pd.concat(all_years_nominees)
all_years_nominees_df.reset_index(drop=True).head(10)

# ### Parsing strategy for retrieving data
#
# Frustratingly, this iterating strategy may not always hold: maybe some years are skipped or the naming convention changes at some point. We will cover some basics of [error-handling in Python](https://realpython.com/python-exceptions/) that could let us work around errors as they pop up, but this may result in an incomplete collection if the naming conventions are systematic. What we would want to do is to identify all the links ahead of time by parsing them from list and then work through that list to get the complete data collection.
#
# What this means in the context of our Oscars example is assuming that we cannot trust that the sequential numbering of the years is a realiable guide to get all the data. Instead, we should get a list of the URLs for each of the awards pages from the "ceremonies-decade-scroller" (from Inspect) at the top. This scroller *should* be consistent across all the pages, but start with the nominees for 2019 just to be safe:

oscars2019_raw = requests.get('https://www.oscars.org/oscars/ceremonies/2019').text
oscars2019_soup = BeautifulSoup(oscars2019_raw)

# Using the Inspect tool, there is a `<div class="years">` that contains the links to each of the years. Run a `.find_all` to get all these href locations.

# +
# Get the <div class="years"> as a parent tag first, just in case there are <a class="years"> elsewhere
oscars2019_years_div = oscars2019_soup.find_all('div',{'class':'years'})[0]

# Now get the <a class="years"> underneath only the oscars2019_years_div
oscars2019_years_a = oscars2019_years_div.find_all('a',{'class':'year'})

# Inspect the first 10
oscars2019_years_a[:10]
# -

# Each of these `<a>` tags contains an "href", or the URL element where the page lives, and a text element for what's displayed.

oscars2019_years_a[0]['href']

oscars2019_years_a[0].text

# Now we can write a loop to print out the URL locations for all the other award years based on the "official" links in the "ceremonies-decade-scroller" navigation rather than assuming the years are sequential—I promise this will pay dividends in the future when inconsistent design wreaks havoc on your sequential data strategies!

for a in oscars2019_years_a[-10:]:
    href = a['href']
    print('https://www.oscars.org' + href)

# We can now use the `parse_nominees` function for these pages as well.

# +
# Create an empty list to store the data we get
all_years_nominees = dict()

# For the 10 most recent years
for a in oscars2019_years_a[-10:]:
    
    # Pause for a second between each request
    sleep(1)
    
    # Get the href
    href = a['href']
    
    # Get the year
    year = a.text
    
    # Get the raw HTML
    url = 'https://www.oscars.org' + href
    year_raw_html = requests.get(url).text

    # Soup-ify
    year_souped_html = BeautifulSoup(year_raw_html)
    
    # Get the parent group
    year_parent_group = year_souped_html.find_all('div',{'id':'quicktabs-tabpage-honorees-0'})
    
    # Get the true groups under the parent group
    year_true_groups = year_parent_group[0].find_all('div',{'class':'view-grouping'})
    
    # Use our parsing function, passing the year from above
    year_nominees = parse_nominees(year_true_groups,year)
    
    # Convert the year_nominees to a DataFrame and add them to all_years_nominees
    all_years_nominees[year] = pd.DataFrame(year_nominees)
# -

# Combine each of the DataFrames in `all_years_nominees` into a giant DataFrame of all the nominees from 2010-2019.

all_years_nominees_df = pd.concat(all_years_nominees)
all_years_nominees_df.reset_index(drop=True).head(10)

# ## Project time
#
# Let's take a look at websites you're interested in scraping and see what kinds of challenges and opportunities exist for scraping their data.



StarcoderdataJupyterScriptsDedupFiltered