# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .r
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: R
#     language: R
#     name: ir
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/xvariable/rvest_tutorial/blob/master/rvest_tutorial_dmi_winterschool_2020.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] colab_type="text" id="L7KD3mNCnVV_"
# # Tutorial Webscraping in R
#
# <NAME>, M.A. & Dr. <NAME><br />Hanover University of Music, Drama and Media | Department of Journalism and Communication Research <br /><br />Welcome to our tutorial! 
#

# + [markdown] id="tz8XS812typW" colab_type="text"
# #0. Prerequisites
#
# *   Tutorial code in jupyter notebook, download here:
# *   Register for https://colab.research.google.com, put your copy of the notebook there and open it
# * Selector Gadget plugin for Google Chrome: https://chrome.google.com/webstore/detail/selectorgadget/

# + [markdown] colab_type="text" id="Kjs_G23PkzTI"
# # 1. What is Webscraping?
#
#
# <b>Saving data from websites automatically</b> <br /> Step 1: Calling up a website <br />Step 2: Extraction of relevant content<br /><br /> <b> Step 1: Calling up a website</b> <br /> Most ideally, all data is located on one website and can be extracted directly. If the data is located on several web pages, links can also be extracted and called up one after the other.<br /><br /><b>Step 2: Extraction of relevant content</b><br />Textual data on websites is available as HTML. Using HTML tags and CSS attributes, you can select specific content elements and save it.
#
#
#

# + [markdown] colab_type="text" id="CujjG4fGofXy"
# # 2. HTML & CSS
#
#
# <b>HTML (Hypertext Markup Language)</b> <br /> 
# <li> Markup language that can be used to describe how digital documents are semantically structured </li> 
# <li>Formatting by web browser or CSS</li> 
# <li> Based on XML </li> 
# <li>Setup with "tags" </li><br /> 
# <b>CSS(Cascading Style Sheets)</b>
# <li>Design instructions for the HTML</li><br /><b>
#
# Example HTML:
# ```
# <html>
# <body>
# <h1>Star Wars Intro</h1>
# <p class = "sw_text">A long time ago in a galaxy far, far away....</p>
# ...
# ```
#
# Example CSS class:
# ```
# p.error {
#   font-family: courier;
#   color: red;
#   font-size: 160%;
# }
# ```
#
#
#
# </b><b>Exercise</b></b> <br />Visit https://www.imdb.com/title/tt2527338/?ref_=nv_sr_srsg_1, activate your Inspector Gadget in Chrome. <li> Find the HTML-Tag or CSS class that identifies the Rating (6,9) of Star Wars Episode IX <li>How can you select several elements at once? <li>How can you deselect elements?</li>
#
#
#

# + [markdown] colab_type="text" id="J9st0d_dsOly"
# # 3. Webscraping with R | Scraping content from a single web page
#
# For Webscraping you need the R package rvest (Easily Ha<b>rvest</b> (Scrape) web pages)<br /> Important notes and documentation on rvest: https://cran.r-project.org/web/packages/rvest/rvest.pdf<br /> <br /> We practice web scraping using the website of the EU Parliament. As a first exercise we want to scrape the names of the parliament members. First we want to scrape only the names that are visible on the first page of the overview (https://www.europarl.europa.eu/meps/en/full-list). <br /> <br />Run the following codes to scrape the names of the parliament members visibly on the first page
#
#
#
#

# + id="itPE53lLtt5H" colab_type="code" outputId="2f93d5d3-72d6-4a9f-cadf-414356a67f19" colab={"base_uri": "https://localhost:8080/", "height": 118}
# install.package("rvest"))
library(rvest)

# Store the URL you want to scrape as a new object 
eu_url <- "https://www.europarl.europa.eu/meps/en/full-list"

# Download HTML code from that URL and save it in a new object
eu_html <- read_html(eu_url, encoding = "UTF-8")

# View the source code we just downloaded: 
eu_html

# + [markdown] id="qiB54dWsdv3P" colab_type="text"
# Now we have extracted the whole HTML code of the web page. But if we only want to extract certain content - such as the names of the members of parliament - we have to address certain HTML nodes that contain this content. 
#
# Since we don´t know how the node could be identifies we use the Inspector gadget in Chrome and identify the CSS class that is used on the site to format parlamentarians names.

# + id="mmSI8n_Q2v6S" colab_type="code" outputId="dbb26659-fabb-4bc6-a3fe-75d22104650f" colab={"base_uri": "https://localhost:8080/", "height": 386}
# Exercise: Find the CSS class used for the names of parliament members
eu_names_node <- html_nodes(eu_html,"FIND THE CSS class") 
eu_names_node <- html_nodes(eu_html,".member-name") 

# View the list of html_nodes:
eu_names_node

# + [markdown] id="D8UYfyi_e2Zl" colab_type="text"
# Alright, the names are here. But there are still the HTML tags arouns the names of the parlamentarians. To get rid of any HTML we can use the function `html_text()`:

# + id="lY6FTG0i5FdL" colab_type="code" outputId="062f35c7-e581-4d11-ae7e-ebf95409a4ca" colab={"base_uri": "https://localhost:8080/", "height": 666}
#Now we can extract the text of the specific node
eu_parl_names <- html_text(eu_names_node)

#Viewing list of names:
eu_parl_names

# + [markdown] id="MO4kUTkVXbDb" colab_type="text"
# <b>Exercise</b><br /> Now you now how to scrape text from a single webpage. Try to reproduce the code and scrape the country of origin of every parliament member (instead of just the name).

# + colab_type="code" id="r71uMUGNXUcZ" outputId="4f7b5680-3c30-4e65-b7f2-70635d09a7c5" colab={"base_uri": "https://localhost:8080/", "height": 128}
#Fill in your code (you can copy the code from above and simply change the specific elements)
eu_parl_countries <- # YOUR CODE HERE

# We can now combine the two variables together and have a nice little dataset:
data <- cbind(eu_parl_names, eu_parl_countries)
head(data)

# + [markdown] colab_type="text" id="uegV-LznZUxg"
# # 4. Webscraping with R | Scraping text from multiple web pages
#
# By now you know how to scrape text from a single URL. But what if we don't want to scrape the names of the members of parliament, but their date of birth. This is not on the first page, but on the bottom of each Member's page. <br /><br />To do so, we need to visit the subpage of every parlamentarian and extract the birth date from there. But first we need a list of the links to all subpages we want to visit.
# Afterwards we can use a loop to visit each subpage one after the other and extract the date of birth from there.
#
# But how do we get a list of the URLs? They are not written on the page, but obviusly they are somewhere in the source code. Links are an attribute of the HTML tag `<a>`. We can extract attributes of tags using the rvest function `html_attr()`.
#

# + id="xiaIXheL9BFM" colab_type="code" outputId="883e9f9f-cb71-4f7a-9efd-111266bba0e5" colab={"base_uri": "https://localhost:8080/", "height": 666}
# We nee a slightly different node this time:
eu_links_nodes <- html_nodes(eu_html,".single-member-container a") 

# Now using the html_attr() function on the new nodes
# we are telling it to extract the "href" attribute:
eu_links <- html_attr(eu_links_nodes, "href")

# We need to paste the url to the links:
eu_links <- paste0("https://www.europarl.europa.eu", eu_links)

# View the list of links:
eu_links

# + [markdown] id="oDX6_NV5t31H" colab_type="text"
# Now we have the URLs to the subpages of the members of parliament. We can use the `read_html()` function again to scrape text (birth dates) from each subpage. 
# But since we want to scrape several pages, we would have to adapt the code for each page and execute them one after the other. We can work around this with a for loop that automatically executes the same code for each page in turn.

# + id="YlMbXicn-RGu" colab_type="code" outputId="346439ef-2a6e-42a3-a2f9-456e68435542" colab={"base_uri": "https://localhost:8080/", "height": 134}
# Before we start our for loop, we have to create an empty container object. 
# This empty container can be filled with specific content (in our case the dates of birth) through our for loop.
eu_birthdates <- as.character()

# In our for loop, we must define how often the element i of the link list is passed through. 
for (i in 1:length(eu_links)){

  # Download html of current eu parlamentarian subpage
  html_i <- read_html(eu_links[i], encoding = "UTF-8")

  # Find the birth date node (inspector gadget)
  birth_node_i <- html_nodes(html_i, "#birthDate")

  # Now we extract the text of our specific node
  birth_i <- html_text(birth_node_i)

  # Not every parlamentarian has a birth date on their website. Therefore we need to check
  if (length(birth_i) == 0) {
    birth_i <- NA
  }

  print(birth_i)

  # We append the new birth date to our previously empty container object
  eu_birthdates <- append(eu_birthdates, birth_i)
}

# We now have a list of names and a list of birth dates. 
# Let´s bind them to our data set
data <- cbind(data, eu_birthdates)

# Outputting content to our console
head(data)

# Within a for loop you can include multiple nodes from which you want to extract content. 
# For each content you want to scrape, you only need to create an empty container variable first, which is then filled in the for loop. 

# + [markdown] colab_type="text" id="mdgYWZLLZh1K"
# <b>Exercise</b><br /> We have only scraped the subpages of parlamentarians who are on the first page. And we know only two things about them: Their name and their birth date. 
#
# But what if that's not enough? We probably want to create a dataset of all members of the parliament (from a - z). Also, we need more information, e.g. on their parlamentary group and country. 
#
# This is your task! Use the following script to create a small dataset of all eu parlamentarians including the variables name, birth_date, group, country and party. 

# + id="yMuRAuaXx7YE" colab_type="code" colab={}
# First find the right link for your URL
url <- "INSERT LINK HERE"

# Now extract the variables eu_names, eu_group, eu_country and eu_party
eu_names_nodes <- html_nodes(url, "")
eu_names <- html_text()

eu_group_nodes <- 
...

eu_country_nodes <- 
...

eu_party_nodes <- 
...

# Insert a for loop that scrapes all the birth dates
eu_birhdates <- as.character()
for...

# combine all variables into one matrix
data <- 

# View dataset
head(data)

# + [markdown] colab_type="text" id="38OwvXDluk2S"
# # 5. Analysis
#
# What to do with your scraped data? For example, you can calculate the average age per parliamentary group. 
#

# + id="U2aeYEE3vUg5" colab_type="code" outputId="2b7aeb7b-8f0b-4fb4-d9d5-f7abe95fea7a" colab={"base_uri": "https://localhost:8080/", "height": 162}
library(stringr)
age <- 2020 - as.integer(substr(eu_birthdates, 7, 10)) 
summary(age)

# Now we have the age of every parliament member. If we want to know the average age per parliamentary group, we need to differ between the groups
tapply(age, eu_country, mean)

data <- as.data.frame(cbind(eu_country,age))

# Finally, we can visualize our means
ggplot(filter(data, !is.na(age)) + 
  stat_summary(
    aes(x = eu_country, y = age),
    fun.ymin = min,
    fun.ymax = max,
    fun.y = median, 
  )

# + id="4sqEUdg0qnfl" colab_type="code" colab={}


# + [markdown] id="zEeJ62EeRYWF" colab_type="text"
# # 6. Outlook
#
# <b>Scraping as suitable method in the post-API era</b><li> Web scraping offers a good opportunity for data collection - even in the post-API era</li><li> Web scraping is particularly easy when a website is properly programmed</li><li>Always allow enough time for data cleansing (often takes longer with scraped data)</li><li>Besides extracting text and links, it is also possible to fill out and send forms</li> <br /><b>Ethical and legal aspects</b><li>It is particularly important to act in a legally and ethically responsible manner (e.g. personal data) <li>Before scraping, check the terms of a website. If scraping is prohibited, the page should not be scraped<li>Never scrape data behind a login or paywall (even if you technically could)</b><br /><br /><b>Using Xpath</b><li>XPath can be used to navigate through elements and attributes in an XML document</li><li>With XPath, you can extract data based on text elements' contents, and not only on the page structure <li>So when you are scraping the web and you run into a hard-to-scrape website, XPath may just save the day</li><li>You can use the Selector Gadget to find the Xpath (just click on Xpath in the box)</li><li>Instead of using the CSS class in your code <b>html_nodes(object_name, "CSS class")</b> you use the Xpath <b>html_nodes(object_name, xpath='XpathClass')</b></li><br />Example: You want to scrape a specific element of a web page. The CSS class of this specific element is assigned a multiple times on the web page. Therefore, more than the specific element is scraped with the CSS class. Using Xpath is your solution because the Xpath is unique for every element on a web page. <br /><br /><b>Further reading</b><br />
# <NAME>., <NAME>., <NAME>., & <NAME>. (2015). Automated Data Collection with R: A practical Guide to Web Scraping and Text Mining. Chichester, United Kingdom: <NAME>iley & Sons. 
#
# For data wrangling and cleansing see https://r4ds.had.co.nz/
#
#

# + id="CjPNEIyc1vHo" colab_type="code" colab={}


StarcoderdataJupyterScriptsDedupFiltered