# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: 'Python 3.9.2 64-bit (''webscraping_and_data_labelling-mY7GX9EY'':
#       pipenv)'
#     name: python3
# ---

# + [markdown] pycharm={"name": "#%% md\n"}
# ##  Parsing HTML with BeautifulSoup
# -

# BeautifulSoup is a Python library for extracting data from HTML or XML files. Refer to the [documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) for more information. 

# +
# pipenv shell

# + pycharm={"name": "#%%\n"}
import requests
from bs4 import BeautifulSoup

# + pycharm={"name": "#%%\n"}
url = "https://www.bbc.com/pidgin/topics/c2dwqd1zr92t"

# + pycharm={"name": "#%%\n"}
response = requests.get(url)
print(dir(response))   # dir allows us to check the methods available for an object

# + pycharm={"name": "#%%\n"}
page_html = response.content
print(page_html)

# + pycharm={"name": "#%%\n"}
print(type(page_html))   # Without bs4, we obtain a string object which is tedious to interact with

# + pycharm={"name": "#%%\n"}
page_soup = BeautifulSoup(page_html, "html.parser") 
print("type: ",type(page_soup))
print(f"page soup: {page_soup}")
# -

# BeautifulSoup allows us to use a variety of parsers. You can check this out :) 

# + pycharm={"name": "#%%\n"}
print(len(page_soup))  # we get a rich list of methods available for our BeautifulSoup object
# -

# **`TODO`**: Complete the `get_page_soup` function in `scraper.py`

# ## Locating elements

# ### 1. Locating elements by tag

# + pycharm={"name": "#%%\n"}
all_urls = page_soup.findAll("a")

# + pycharm={"name": "#%%\n"}
print(len(all_urls))

# + pycharm={"name": "#%%\n"}
print(type(all_urls[0]))

# + pycharm={"name": "#%%\n"}
print(dir(all_urls[0]))
# -

all_urls[0].get("href")     # hrefs may be absolute

all_urls[12].get("href")     # hrefs may also be relative

all_urls

# ### 2. Locating elements by tag & class

# To obtain the class of an element ona web page, right click the element and select `Inspect`. 
#
# ![InspectElement](static/inspect_element.png)

headline = page_soup.find(
        "h1", attrs={"class": "bbc-13dm3d0 e1yj3cbb0"}
        )

# ### How many pages of article are there in a category?
# **`TODO`**: Go to any of the category pages of BBC Pidgin. Can you retrieve the `class` of the `span` element that contains the number of total pages as text? In the picture below, for example, retrieve the `class` of the `span` that contains `100` as text. 
#
# ![PageNumber](static/PageNumber.png)

# ## What is a valid article link?

# Website may have a pattern for successive pages in a category. This eases our work. Check through a number of articles. Do you notice a pattern to the article URLS?

for url in all_urls:
    href = url.get("href")
    
    if (href.startswith("/pidgin/tori") or \
        href.startswith("pidgin/world") or \
        href.startswith("pidgin/sport")) and href[-1].isdigit():
        print(href)

# valid_url = "https://www.bbc.com" + href

base_url = "https://www.bbc.com"

article = requests.get(# insert a valid URL here) 
# 

# **`TODO`**: Complete the `get_valid_urls` and `get_urls` functions in `scraper.py`

# ## Getting article text

response = requests.get("https://www.bbc.com/pidgin/sport-58110814")

page_html = response.text
page_soup = # TODO: Create a BeautifulSoup object from `page_html`

# Text on websites are frequently embedded within one or more `div` elements. Open up the article we requested above in your browser.`Inspect` any paragraph from the article. What do you notice about the `div` elements? What is the value of their `class` attribute?

story_div = page_soup.find_all(
        "div", attrs={"class": "bbc-19j92fr e57qer20"}
        )

story_div

# Inside `div` elements, text is usually written in `p` elements. BeautifulSoup allows us to search the `div` element we retrieved above for elements it contains. 

p_elements = [div.findAll("p") for div in story_div]
p_elements

p_elements[0]

p_elements[0][0]

p_elements[0][0].text

# **`TODO`**: Complete the `get_article_data` and `scrape` functions in `scraper.py`. 

# **`TODO`**: Complete the `get_parser` function in `scraper.py`

StarcoderdataJupyterScriptsDedupFiltered