# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# ## Beautiful Soup: Build a Web Scraper With Python
#
# ## What Is Web Scraping?
#
# **Web scraping** - is the process of gathering information from the Internet. Even copying and pasting the lyrics of your favorite song is a form of web scraping! However, the words “web scraping” usually refer to a process that involves automation. Some websites don’t like it when automatic scrapers gather their data, while others don’t mind.
#
# If you’re scraping a page respectfully for educational purposes, then you’re unlikely to have any problems. Still, it’s a good idea to do some research on your own and make sure that you’re not violating any Terms of Service before you start a large-scale project.
#
# ## Challenges of Web Scraping
#
# The Web has grown organically out of many sources. It combines many different technologies, styles, and personalities, and it continues to grow to this day. In other words, the Web is a hot mess! Because of this, you’ll run into some challenges when scraping the Web:
#
# * **Variety**: Every website is different. While you’ll encounter general structures that repeat themselves, each website is unique and will need personal treatment if you want to extract the relevant information.
# * **Durability**: Websites constantly change. Say you’ve built a shiny new web scraper that automatically cherry-picks what you want from your resource of interest. The 1st time you run your script, it works flawlessly. But when you run the same script only a short while later, you run into a discouraging and lengthy stack of tracebacks!
#
# Unstable scripts are a realistic scenario, as many websites are in active development. Once the site’s structure has changed, your scraper might not be able to navigate the sitemap correctly or find the relevant information. The good news is that many changes to websites are small and incremental, so you’ll likely be able to update your scraper with only minimal adjustments.
#
# However, keep in mind that because the Internet is dynamic, the scrapers you’ll build will probably require constant maintenance. You can set up continuous integration to run scraping tests periodically to ensure that your main script doesn’t break without your knowledge.
#
# ## An Alternative to Web Scraping: APIs
#
# Some website providers offer **application programming interfaces (APIs)** that allow you to access their data in a predefined manner. With APIs, you can avoid parsing HTML. Instead, you can access the data directly using formats like JSON and XML. HTML is primarily a way to present content to users visually.
#
# When you use an API, the process is generally more stable than gathering the data through web scraping. That’s because developers create APIs to be consumed by programs rather than by human eyes.
#
# The front-end presentation of a site might change often, but such a change in the website’s design doesn’t affect its API structure. The structure of an API is usually more permanent, which means it’s a more reliable source of the site’s data.
#
# However, APIs can change as well. The challenges of both variety and durability apply to APIs just as they do to websites. Additionally, it’s much harder to inspect the structure of an API by yourself if the provided documentation lacks quality.
#
# ## Scrape the Fake Python Job Site
#
# In this tutorial, you’ll build a web scraper that fetches Python software developer job listings from the Fake Python Jobs site. It’s an example site with fake job postings that you can freely scrape to train your skills. Your web scraper will parse the HTML on the site to pick out the relevant information and filter that content for specific words.
#
# ## Step 1: Inspect Your Data Source
#
# Before you write any Python code, you need to get to know the website that you want to scrape. You’ll need to understand the site structure to extract the information that’s relevant for you. Start by opening the site you want to scrape with your favorite browser.
#
# ## Decipher the Information in URLs
#
# A programmer can encode a lot of information in a URL. Your web scraping journey will be much easier if you 1st become familiar with how URLs work and what they’re made of. For example, you might find yourself on a details page that has the following URL:
#
# `https://realpython.github.io/fake-jobs/jobs/senior-python-developer-0.html`
#
# You can deconstruct the above URL into two main parts:
#
# The **base URL** - represents the path to the search functionality of the website. In the example above, the base URL is `https://realpython.github.io/fake-jobs/`. The specific site location that ends with `.html` is the path to the job description’s unique resource. Any job posted on this website will use the same base URL. However, the unique resources’ location will be different depending on what specific job posting you’re viewing.
#
# URLs can hold more information than just the location of a file, some websites use **query parameters** to encode values that you submit when performing a search. You can think of them as query strings that you send to the database to retrieve specific records.
#
# You’ll find query parameters at the end of a URL. For example, if you go to Indeed and search for “software developer” in “Australia” through their search bar, you’ll see that the URL changes to include these values as query parameters:
#
# `https://au.indeed.com/jobs?q=software+developer&l=Australia`
#
# The query parameters in this URL are `?q=software+developer&l=Australia`. Query parameters consist of 3 parts:
#
# 1. **Start**: The beginning of the query parameters is denoted by a question mark (`?`).
# 2. **Information**: The pieces of information constituting 1 query parameter are encoded in key-value pairs, where related keys and values are joined together by an equals sign (`key=value`).
# 3. **Separator**: Every URL can have multiple query parameters, separated by an ampersand symbol (`&`).
#
# Equipped with this information, you can pick apart the URL’s query parameters into two key-value pairs:
#
# * `q=software+developer` - selects the type of job
# * `l=Australia` - selects the location of the job
#
# ## Inspect the Site Using Developer Tools
#
# You’ll need to understand the page structure to pick what you want from the HTML response that you’ll collect in one of the upcoming steps. 
#
# **Developer tools** can help you understand the structure of a website. In Chrome on macOS, you can open up the developer tools through the menu by selecting **View → Developer → Developer Tools**. On Windows and Linux, you can access them by clicking the top-right menu button (`⋮`) and selecting **More Tools → Developer Tools**. You can also access your developer tools by right-clicking on the page and selecting the Inspect option or using a keyboard shortcut:
#
# * Mac: `Cmd+Alt+I`
# * Windows/Linux: `Ctrl+Shift+I`
#
# **Developer tools** - allow you to interactively explore the site’s **document object model** (**DOM**) to better understand your source.
#
# ## Step 2: Scrape HTML Content From a Page
#
# First, you’ll want to get the site’s HTML code into your Python script so that you can interact with it. For this task, you’ll use Python’s `requests` library:

# +
import requests

URL = "https://realpython.github.io/fake-jobs/"
page = requests.get(URL)

print(page.text[0:350], '\n...')
# -

# This code retrieves the HTML data that the server sends back and stores that data in a Python object.
#
# ## Static Websites
#
# The website that you’re scraping in this tutorial serves **static HTML content**. The server that hosts the site sends back HTML documents that **already contain all the data that you’ll get to see as a user**.
#
# The HTML of this job board has descriptive **class names** on the elements that you’re interested in:
#
# * `class="title is-5"` - contains the title of the job posting
# * `class="subtitle is-6 company"` - contains the name of the company that offers the position
# * `class="location"` - contains the location where you’d be working
#
# However, there are more challenging situations that you might encounter when you’re scraping websites. Before you learn how to pick the relevant information from the HTML that you just scraped, you’ll take a quick look at 2 of these more challenging situations.
#
# ## Hidden Websites
#
# Some pages contain information that’s hidden behind a login. That means you’ll need an account to be able to scrape anything from the page. The process to make an HTTP request from your Python script is different from how you access a page from your browser. Just because you can log in to the page through your browser doesn’t mean you’ll be able to scrape it with your Python script.
#
# The `requests` library comes with the built-in capacity to handle authentication. With these techniques, you can log in to websites when making the HTTP request from your Python script and then scrape information that’s hidden behind a login.
#
# ## Dynamic Websites
#
# **Static sites** - are straightforward to work with because the server sends you an HTML page that already contains all the page information in the response. You can parse that HTML response and immediately begin to pick out the relevant data.
#
# On the other hand, with a **dynamic website**, the server might not send back any HTML at all. Instead, you could receive **JavaScript code as a response**. Many modern web applications are designed to provide their functionality in collaboration with the clients’ browsers. Instead of sending HTML pages, these apps send JavaScript code that instructs your browser to create the desired HTML. Web apps deliver dynamic content in this way to offload work from the server to the clients’ machines as well as to avoid page reloads and improve the overall user experience.
#
# What happens in the browser is not the same as what happens in your script. Your browser will diligently execute the JavaScript code it receives from a server and create the DOM and HTML for you locally. However, if you request a dynamic website in your Python script, then you won’t get the HTML page content.
#
# When you use `requests`, you only receive what the server sends back. In the case of a dynamic website, you’ll end up with some JavaScript code instead of HTML. **The only way to go from the JavaScript code you received to the content that you’re interested in is to execute the code**, just like your browser does. The `requests` library can’t do that for you, but there are other solutions that can.
#
# For example, `requests-html` is a project created by the author of the `requests` library that **allows you to render JavaScript** using syntax that’s similar to the syntax in `requests`. It also includes capabilities for parsing the data by using `Beautiful Soup` under the hood.
#
# **NOTE:** Another popular choice for scraping dynamic content is `Selenium`. You can think of `Selenium` as a slimmed-down browser that executes the JavaScript code for you before passing on the rendered HTML response to your script.
#
# ## Step 3: Parse HTML Code With Beautiful Soup
#
# You’ve successfully scraped some HTML from the Internet, but when you look at it, it just seems like a huge mess. There are tons of HTML elements here and there, thousands of attributes scattered around—and wasn’t there some JavaScript mixed in as well? It’s time to parse this lengthy code response with the help of Python to make it more accessible and pick out the data you want.
#
# `Beautiful Soup` - is a Python library for parsing structured data. It allows you to interact with HTML in a similar way to how you interact with a web page using developer tools. The library exposes a couple of intuitive functions you can use to explore the HTML you received. To get started, use your terminal to install Beautiful Soup:

# +
from bs4 import BeautifulSoup
import requests

URL = "https://realpython.github.io/fake-jobs/"
page = requests.get(URL)
print('type:', type(page.content), '\n-------\n', page.content[:350], '\n...', sep='')

soup = BeautifulSoup(page.content, "html.parser")
print('-------\ntype:', type(soup))
# -

# `BeautifulSoup` object takes `page.content`, which is the HTML content you scraped earlier, as its input.
#
# **NOTE:** You’ll want to pass `page.content` instead of `page.text` to avoid problems with character encoding. The `.content` attribute holds raw bytes, which can be decoded better than the text representation you printed earlier using the `.text` attribute.
#
# The 2nd argument, `"html.parser"`, makes sure that you use the appropriate parser for HTML content.
#
# ## Find Elements by ID
#
# In an HTML web page, every element can have an **id attribute** assigned. As the name already suggests, that **id attribute** - makes the element uniquely identifiable on the page. You can begin to parse your page by selecting a specific element by its ID.
#
# The element we’re looking for is a `<div>` with an id attribute that has the value `"ResultsContainer"`. It has some other attributes as well, but below is the gist of what you’re looking for:
#
# `<div id="ResultsContainer">`<br>
#     `<!-- all the job listings -->`<br>
# `</div>`
#
# Beautiful Soup allows you to find that specific HTML element by its ID:

# +
results = soup.find(id="ResultsContainer")

# For easier viewing, you can prettify any Beautiful Soup object 
# When you print it out. If you call `.prettify()` on the results variable 
# That you just assigned above, then you’ll see all the HTML contained within the <div>:

print(results.prettify()[:350], '\n...')

# When you use the element’s ID, you can pick out 1 element 
# From among the rest of the HTML. Now you can work with only this 
# Specific part of the page’s HTML. It looks like the soup just 
# Got a little thinner! However, it’s still quite dense.
# -

# ## Find Elements by HTML Class Name
#
# You’ve seen that every job posting is wrapped in a `<div>` element with the `class` card-content. Now you can work with your new object called results and select only the job postings in it:

# +
# `find_all()` returns all the tags and strings that match your filters
# This can take a while if the document is large.
# `.find_all()` - returns an iterable containing all the HTML 
# For all the job listings displayed on that page.
job_elements = results.find_all("div", class_="card-content")

for job_element in job_elements[:2]:
    print(job_element, end="\n"*2)
print('--------------------------------------')

# There’s still a lot of HTML! You saw earlier that your page has descriptive class names
# On some elements. You can pick out those child elements 
# From each job posting with .find():

for job_element in job_elements[:2]:
    title_element = job_element.find("h2", class_="title")
    company_element = job_element.find("h3", class_="company")
    location_element = job_element.find("p", class_="location")
    print(title_element)
    print(company_element)
    print(location_element)
    print()
# -

# Each `job_element` is another `BeautifulSoup()` object. Therefore, you can use the same methods on it as you did on its parent element, results.
#
# With this code snippet, you’re getting closer and closer to the data that you’re actually interested in. Still, there’s a lot going on with all those HTML tags and attributes floating around:
#
# `<h2 class="title is-5">Senior Python Developer</h2>`<br>
# `<h3 class="subtitle is-6 company"><NAME> and Davis</h3>`<br>
# `<p class="location">Stewartbury, AA</p>`
#
# ## Extract Text From HTML Elements
#
# You only want to see the title, company, and location of each job posting. And behold! Beautiful Soup has got you covered. You can add `.text` to a Beautiful Soup object to return only the text content of the HTML elements that the object contains:

# +
for job_element in job_elements[:2]:
    title_element = job_element.find("h2", class_="title")
    company_element = job_element.find("h3", class_="company")
    location_element = job_element.find("p", class_="location")
    print(title_element.text)
    print(company_element.text)
    print(location_element.text)
    print()
print('--------------------')
# Run the above code snippet, and you’ll see the text of each element displayed. 
# However, it’s possible that you’ll also get some extra whitespace. 
# Since you’re now working with Python strings, you can `.strip()` 
# The superfluous whitespace. You can also apply any other familiar Python string methods 
# To further clean up your text:

for job_element in job_elements[:2]:
    title_element = job_element.find("h2", class_="title")
    company_element = job_element.find("h3", class_="company")
    location_element = job_element.find("p", class_="location")
    print(title_element.text.strip())
    print(company_element.text.strip())
    print(location_element.text.strip())
    print()
# -

# ## Find Elements by Class Name and Text Content
#
# Not all of the job listings are developer jobs. Instead of printing out all the jobs listed on the website, you’ll first filter them using keywords. You know that job titles in the page are kept within `<h2>` elements. To filter for only specific jobs, you can use the string argument:

# +
# This code finds all `<h2>` elements where the contained string matches "Python" exactly. 
# NOTE: that you’re directly calling the method on your 1st results variable. 
# If you go ahead and `print()` the output of the above code snippet to your console, 
# Then you might be disappointed because it’ll be empty:
python_jobs = results.find_all("h2", string="Python")

print(python_jobs)
# -

# There was a Python job in the search results, so why is it not showing up?
#
# When you use `string=` as you did above, your program looks for that string exactly. Any differences in the spelling, capitalization, or whitespace will prevent the element from matching. In the next section, you’ll find a way to make your search string more general.
#
# ## Pass a Function to a Beautiful Soup Method
#
# In addition to strings, you can sometimes pass functions as arguments to `BeautifulSoup` methods. You can change the previous line of code to use a function instead:

# +
# Now you’re passing an anonymous function to the `string=` argument. 
# The `lambda` function looks at the text of each `<h2>` element, 
# Converts it to lowercase, and checks whether the substring "python" is found anywhere. 
# You can check whether you managed to identify all the Python jobs with this approach:
python_jobs = results.find_all(
    "h2", string=lambda text: "python" in text.lower()
)

# Your program has found 10 matching job posts 
# That include the word "python" in their job title!
print(len(python_jobs))

# + [markdown] tags=[]
# Finding elements depending on their text content is a powerful way to filter your HTML response for specific information. `BeautifulSoup` allows you to use either exact strings or functions as arguments for filtering text in `BeautifulSoup` objects.
#
# However, when you try to run your scraper to print out the information of the filtered Python jobs, you’ll run into an error:
#
# `AttributeError: 'NoneType' object has no attribute 'text'`
#
# This message is a common error that you’ll run into a lot when you’re scraping information from the Internet. Inspect the HTML of an element in your `python_jobs` list. What does it look like? Where do you think the error is coming from?
#
# ## Identify Error Conditions
#
# When you look at a single element in `python_jobs`, you’ll see that it consists of only the `<h2>` element that contains the job title.
#
# When you revisit the code you used to select the items, you’ll see that that’s what you targeted. You filtered for only the `<h2>` title elements of the job postings that contain the word "`python`". As you can see, these elements don’t include the rest of the information about the job.
#
# You tried to find the job title, the company name, and the job’s location in each element in `python_jobs`, but each element contains only the job title text.
#
# Your diligent parsing library still looks for the other ones, too, and returns `None` because it can’t find them. Then, `print()` fails with the shown error message when you try to extract the `.text` attribute from one of these `None` objects.
#
# The text you’re looking for is nested in sibling elements of the `<h2>` elements your filter returned. `BeautifulSoup` can help you to select sibling, child, and parent elements of each `BeautifulSoup` object.
# -

# ## Access Parent Elements
#
# One way to get access to all the information you need is to step up in the hierarchy of the DOM starting from the `<h2>` elements that you identified.
#
# The `<div>` element with the card-content class contains all the information you want. It’s a third-level parent of the `<h2>` title element that you found using your filter.
#
# With this information in mind, you can now use the elements in `python_jobs` and fetch their great-grandparent elements instead to get access to all the information you want:

# +
python_jobs = results.find_all(
    "h2", string=lambda text: "python" in text.lower()
)

python_job_elements = [
    h2_element.parent.parent.parent for h2_element in python_jobs
]

print(python_job_elements[0])
# -

# You added a list comprehension that operates on each of the `<h2>` title elements in `python_jobs` that you got by filtering with the `lambda` expression. You’re selecting the parent element of the parent element of the parent element of each `<h2>` title element. That’s 3 generations up!
#
# When you were looking at the HTML of a single job posting, you identified that this specific parent element with the class name card-content contains all the information you need.
#
# Now you can adapt the code in your for loop to iterate over the parent elements instead:

for job_element in python_job_elements:
    # -- snip --

# When you run your script another time, you’ll see that your code once again has access to all the relevant information. That’s because you’re now looping over the `<div class="card-content">` elements instead of just the `<h2>` title elements.
#
# Using the `.parent` attribute that each `BeautifulSoup` object comes with gives you an intuitive way of stepping through your DOM structure and addressing the elements you need. You can also access child elements and sibling elements in a similar manner. 
#
# ## Extract Attributes From HTML Elements
#
# At this point, your Python script already scrapes the site and filters its HTML for relevant job postings. Well done! However, what’s still missing is the link to apply for a job.
#
# While you were inspecting the page, you found 2 links at the bottom of each card. If you handle the link elements in the same way as you handled the other elements, you won’t get the URLs that you’re interested in:

for job_element in python_job_elements[:2]:
    # -- snip --
    links = job_element.find_all("a")
    for link in links:
        print(link.text.strip())

# If you run this code snippet, then you’ll get the **link texts** Learn and Apply instead of the associated URLs.
#
# That’s because the `.text` attribute leaves only the visible content of an HTML element. It strips away all HTML tags, including the HTML attributes containing the URL, and leaves you with just the link text. To get the URL instead, you need to extract the value of one of the HTML attributes instead of discarding it.
#
# The URL of a link element is associated with the `href` attribute. The specific URL that you’re looking for is the value of the `href` attribute of the second `<a>` tag at the bottom the HTML of a single job posting:
#
# Start by fetching all the `<a>` elements in a job card. Then, extract the value of their `href` attributes using square-bracket notation:

for job_element in python_job_elements[:2]:
    # -- snip --
    links = job_element.find_all("a")
    for link in links:
        link_url = link["href"]
        print(f"Apply here: {link_url}\n")

# In this code snippet, you: 
#
# 1. First fetched all links from each of the filtered job postings
# 2. Then you extracted the href attribute, which contains the URL, using `["href"]`
# 3. Printed it to your console

StarcoderdataJupyterScriptsDedupFiltered