Scraping tables from a JavaScript webpage using Selenium, BeautifulSoup, and Pandas
Running the script shows an error “ValueError: No tables found”. This is because Pandas read_html() function searches for <table> elements and can’t find any from the webpage.

If we look at the page source and search for <table , it shows 0 result.

image by author

The reason why we can’t find any table when looking at page source is that the browser is executing JavaScript code that loads content from a remote server and renders tables. This means if we try just scrapping the HTML, the JavaScript won’t be executed, and thus, the Pandas read_html() won’t find any <table> element.

This brings us to use Selenium.

If you are looking for Pandas read_html() tutorial, please check out:

2. Install libraries and download Selenium web driver

Before we move onto install libraries, it is highly recommended to create an isolated Python environment to avoid any conflict.

Please check out the following articles for creating Python Virtual Environment.

Install libraries

In order to run this example, you need to install selenium, beautifulsoup4, and Pandas. If you have pip on your system, you can simply install them by

pip install selenium

pip install beautifulsoup4

pip install pandas

Download Selenium web driver

Selenium requires a driver to interface with the chosen browser. For example, Chrome requires ChromeDriver, which needs to be installed before running the example.

To install ChromeDriver, you should:

download the same version ChromeDriver as the Chrome you are using.

place the ChromeDriver executable in your PATH, for example, place it in /usr/bin or /usr/local/bin .

You will get a WebDriverException error if you don’t have a proper Selenium web driver in your PATH.

Other supported browsers have their own drivers, please check out their drivers below

3. Scraping tables using Selenium, BeautifulSoup, and Pandas

We will be scrapping tables from the following article.

Let’s import libraries and specify the url .

import pandas as pd

from selenium import webdriver

from bs4 import BeautifulSoup url = ' https://pubs.rsc.org/en/content/articlelanding/2020/na/d0na00118j

Step 1: Create a session and load the page

The first step we need to do is to create a web driver session, for example, a new Chrome session.

# Create a new Chrome session

driver = webdriver.Chrome()

If your web driver executable is not available in your PATH, you are able to define a custom executable path by setting the argument executable_path

# Create a new Chrome session with a custom executable path

driver = webdriver.Chrome(executable_path='path_to_web_driver/')

Once set up, we can now load the web page. We also call implicitly_wait(5) to wait for 5 seconds for the page to fully load.

# Load the web page

driver.get(url) # Wait for the page to fully load

driver. implicitly_wait(5)

If we run the script now (You should add driver.quit() at the end to ensure the browser closes), it will launch a Chrome window and open the URL specified. Hopefully, you should see page content load up before the script quits the session.

Step 2: Parse HTML code and grab tables with Beautiful Soup

Next, we need to scrape information from the web page. Beautiful Soup is one of the best ways to traverse the DOM and scrape the data.

In this tutorial, we are going to use lxml parser. If you have pip on your system, you can simply install it by

pip install lxml

By running the following statement, Beautiful Soup grabs all page source on the page.

soup = BeautifulSoup(driver.page_source, 'lxml')

Next, it is time to ask Beautiful Soup to find all the tables on the page.

tables = soup.find_all('table')

By now, you should be able to see a list of tables if running with debugging mode.

image by author

Step 3: Read tables with Pandas read_html()

Next, we need to give the HTML tables to Pandas to put them in a DataFrames object.

Pandas already have a built-in function read_html() to convert the table on the web to a DataFrame.

dfs = pd.read_html(str(tables))

The result we get is not a Pandas DataFrame but a Python list. If we want to get the table, we can use the index to access it:

print(dfs[0]) Samples n (1019 cm−3) μH (cm2 V−1 s−1) σ (S cm−1)

0 Bi2Te3 1.33 59.08 125.7

1 0.5% rGO/Bi2Te3 1.26 79.79 160.8

2 1% rGO/Bi2Te3 1.21 103.27 199.9

3 2% rGO/Bi2Te3 1.17 138.79 259.8

4 rGO film 3.59 48.25 277.3

The full Python code

Here is the full Python code.

Conclusion

This article shows a method we can use to scrape tables from a JavaScript web. Web scraping with Selenium, Beautiful Soup, and Pandas is an excellent tool to have within your skillset.

I hope this article will help you to save time in scrapping data from the webpage. I recommend you to check out the documentation for the Selenium and Beautiful Soup, and to know about other things you can do.

Thanks for reading

You may be interested in some of my other articles:

More tutorials can be found on my Github
['B. Chen']
['Python', 'Web Scraping', 'Selenium', 'Data Science', 'Pandas']