Scrape data from website into dataframe(s) using Split function Outline:

This code uses the Split function to extract specific information from the following website: https://www.webscraper.io/test-sites/tables.

The required information are the four tables visible on the page with headers "#", "First Name","Last Name","Username".  I am extracting the information within these into 4 dataframes. 



Example table:





Description:

I use the requests library to make the GET request, and split the response text on "table table-bordered" to generate my individual table chunks. 

There is a fair amount of annoying fiddly indexing to get just the info I want, but the tutorial I am following requires the use of the Split function, and not something far more logical, to my mind, like Beautiful Soup, where I could just apply CSS selectors, for example, and grab what I want. The latter method would be less fragile as well.

I have written a function, GetTable, to parse the required information from each chunk and return a dataframe. There is a difference between the Split delimiter for table 1 versus 2-4. 

There isn't an awful lot of code but I would appreciate any pointers on improving the code I have written.

I am running this from Spyder 3.2.8 with Python 3.6.



Code:

def GetTable(tableChunk):
    split1 = tableChunk.split('tbody')[1]
    split2 = split1.split('&lt;table')[0]
    values = []

    aList = split2.split('&gt;\n\t\t\t\t&lt;') 
    if len(aList) !=1:
        for item in aList[1:]:
                values.append(item.split('&lt;/')[0].split('d&gt;'[1])[1])
    else:
        aList = split2.split('&lt;/td')
        for item in aList[:-1]:
           values.append(item.split('td&gt;')[1])

    headers =  ["#", "First Name", "Last Name", "User Name"]  
    numberOfColumns = len(headers)
    numberOfRows = int((len(values) / numberOfColumns))

    df = pd.DataFrame(np.array(values).reshape( numberOfRows, numberOfColumns ) , columns = headers)
    return df

import requests as req
import pandas as pd
import numpy as np

url = "http://webscraper.io/test-sites/tables"
response = req.get(url)
htmlText = response.text  
tableChunks = htmlText.split('table table-bordered')

for tableChunk in tableChunks[1:]:
   print(GetTable(tableChunk))
   print('\n')
Don't parse HTML manually, you should use the BeautifulSoup module!
import should be at the top of the file
Use a if __name__ == '__main__' guard
Functions and variable should be snake_case




First you can rewrite the getTable() alot using the BeautifulSoup module

import requests
from bs4 import BeautifulSoup

url = "http://webscraper.io/test-sites/tables"
soup = BeautifulSoup(requests.get(url).text, 'html.parser')
for table in soup.select('.table'):
    new_table = [[c.text for c in row.find_all('td')] for row in table.find_all('tr')]


The only problem is that it will also give back None values in the table, so we'd need to catch the None values and only yield when the list is not filled with None

Revised Code

import requests
import pandas as pd
from bs4 import BeautifulSoup

def parse_table(table):
    for row in table.find_all('tr'):
        col = [c.text for c in row.find_all('td')]
        if not all(c is None for c in col):
            yield col

def scrape_tables():
    url = "http://webscraper.io/test-sites/tables"
    soup = BeautifulSoup(requests.get(url).text, 'html.parser')
    for table in soup.select('.table'):
        parsed_table = [col for col in parse_table(table)]
        df = pd.DataFrame(parsed_table, columns=["#", "First Name", "Last Name", "User Name"])
        print()
        print(df)

if __name__ == '__main__':
    scrape_tables()