Python web-scraper to download table of transistor counts from Wikipedia I have been looking for answers for how to easily scrape data from Wikipedia into a CSV file with Beautiful Soup. This is the code so far. Is there an easier way to do it?

import urllib.request
from bs4 import BeautifulSoup
import csv


#the websites
urls = ['https://en.wikipedia.org/wiki/Transistor_count']
data =[]

#getting the websites and the data
for url in urls:   
##    my_url = requests.get(url)
    my_url = urllib.request.urlopen(url)

    html = my_url.read()
    soup = BeautifulSoup(html,'html.parser')

    My_table = soup.find('table',{'class':'wikitable sortable'})
    My_second_table = My_table.find_next_sibling('table') 

    with open('data.csv', 'w',encoding='UTF-8', newline='') as f:
        fields = ['Title', 'Year']
        writer = csv.writer(f, delimiter=',')
        writer.writerow(fields)


    with open('data.csv', "a", encoding='UTF-8') as csv_file:
        writer = csv.writer(csv_file, delimiter=',')    
        for tr in My_table.find_all('tr')[2:]: # [2:] is to skip empty and header 
            tds = tr.find_all('td')
            try:
                title = tds[0].text.replace('\n','')
            except:
                title = ""
            try:
                year = tds[2].text.replace('\n','')
            except:
                year = ""

            writer.writerow([title, year])

    with open('data.csv', "a", encoding='UTF-8') as csv_file:
        writer = csv.writer(csv_file, delimiter=',')    
        for tr in My_second_table.find_all('tr')[2:]: # [2:] is to skip empty and header 
            tds = tr.find_all('td')
            row = "{}, {}".format(tds[0].text.replace('\n',''), tds[2].text.replace('\n',''))
            writer.writerow(row.split(','))
Is there a way to simplify this code?


Yes. Don't scrape Wikipedia. Your first thought before "should I need to scrape this thing?" should be "Is there an API that can give me the data I want?" In this case, there super is.

There are many informative links such as this StackOverflow question, but in the end reading the API documentation really is the right thing to do. This should get you started:

from pprint import pprint
import requests, wikitextparser

r = requests.get(
    'https://en.wikipedia.org/w/api.php',
    params={
        'action': 'query',
        'titles': 'Transistor_count',
        'prop': 'revisions',
        'rvprop': 'content',
        'format': 'json',
    }
)
r.raise_for_status()
pages = r.json()['query']['pages']
body = next(iter(pages.values()))['revisions'][0]['*']
doc = wikitextparser.parse(body)
print(f'{len(doc.tables)} tables retrieved')

pprint(doc.tables[0].data())


This may seem more roundabout than scraping the page, but API access gets you structured data, which bypasses an HTML rendering step that you shouldn't have to deal with. This structured data is the actual source of the article and is more reliable.