Web Crawling with 25 Lines of Python Code
Web crawling and web scraping are two very similar and complementary fields. The best definition I have seen for these two states that a crawler, also known as a spider, is a bot designed to move through a website, crawling page by page. On the other hand, scraping is the act of extracting data from websites.

In this article, I’ll cover how to use a recursive function to create a very simple, but efficient crawler with only a few lines of code and also how to use it to scrape some data.

Wikipedia will be used as an example for two main reasons. First, it allows scraping and crawling, which is something you should always check before beginning; also, its structure makes sense for what we’re performing in this article since we can move from article to article using the links referenced in each one of them.

Writing the Code

Let’s start with the imports. We’ll only need the Python libraries BeautifulSoup and Requests for this task. I’m assuming you already have them installed, but if you don’t, you can do it using pip:

pip install beautifulsoup4 pip install requests

With this set, we’ll now write the function that will do the entire work. It’s particularly important to write the crawler as a function in this case, so the crawler can call itself multiple times recursively in order to get the links from page after page.

The function will receive only one argument, the website’s URL, and will go straight to getting the page with requests and parsing it with BeautifulSoup. This URL is where the crawler will start. We’ll also instantiate a list to keep track of the pages as we crawl them.

pages_crawled = []

def crawler(url):

page = requests.get(url)

soup = BeautifulSoup(page.text, 'html.parser')

From now on, all the code will be inside the function.

Our task here consists of getting all the links to other pages that are on the starting page and then going on each of these pages to get all the links inside them and so on, indefinitely.

Links to other pages are commonly stored inside the a tags. So, what we need to do is to get all these tags on the page. The find_all method is the best option for this:

links = soup.find_all('a')

This line of code returns an iterable containing all the a tags on the page. We’ll then iterate over it to call the crawler function again, passing each of the links as an argument.

But before we do this, some conditions have to be met:

The tag must contain a href attribute. href stands for Hypertext Reference and is where the crawler will find the links. It’s important to check this because although links are usually inside a tags, not all these tags will contain links. Since we’re only interested in article pages, the link must start with “/wiki” and not contain semicolons, otherwise, the crawler would eventually be redirected to other parts of Wikipedia or even to other completely unrelated websites. The link must not have been crawled yet. For this, append the crawled links to a list, and, for each new link, we’ll check if it’s in this list already.

This is how all this looks:

for link in links:

if 'href' in link.attrs:



if link['href'].startswith('/wiki') and ':' not in link['href']:



if link['href'] not in pages_crawled:

With these conditions met, we’ll now append the link to the list of crawled links and create the full URL for the next page:

new_link = f"https://en.wikipedia.org{link['href']}"

pages_crawled.append(link['href'])

Collecting Data

Before you call the function with another URL, it’s time to gather the data you came for. You could, in theory, scrape everything you want: the page title, the article name, the table of contents, the subtitles, the entire text of the article, and way more.

However, I’m not actually scraping data from Wikipedias here, I’m just using the website as an example. But let’s say we’d like to get the page’s title, the name of the article, and the article’s URL for example. What we could do is to use a context manager to open a text file and write the information.

with open('data.csv', 'a') as file:

file.write(f'{soup.title.text}; {soup.h1.text}; {link["href"]}

')

In this case, it would be important to use semicolons as separators because as we’re dealing with text data, we should expect to find some commas in it.

Now, we can just call the function again and the code will indefinitely grab more and more pages.

crawler(new_link)

And this is the entire code:

Notice that our crawler starts on the home page, but it could start on any page you want inside Wikipedia. Also, a try clause was added to prevent the code from crashing in case it finds any problems when writing the data in the .csv file or when calling the function recursively.

Randomizing the Crawling

Another possibility for crawling Wikipedia is to do it randomly. This wouldn’t be the smartest idea if the goal was to get every single page on the website, but that’s not what we’re doing here.

Choosing random articles to crawl would still generate a great amount of data, since Wikipedia is almost endless, without making too many requests to their server, which is something we always need to consider as we don’t intend to cause any harm.

We can accomplish that with just a few changes in the code and some new imports. First, we need to specify to the find_all method that we’re not only looking for all the a tags on the page neither for all the a tags that contain the href attribute, but that we want all the a tags that contain a href attribute with a link that starts with “/wiki” and does not contain a semicolon.

We can achieve that by specifying using the Re module. We then write a regular expression that satisfies these requirements.

links = soup.find_all('a',

{'href': re.compile('^\/wiki\/((?!:).)*$')})

We then use Numpy to select a random tag from links .

link = links[np.random.randint(1, len(links) + 1)]

As the first two of the three conditions we established earlier are already met, we only need to check the last one. So if the link hasn’t been crawled yet, we just need to append it to the crawled links list, get the data, create the full URL and call the function again.

if link not in pages_crawled:

pages_crawled.append(link)



with open('data.csv', 'a') as file:

file.write(f'{soup.title.text}; {soup.h1.text}; {link["href"]}

') new_link = f"https://en.wikipedia.org{link['href']}"

random_crawl(new_link)

And that’s done. If you want reproducible results, you can specify the random seed or if you want different random results every time you run the code, you can use the time module to generate different seeds.

And this is the complete code:

Final Thoughts

This is how we crawl the entire Wikipedia with no more than twenty-five lines of code. It’s a very simple and efficient first web crawler to write and a good way to start with web crawling.

However, keep in mind that, as already mentioned, Wikipedia allows you to do this and other websites may not. But even Wikipedia has some conditions for this and you can see in their robots.txt:

Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please.

They specify the low-speed bots are welcome, therefore, you definitely should include some random pauses in your code in order not to overload the website’s server. You can use time.sleep() to get this done.

Also, when writing production-level code, which is not the case in these articles as this is just an example, it’s a good practice to have your code even more ready to handle exceptions and maybe use a proxy provider, such as Infatica, as they’ll be able to provide you a better infrastructure of IP addresses so you can be sure your code will keep running. This is even more important when crawling a website because of the number of requests sent to the server.

That’s it! I hope you’ve enjoyed this and that it can maybe be useful somehow. If you have a question, a suggestion, or just want to be in touch, feel free to contact me through Twitter, GitHub, or Linkedin.
['Otávio Simões Silveira']
['Crawling', 'Python', 'Web Scraping', 'Beautifulsoup', 'Data Science']