Performance and Readability Improvements for HTML Parser with BeautifulSoup
There are multiple things you can do to speed things up:


faster JSON parsing - try out ujson or simplejson; or even simplejson on PyPy
use lxml instead of html.parser (requires lxml to be installed)
do not parse the complete HTML, parse only what you need with SoupStrainer:

pattern = re.compile(r"^h[0-9]$")
parse_only = SoupStrainer(pattern)
soup = BeautifulSoup(data, "lxml", parse_only=parse_only)


You can then do simple soup.find_all() (or via a shortcut - soup()) since you'll only have header elements in your HTML soup.


Here is a demonstration of what SoupStrainer would do:

In [1]: import re

In [2]: from bs4 import BeautifulSoup, SoupStrainer

In [3]: data = """
   ...: &lt;body&gt;
   ...:     &lt;h1&gt;Some Paragraph 1&lt;/h1&gt;
   ...:     &lt;div&gt;
   ...:         &lt;h2&gt;Some Paragraph 2&lt;/h2&gt;
   ...:     &lt;/div&gt;
   ...:     
   ...:     &lt;span&gt;
   ...:         &lt;div&gt;
   ...:             &lt;h3&gt;Some Paragraph 3&lt;/h3&gt;
   ...:         &lt;/div&gt;
   ...:         &lt;h4&gt;Some Paragraph 4&lt;/h4&gt;
   ...:     &lt;/span&gt;
   ...:     &lt;h5&gt;Some Paragraph 2&lt;/h5&gt;
   ...: &lt;/body&gt;
   ...: """

In [4]: pattern = re.compile(r"^h[0-9]$")

In [5]: parse_only = SoupStrainer(pattern)

In [6]: soup = BeautifulSoup(data, "lxml", parse_only=parse_only)

In [7]: print(soup.prettify())
&lt;h1&gt;
 Some Paragraph 1
&lt;/h1&gt;
&lt;h2&gt;
 Some Paragraph 2
&lt;/h2&gt;
&lt;h3&gt;
 Some Paragraph 3
&lt;/h3&gt;
&lt;h4&gt;
 Some Paragraph 4
&lt;/h4&gt;
&lt;h5&gt;
 Some Paragraph 2
&lt;/h5&gt;