Performance and Readability Improvements for HTML Parser with BeautifulSoup This function takes as an argument a json file (could contain anything in json format, since I scrap hundreds of random pages) and returns a list of dictionaries where a URL is mapped to its corresponding headers, based on the extraction of headers using beautifulsoup and a regex pattern.

I'm looking for suggestions regarding performance readability and clarity.

Following  my 1st iteration I improved my code and here is the result: 

import json
from tqdm import tqdm
import re
from bs4 import BeautifulSoup
import csv
import string

"Load HTML body, and fetch headers"
def get_headers_from_json(local_path):
    """
    The function takes a json file with html_body and returns a list of headers.
    It parses the titles, based on tags starting with 'h' + num.
    """
    data = json.loads(open(local_path).read())
    pattern = re.compile(r"^h[0-9]$")
    headers_urls = []
    printable = set(string.printable)
    for dict in tqdm(data):
        headers = []
        for val in dict.values():
            soup = BeautifulSoup(val, 'html.parser')
            url = dict.values()[0]
        for element in soup.find_all(pattern):
            element = element.get_text().strip().encode('utf-8')
            element = filter(lambda word: word in printable, element)  
            headers.append(element)
        cleaned_data = {"url": url, "headers": headers}
        headers_urls.append(cleaned_data)
    return headers_urls


Example of json input:

[["&lt;body class=\" catalog-category-view categorypath-sale-html category-sale\"&gt;\n&lt;script type=\"text/javascript\"&gt;\n//&lt;![CDATA[\nif (typeof(Varien.searchForm) !== 'undefined') {\n    Varien.searchForm.prototype._selectAutocompleteItem = function(element) {\n        var link = element.down();\n        if (link &amp;&amp; link.tagName == 'A') {\n            setLocation(link.href);\n        } else {\n            if (element.title){\n                this.field.value = element.title;\n            }\n            this.form.submit();\n        }\n    };\n    Varien.searchForm.prototype.initAutocomplete = function(url, destinationElement) {\n        new Ajax.Autocompleter(\n            this.field,\n            destinationElement,\n            url,\n            {\n                paramName: this.field.name,\n                method: 'get',\n                minChars: 2,\n                frequency: .3,\n                updateElement: this._selectAutocompleteItem.bind(this),\n                onShow : function(element, update) {\n                    if(!update.style.position || update.style.position=='absolute') {\n                        update.style.position = 'absolute';\n                        Position.clone(element, update, {\n                            setHeight: false,\n                            offsetTop: element.offsetHeight\n                        });\n                    }\n                    Effect.Appear(update,{duration:0});\n                }\n\n            }\n        );\n    };\n    Autocompleter.Base.prototype.markPrevious = function() {\n        if (this.index &gt; 0) {\n            this.index--;\n        } else {\n            this.index = this.entryCount - 1;\n        }\n        var entry = this.getEntry(this.index);\n        if (entry.select('a').length === 0) {\n            this.markPrevious(); // Ignore items that don't have link\n        }\n    };\n    Autocompleter.Base.prototype.markNext = function() {\n        if (this.index &lt; this.entryCount - 1) {\n
There are multiple things you can do to speed things up:


faster JSON parsing - try out ujson or simplejson; or even simplejson on PyPy
use lxml instead of html.parser (requires lxml to be installed)
do not parse the complete HTML, parse only what you need with SoupStrainer:

pattern = re.compile(r"^h[0-9]$")
parse_only = SoupStrainer(pattern)
soup = BeautifulSoup(data, "lxml", parse_only=parse_only)


You can then do simple soup.find_all() (or via a shortcut - soup()) since you'll only have header elements in your HTML soup.


Here is a demonstration of what SoupStrainer would do:

In [1]: import re

In [2]: from bs4 import BeautifulSoup, SoupStrainer

In [3]: data = """
   ...: &lt;body&gt;
   ...:     &lt;h1&gt;Some Paragraph 1&lt;/h1&gt;
   ...:     &lt;div&gt;
   ...:         &lt;h2&gt;Some Paragraph 2&lt;/h2&gt;
   ...:     &lt;/div&gt;
   ...:     
   ...:     &lt;span&gt;
   ...:         &lt;div&gt;
   ...:             &lt;h3&gt;Some Paragraph 3&lt;/h3&gt;
   ...:         &lt;/div&gt;
   ...:         &lt;h4&gt;Some Paragraph 4&lt;/h4&gt;
   ...:     &lt;/span&gt;
   ...:     &lt;h5&gt;Some Paragraph 2&lt;/h5&gt;
   ...: &lt;/body&gt;
   ...: """

In [4]: pattern = re.compile(r"^h[0-9]$")

In [5]: parse_only = SoupStrainer(pattern)

In [6]: soup = BeautifulSoup(data, "lxml", parse_only=parse_only)

In [7]: print(soup.prettify())
&lt;h1&gt;
 Some Paragraph 1
&lt;/h1&gt;
&lt;h2&gt;
 Some Paragraph 2
&lt;/h2&gt;
&lt;h3&gt;
 Some Paragraph 3
&lt;/h3&gt;
&lt;h4&gt;
 Some Paragraph 4
&lt;/h4&gt;
&lt;h5&gt;
 Some Paragraph 2
&lt;/h5&gt;