Extract html content based on tags, specifically headers I want the function to take as an input json file containing html_body with its corresponding url and return list of tuples containing headers and their corresponding url (so could be tuple with one item as a list of the headers).

So far I have been able to do so, but I'm missing some headers and more over, I'm sure that this code can be improved.

Commented out possible extensions for &lt;h2&gt; headers extraction, and a different regex pattern I tried. 

A taste of the source:

[["&lt;body class=\" catalog-category-view categorypath-sale-html category-sale\"&gt;\n&lt;script type=\"text/javascript\"&gt;\n//&lt;![CDATA[\nif (typeof(Varien.searchForm) !== 'undefined') {\n    Varien.searchForm.prototype._selectAutocompleteItem = function(element) {\n        var link = element.down();\n        if (link &amp;&amp; link.tagName == 'A') {\n            setLocation(link.href);\n        } else {\n            if (element.title){\n                this.field.value = element.title;\n            }\n            this.form.submit();\n        }\n    };\n    Varien.searchForm.prototype.initAutocomplete = function(url, destinationElement) {\n        new Ajax.Autocompleter(\n            this.field,\n            destinationElement,\n            url,\n            {\n                paramName: this.field.name,\n                method: 'get',\n                minChars: 2,\n                frequency: .3,\n                updateElement: this._selectAutocompleteItem.bind(this),\n                onShow : function(element, update) {\n                    if(!update.style.position || update.style.position=='absolute') {\n                        update.style.position = 'absolute';\n                        Position.clone(element, update, {\n                            setHeight: false,\n                            offsetTop: element.offsetHeight\n                        });\n                    }\n                    Effect.Appear(update,{duration:0});\n                }\n\n            }\n        );\n    };\n    Autocompleter.Base.prototype.markPrevious = function() {\n        if (this.index &gt; 0) {\n            this.index--;\n        } else {\n            this.index = this.entryCount - 1;\n        }\n        var entry = this.getEntry(this.index);\n        if (entry.select('a').length === 0) {\n            this.markPrevious(); // Ignore items that don't have link\n        }\n    };\n    Autocompleter.Base.prototype.markNext = function() {\n        if (this.index &lt; this.entryCount - 1) {\n   


Code:

from pandas import DataFrame, Series
from tqdm import tqdm
import html5lib
import re
from bs4 import BeautifulSoup

def get_headers_from_json(local_path):
"""
The function takes a json file with html_body and returns a list of headers.
It parses the headers, based on tags starting with 'h'.
It also parses urls and return them in a tuple: (headers, url)
"""
data = json.loads(open(local_path).read())
pattern = "h1" # | h2 | h3"   
# pattern = re.compile("^(?!.*(href)).*^h", re.IGNORECASE)
headers_urls = []
all_headers = []
for x in tqdm(data):
    soup = BeautifulSoup(x[0], 'html.parser')
    html_headers = soup.find_all(re.compile(pattern))
    all_headers.append(html_headers)
    url = x[1]
    tuple_title_url = all_headers, url
    headers_urls.append(tuple_title_url)
return headers_urls


PS - I checked the following links but couldn't find a great solution:
HTML tag encoder
HTML parsing algorithm for extracting &lt;a&gt; tags
Scraping HTML using Beautiful Soup
Currently, you are matching a single h1 tag only, but, from what I understand, you want to catch all the "header" elements. There are multiple ways to do that. 

One is with a regular expression:

headers = soup.find_all(re.compile(r"^h"))


Or, with a lambda function:

headers = soup.find_all(lambda tag: tag and tag.name.startswith("h"))


Or, with a list of explicitly specified tags:

headers = soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5'])


Note that in order to get the header texts, you would use .get_text() method:

[header.get_text() for header in headers]




Other notes:


if every item returned from tqdm(data) consists of two items, you can unpack them in a for loop:

for html, url in tqdm(data):


Then, you can use html instead of x[0] and url instead of x[1].
if you are going to go with a regular expression approach to match the headers, you should pre-compile the expression - define it as a constant:

HEADER_PATTERN = re.compile(r"^h")

you can use json.load() to read JSON from a file directly, replace:

data = json.loads(open(local_path).read())


with:

with open(local_path) as input_file:
    data = json.load(input_file)



If performance matters:


you can use a faster JSON parser: ujson instead of json
use lxml instead of html.parser as a BeautifulSoup underlying parser
you can parse header elements only with SoupStrainer ignoring all other elements - this should boost HTML parsing performance as well


Improved code (with some of the suggestions applied):

import re

from bs4 import BeautifulSoup, SoupStrainer
from tqdm import tqdm
import ujson


HEADER_PATTERN = re.compile(r"^h")


def get_headers_from_json(local_path):
    """
    The function takes a json file with html_body and returns a list of headers.
    It parses the headers, based on tags starting with 'h'.
    It also parses urls and return them in a tuple: (headers, url)
    """
    with open(local_path) as input_file:
        data = ujson.load(input_file)

    parse_only = SoupStrainer(HEADER_PATTERN)
    return [
        ([header.get_text() for header in BeautifulSoup(html, 'lxml', parse_only=parse_only)(HEADER_PATTERN)], url)
        for html, url in tqdm(data)
    ]