Beautifulsoup and lxml (xpath) too slow with respect to regex when parsing HTML I agree that using regex to parse HTML is not a good way, in particular I am worried about their fragility with respect to change in the HTML.

The problem is that any alternatives are really too slow.

I am parsing this web page which is quite big (3MB). Somewhere in the html there is:

&lt;form class="form-inline"  id="exportform" action="/references" class="hidden"&gt;
    &lt;input type="hidden" name="format" value="" /&gt;

    &lt;input type="hidden" name="item_id" value="678860" /&gt;

    &lt;input type="hidden" name="item_id" value="656680" /&gt;

    &lt;input type="hidden" name="item_id" value="678846" /&gt;

    &lt;input type="hidden" name="item_id" value="650319" /&gt;

    &lt;input type="hidden" name="item_id" value="651504" /&gt;

    &lt;input type="hidden" name="item_id" value="678849" /&gt;

    &lt;input type="hidden" name="item_id" value="649821" /&gt;

    &lt;input type="hidden" name="item_id" value="649835" /&gt;

    &lt;input type="hidden" name="item_id" value="651512" /&gt;

    &lt;input type="hidden" name="item_id" value="651510" /&gt;

&lt;/form&gt;


I need to extract the 10 numbers in the value field. I know they are exactly 10.

I tried different approaches:

BeautifulSoup

from bs4 import BeautifulSoup

def soup(html):
    soup = BeautifulSoup(html, 'lxml')
    form = soup.find("form", {'id': 'exportform'})
    if not form:
        return []
    entries = form.find_all("input", {'name': 'item_id'})
    result = [entry['value'] for entry in entries]
    return result


Regex

import re
import itertools

def regex(html):
    r = re.compile(r'&lt;input type="hidden" name="item_id" value="([0-9]+)"&gt;')

    s = r.finditer(html)
    result = []
    for ss in itertools.islice(s, 10):
        result.append(ss.group(1))
    return result


xpath

import lxml.html
from lxml import etree # same performance with this

def xpath(html):
    root = lxml.html.fromstring(html)
    result = root.xpath('//form[@class="form-inline"]/*[@name="item_id"]')
    return [r.attrib['value'] for r in result]


I get the same results from the three, but very different timing:


BeautifulSoup: 3.3
regex: 0.00013
xpath: 0.57


In the regex case I am able to use the trick that I know there are exactly 10 numbers to find, gaining a factor 10. But also without that trick the regex approach is 400 times faster than xpath.

In addition asking xpath to give me the first 10:

(//form[@class="form-inline"]/*[@name="item_id"])[position() &lt; 11]


is not improving
Well, your regex solution is obviously badly broken - it will fail if the attributes are in a different order, if they are separated by newlines, if they are delimited by single quotes, etc etc. If you try to replace it with a more correct regex (it will never be 100% correct of course) then you are quite likely to lose some of this speed - perhaps dramatically, if it starts backtracking.

The cost of the XPath solution is not actually in evaluating the XPath, it is (a) parsing the source document and building a tree representation, and (b) compiling the XPath expression. The key to getting a performance improvement is to make sure you are doing these operations only once. I don't know whether that's possible in your overall application scenario.