Python - Iterating over contents of BeautifulSoup element very slow I have a large HTML document where everything is inside a main div, structured like this:

&lt;div class="main"&gt;
    \\block 1
    &lt;div class="header"&gt;&lt;span&gt;content1a&lt;/span&gt;&lt;span&gt;content1b&lt;/span&gt;&lt;/div&gt;
    &lt;p&gt;content1c&lt;/p&gt;

    \\block 2
    &lt;div class="header"&gt;&lt;span&gt;content2a&lt;/span&gt;&lt;span&gt;content2b&lt;/span&gt;&lt;/div&gt;
    &lt;p&gt;content2c&lt;/p&gt;

    ...
&lt;/div&gt;


As you can see the contents of the header divs are related to the paragraphs, so my Python code separates these tag blocks into lists of a list so I can take out their contents later:

main_div = soup.find("div", class_="main")
headers = main_div.find_all("div", class_="header")

all_blocks = []
current_block = []

for tag in main_div.contents:
    if tag in headers:
        all_blocks.append(current_block)
        current_block = [tag]
    else:
        current_block.append(tag)

all_blocks.append(current_block)  # append final block
all_blocks = all_blocks[1:]  # take off first empty list


Problem is this seems to take forever on a ~12MB HTML file I'm feeding it (less than 1% done after 1h, while parsing the file with bs4 only took 25s).
Is there something really inefficient in my code or is this going to be slow anyway?
It seems like you have the following:

&lt;div class="main"&gt;
    &lt;div class="header"&gt;a&lt;/div&gt;
    &lt;p&gt;b&lt;/p&gt;

    &lt;div class="header"&gt;c&lt;/div&gt;
    &lt;p&gt;d&lt;/p&gt;
&lt;/div&gt;


And you want to extract a list of the headers grouped with the content:

[('a', 'b'), ('c', 'd')]


The slow part of your code is most certainly the if tag in headers. If header is a list of 10k elements, then for each element in .main you are searching through potentially 10k elements. This is horribly inefficient. If headers was a set, this would be more efficient, but there's no need for it to be (or for you to check if tag in headers).

def get_header_content_pairs(doc):
    main = doc.find("div", class_="main")
    headers = main.find_all("div", class_="header")

    for header in headers:
        yield (header, list(get_content(header.nextSibling)))

def get_content(element):
    while element.name == 'p':
        yield element
        element = element.nextSibling


Haven't tested this, but the gist is instead of doing all that work, you take advantage of the fact that each BS element knows about its nextSibling. You find all of the .headers. For each of them, you continue checking nextSibling until you find something that isn't a &lt;p&gt;. You aggregate these and return them with their header.