Python - Iterating over contents of BeautifulSoup element very slow
It seems like you have the following:

&lt;div class="main"&gt;
    &lt;div class="header"&gt;a&lt;/div&gt;
    &lt;p&gt;b&lt;/p&gt;

    &lt;div class="header"&gt;c&lt;/div&gt;
    &lt;p&gt;d&lt;/p&gt;
&lt;/div&gt;


And you want to extract a list of the headers grouped with the content:

[('a', 'b'), ('c', 'd')]


The slow part of your code is most certainly the if tag in headers. If header is a list of 10k elements, then for each element in .main you are searching through potentially 10k elements. This is horribly inefficient. If headers was a set, this would be more efficient, but there's no need for it to be (or for you to check if tag in headers).

def get_header_content_pairs(doc):
    main = doc.find("div", class_="main")
    headers = main.find_all("div", class_="header")

    for header in headers:
        yield (header, list(get_content(header.nextSibling)))

def get_content(element):
    while element.name == 'p':
        yield element
        element = element.nextSibling


Haven't tested this, but the gist is instead of doing all that work, you take advantage of the fact that each BS element knows about its nextSibling. You find all of the .headers. For each of them, you continue checking nextSibling until you find something that isn't a &lt;p&gt;. You aggregate these and return them with their header.