recursing a list of 73033 elements with HTML tags and get context from it
Question: I have a long list of elements with a length of 73,033. I would like to get
the context from it. In the list, each element has the same structure (if the
block of the following code helps), and it looks like this `<div
align="center" class="photocaption"> Author/Designer Carleton Varney with Jim
Druckman </div>`. What I am interested in getting is the text `Author/Designer
Carleton Varney with Jim Druckman.`

**Main code**

    
    
    NewSoups = [BeautifulSoup(NewR) for NewR in NewRs]. 
    captions = [soup.find_all("div", class_ = "photocaption") for soup in NewSoups]
    flattened_captions = []
    for x in captions:
        for y in x:
            flattened_captions.append(y)
    
    print(len(flattened_captions)) #73033
    
    import re
    results = [re.sub('<[^>]*>', '', y) for y in flattened_captions] #where the error comes from
    

**Error**

    
    
    Traceback (most recent call last):
    File "picked.py", line 22, in <module>
    results = [re.sub('<[^>]*>', '', y) for y in flattened_captions]
    File "/opt/conda/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
    TypeError: expected string or buffer
    

I am wondering if there is a convenient way to loop through the long list of
`<div ></div>`. If not, what would be the best way to extract all text that I
desire? Thank you very much.


Answer: What I am going to post is not the most elegant or efficient way to deal with
the posted problem. As Welbog pointed out, BeautifulSoup itself has provided
the function of extracting context. As I received the error as I posted my
original question, however, I was just curious where the error came from. It
turned out that the returned things from flattented_captions were not strings.
It's quite simple to solve. Method as below.

    
    
    str_flattened_captions = [str(flattened_captions[i]) for i in range(len(flattened_captions))]
    
    gains = [re.sub('<[^>]*>', '', item) for item in str_flattened_captions]
    

To test

    
    
    print(gains[:5])
    r Barbara Schorr ', ' Architect Joan Dineen with Alyson Liss ', ' Author/Designer Carleton Varney with Jim Druckman ', ' Designers Richard Cerrone, Lisa Hyman and Rhonda Eleish (front) in their room called "Holiday Nod To Nature" ']
    

