Python regex findall to capture repeated groups
Question: # Context

I am using python [regex](https://pypi.python.org/pypi/regex "regex") to parse
some HTMLs because they are too broken to use processors better suited for
those tasks (e.g. scrapy selectors). An excerpt of the HTML I want to parse
looks like this:

    
        <LI><B>First list title</B> Additional info
      <UL>
      <LI><I>List element 1</I> additional info
      </UL>
    
    <LI><B>Second list title</B> Additional info
      <UL>
      <LI><I>List element 1</I> additional info1
      <LI><I>List element 2</I> additional info2
      <LI><I>List element 3</I> additional info3
      <LI><I>List element 4</I> additional info4
      </UL>
    
    <!-- many more elements like the ones above --> 
    

I need to capture the List title (and additional information), and for each
title all nested elements it has with their additional information as well.

# Approaches

    
    
    import regex as re
    

## re.findall

    
    
    reg = re.compile("<li><b>(.*)\n\s*<ul>\n(\s*<li>.+\n)+\s*</ul>", re.IGNORECASE)
    g_info = re.findall(reg, response.body)
    

If we look info g_info in the above example we will see that for those with
one list element is fine:

    
    
    g_info[0] <- ('First list title</B> Additional info', "  <LI><I>List element 1</I> additional info\n")
    

But when there are multiple sublist elements, only the last one is obtained.

    
    
    g_info[1] <- ('Second list title</B> Additional info', "  <LI><I>List element 4</I> additional info4\n")
    

I would like it to be something like:

    
    
    g_info[1] <- ('Second list title</B> Additional info', "  <LI><I>List element 1</I> additional info1\n", "  <LI><I>List element 2</I> additional info2\n", ...)
    

## re.search and .captures

Using the same regular expression I can use the .captures function to get all
the elements captured. I will tune it slightly so it works with this example:

    
    
    reg = re.compile("<li><b>(.*)\n\s*<ul>\n(\s*<li>.+\n){2,}\s*</ul>", re.IGNORECASE)
    g_info = re.search(reg, response.body)
    

But this way (I would further parse each element with another simpler regexp
to obtain what I want) I only get the first match and not all of them.

    
    
    g_info.captures() <-- '<LI><B>Second list title</B> Additional info\n  <UL>\n  <LI><I>List element 1</I> additional info1\n  <LI><I>List element 2</I> additional info2\n ...'
    

If I could get all of them in this format it would suffice for me.

## re.findall and additional looping and filtering

I could use a simpler regexp to get all of them. Then I could further detect
which is a subelement and which not, because list titles always start with a
bold tag and the others don't.

    
    
    reg = re.compile("(\s*<li>.+\n)", re.IGNORECASE)
    g_info = re.findall(reg, response.body)
    

What I get is something like this:

    
    
    g_info[0] <- '\n\n<LI><B>First list title</B> Additional info\n'
    g_info[1] <- '\n  <LI><I>List element1</I> additional info\n'
    g_info[2] <- '\n\n<LI><B>Second list title</B> Additional info\n'
    g_info[3] <- '\n  <LI><I>List element</I> additional info1\n'
    g_info[4] <- '  <LI><I>List element2</I> additional info2\n'
    g_info[5] <- '  <LI><I>List element3</I> additional info3\n'
    

# Solutions?

The only workable approach I found was the last one, which imho is not
elegant. Could you help me find a better solution? Thanks


Answer: 
    import re
    pattern = re.compile("(?<=<li><b>).*?(?=</ul>)", re.IGNORECASE | re.DOTALL)
    print re.findall(pattern, data)
    

**Output**

    
    
    ['First list title</B> Additional info\n  <UL>\n  <LI><I>List element 1</I> additional info\n  ',
     'Second list title</B> Additional info\n  <UL>\n  <LI><I>List element 1</I> additional info1\n  <LI><I>List element 2</I> additional info2\n  <LI><I>List element 3</I> additional info3\n  <LI><I>List element 4</I> additional info4\n  ']
    

