How to fetch paragraphs from html using Python
Question: How to fetch paragraphs from bad-structured html?

I have this original html text:

    
    
    This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
    <br>
    <ul>
        <li>AA Early Childhood Education, or related field.  </li>
        <li>2+ years experience in a licensed childcare facility  </li>
        <li>Ability to meet state requirements, including finger print clearance.  </li>
        <li>Excellent oral and written communication skills  </li>
        <li>Strong organization and time management skills.  </li>
        <li>Creativity in expanding children's learning through play.<br>  </li>
        <li>Strong classroom management skills.<br>  </li>
    </ul>
    <p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children. 
        <br> 
    </p>
    

I use Python and try to do something like that:

    
    
    soup = BeautifulSoup(html)
    

It returns a new html text with 2 **short** paragraphs:

    
    
    <html>
    
    <body>
        <p>This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
            <br/>
        </p>
        <ul>
            <li>AA Early Childhood Education, or related field. </li>
            <li>2+ years experience in a licensed childcare facility </li>
            <li>Ability to meet state requirements, including finger print clearance. </li>
            <li>Excellent oral and written communication skills </li>
            <li>Strong organization and time management skills. </li>
            <li>Creativity in expanding children's learning through play.
                <br/> </li>
            <li>Strong classroom management skills.
                <br/> </li>
        </ul>
        <p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
            <br/> </p>
    </body>
    
    </html>
    

But it is not what I expected. In the result, I would like to get this html
text:

    
    
    <html>
    
    <body>
        <p>This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
            AA Early Childhood Education, or related field.
            2+ years experience in a licensed childcare facility
            Ability to meet state requirements, including finger print clearance.
            Excellent oral and written communication skills
            Strong organization and time management skills.
            Creativity in expanding children's learning through play.
            Strong classroom management skills.
        </p>
        <p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.</p>
    </body>
    
    </html>
    

For getting above html, I think that the best approach is to remove all html
tags except `<p>` and `</p>` from original html.

For this purpose I tried the following regular expression:

    
    
    new_html = re.sub('<[^<]+?>', '', html)
    

Obviously, the regular expession removes all html tags. So, how to remove all
html tags except `<p>` and `</p>`?

If someone help me to write the r.e. then I feed `new_html` to
`BeautifulSoup()` and get html that I expect.


Answer: This is sort of a manual document manipulation, but, you can loop over the
`li` elements and
[remove](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#extract) them
after
[appending](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#append) to
the first paragraph. Then, remove the `ul` element as well:

    
    
    from bs4 import BeautifulSoup
    
    
    data = """
    This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
    <br>
    <ul>
        <li>AA Early Childhood Education, or related field.  </li>
        <li>2+ years experience in a licensed childcare facility  </li>
        <li>Ability to meet state requirements, including finger print clearance.  </li>
        <li>Excellent oral and written communication skills  </li>
        <li>Strong organization and time management skills.  </li>
        <li>Creativity in expanding children's learning through play.<br>  </li>
        <li>Strong classroom management skills.<br>  </li>
    </ul>
    <p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
        <br>
    </p>"""
    
    soup = BeautifulSoup(data, "lxml")
    
    p = soup.p
    for li in soup.find_all("li"):
        p.append(li.get_text())
        li.extract()
    
    soup.find("ul").extract()
    print(soup.prettify())
    

Prints the 2 paragraphs as you've planned to have:

    
    
    <html>
     <body>
      <p>
       This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
       <br/>
       AA Early Childhood Education, or related field.
       2+ years experience in a licensed childcare facility
       Ability to meet state requirements, including finger print clearance.
       Excellent oral and written communication skills
       Strong organization and time management skills.
       Creativity in expanding children's learning through play.
       Strong classroom management skills.
      </p>
      <p>
       The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
       <br/>
      </p>
     </body>
    </html>
    

Note that there is an important difference in the way `lxml`, `html.parser`
and `html5lib` parse the input HTML you've posted. `html5lib` and
`html.parser` don't automatically create the first paragraph making the code
above really `lxml` specific.

* * *

A better approach would probably be making a separate "soup" object. Sample:

    
    
    from bs4 import BeautifulSoup
    
    
    data = """
    This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
    <br>
    <ul>
        <li>AA Early Childhood Education, or related field.  </li>
        <li>2+ years experience in a licensed childcare facility  </li>
        <li>Ability to meet state requirements, including finger print clearance.  </li>
        <li>Excellent oral and written communication skills  </li>
        <li>Strong organization and time management skills.  </li>
        <li>Creativity in expanding children's learning through play.<br>  </li>
        <li>Strong classroom management skills.<br>  </li>
    </ul>
    <p>The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
        <br>
    </p>"""
    
    soup = BeautifulSoup(data, "lxml")
    
    # create new soup
    new_soup = BeautifulSoup("<body></body>", "lxml")
    new_body = new_soup.body
    
    # create first paragraph
    first_p = new_soup.new_tag("p")
    first_p.append(soup.p.get_text())
    
    for li in soup.find_all("li"):
        first_p.append(li.get_text())
    
    new_body.append(first_p)
    
    # create second paragraph
    second_p = soup.find_all("p")[-1]
    new_body.append(second_p)
    
    print(new_soup.prettify())
    

Prints:

    
    
    <html>
     <body>
      <p>
       This position is responsible for developing and implementing age appropriate lesson and activity plans for preschool children, ages 4-5 years-old. Maintain a fun and interactive classroom that is clean and well organized, provide a safe, healthy and welcoming learning environment. The ideal candidate will have:
       AA Early Childhood Education, or related field.
       2+ years experience in a licensed childcare facility
       Ability to meet state requirements, including finger print clearance.
       Excellent oral and written communication skills
       Strong organization and time management skills.
       Creativity in expanding children's learning through play.
       Strong classroom management skills.
      </p>
      <p>
       The ideal candidate must be a reliable, self-starting professional who is passionate about teaching young children.
       <br/>
      </p>
     </body>
    </html>
    

