How to insert unescaped html fragment in Beautiful Soup 4
Question: I have to parse some nasty government created html
(<http://www.spokanecounty.org/detentionservices/inmateroster/detail2.aspx?sysid=84060>)
and to ease my pain I would like to insert some html fragments into the
document to wrap some content into more easily digested chunks.

BS4, however, escapes the html string fragment I'm trying to insert (`<div
class="case">`) and turns it into this:

    
    
    &lt;div class="case"&gt;
    

The relevant html I'm parsing is this:

    
    
    <div style='float:left; width:100%;border-top:solid 1px #666;height:5px;'>
        &nbsp;
    </div>
    <div style='width:45%; float:left;'>
        <h2 style='margin-top:0px;' rel=121018261>Case Number: 121018261</h2>
    </div>
    <div style='width:45%;float:right; text-align:right;'>
        <div>Added: 10/22/2012</div>
    </div>
    <div style='width:100%;clear:both;'>
        <b>Case Bond:</b> $2,000,000.00 <b style='margin-left:10px;'>Set By:</b> Spokane County Superior Court
    </div>
    <table class='bookinfo 121018261' style='width:100%;'>
        <tr><td><b>Charge 1  <a href='http://apps.leg.wa.gov/rcw/default.aspx?cite=' target='_blank'>RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG <br /> <b>Report Number:</b> 120160423 <b style='margin-left:10px;'>Report Agency:</b> 002 - SPOKANE CITY</td></tr>
        <tr><td><b>Charge 2 <a href='http://apps.leg.wa.gov/rcw/default.aspx?cite=' target='_blank'>RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG<br /><b>Report Number:</b> 120160423<b style='margin-left:10px;'>Report Agency:</b> 002 - SPOKANE CITY</td></tr>
    </table>
    
    <div style='float:left; width:100%;border-top:solid 1px #666;height:5px;'>
        &nbsp;
    </div>
    <div style='width:45%; float:left;'>
        <h2 style='margin-top:0px;' rel=121037010>Case Number: 121037010</h2>
    </div>
    <div style='width:45%;float:right; text-align:right;'>
        <div>Added: 10/21/2012</div>
    </div>
    <div style='width:100%;clear:both;'>
        <b>Case Bond:</b> $150,000.00 <b style='margin-left:10px;'>Set By:</b> Spokane County Superior Court
    </div>
    <table class='bookinfo 121037010' style='width:100%;'>
        <tr><td><b>Charge 1 <a href='http://apps.leg.wa.gov/rcw/default.aspx?cite=9A.44.050' target='_blank'>RCW: 9A.44.050(1)(A)</a>:</b> RAPE-2ND(FORCIBLE) <br /> <b>Report Number:</b> 120345597 <b style='margin-left:10px;'>Report Agency:</b> 001 - SPOKANE COUNTY</td></tr>
    </table>
    

The Python code looks like this:

    
    
    case_top = soup.find_all(style=re.compile("border-top:solid 1px #666"))
    for c in case_top:
        c.insert_before(soup.new_string('<div class="case">'))
    case_bottom = soup.find_all("table", class_="bookinfo")
    for c in case_bottom:
        c.insert_after(soup.new_string('</div'))
    

The results look like this:

    
    
    &lt;div class="case"&gt;<div style="float:left; width:100%;border-top:solid 1px #666;height:5px;"> </div><div style="width:45%; float:left;"><h2 rel="121018261" style="margin-top:0px;">Case Number: 121018261</h2></div><div style="width:45%;float:right; text-align:right;"><div>Added: 10/22/2012</div></div><div style="width:100%;clear:both;"><b>Case Bond:</b> $2,000,000.00 <b style="margin-left:10px;">Set By:</b> Spokane County Superior Court</div><table class="bookinfo 121018261" style="width:100%;"><tr><td><b>Charge 1 <a href="http://apps.leg.wa.gov/rcw/default.aspx?cite=" target="_blank">RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG<br/><b>Report Number:</b> 120160423 <b style="margin-left:10px;">Report Agency:</b> 002 - SPOKANE CITY</td></tr><tr><td><b>Charge 2 <a href="http://apps.leg.wa.gov/rcw/default.aspx?cite=" target="_blank">RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG<br/><b>Report Number:</b> 120160423 <b style="margin-left:10px;">Report Agency:</b> 002 - SPOKANE CITY</td></tr></table>&lt;/div&gt;&lt;div class="case"&gt;<div style="float:left; width:100%;border-top:solid 1px #666;height:5px;"> </div><div style="width:45%; float:left;"><h2 rel="121037010" style="margin-top:0px;">Case Number: 121037010</h2></div><div style="width:45%;float:right; text-align:right;"><div>Added: 10/21/2012</div></div><div style="width:100%;clear:both;"><b>Case Bond:</b> $150,000.00 <b style="margin-left:10px;">Set By:</b> Spokane County Superior Court</div><table class="bookinfo 121037010" style="width:100%;"><tr><td><b>Charge 1 <a href="http://apps.leg.wa.gov/rcw/default.aspx?cite=9A.44.050" target="_blank">RCW: 9A.44.050(1)(A)</a>:</b> RAPE-2ND(FORCIBLE)<br/><b>Report Number:</b> 120345597 <b style="margin-left:10px;">Report Agency:</b> 001 - SPOKANE COUNTY</td></tr></table>&lt;/div&gt;
    

The question is then, how can I insert an unescaped html fragment into the
document?


Answer: You are telling BeautifulSoup to insert **string data** :

    
    
    c.insert_before(soup.new_string('<div class="case">'))
    

Anything not safe for HTML string data will then indeed be escaped. You
instead want to insert a _tag object_ :

    
    
    c.insert_before(soup.new_tag('div', **{'class': 'case'}))
    

This creates a new child element, which does not actually wrap anything.

If you wanted to wrap each individual element in that are, you'd use the
[`Element.wrap()`
method](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#wrap):

    
    
    c.wrap(soup.new_tag('div', **{'class': 'case'}))
    

but this only works on one tag at a time.

For wrapping a _series_ of tags, the only thing that'll do is _moving_ the
tags over; inserting tags that were located in one place into another
effectively moves them over:

    
    
    case_top = soup.find_all(style=re.compile("border-top:solid 1px #666"))
    for case in case_top:
        wrapper = soup.new_tag('div', **{'class': 'case'})
        case.insert_before(wrapper)
        while wrapper.next_sibling:
            wrapper.append(wrapper.next_sibling)
            if wrapper.find('table', class_='bookinfo'):
                # moved over the bookinfo table, time to stop
                break
    

This then moves everything from the `case_top` element all the way to the
`<table class="bookinfo">` element into the new `<div class="case">` element.

Demo:

    
    
    >>> from bs4 import BeautifulSoup
    >>> import re
    >>> sample = '''\
    ... <body>
    ... <div style='float:left; width:100%;border-top:solid 1px #666;height:5px;'>
    ...     &nbsp;
    ... </div>
    ... <div style='width:45%; float:left;'>
    ...     <h2 style='margin-top:0px;' rel=121018261>Case Number: 121018261</h2>
    ... </div>
    ... <div style='width:45%;float:right; text-align:right;'>
    ...     <div>Added: 10/22/2012</div>
    ... </div>
    ... <div style='width:100%;clear:both;'>
    ...     <b>Case Bond:</b> $2,000,000.00 <b style='margin-left:10px;'>Set By:</b> Spokane County Superior Court
    ... </div>
    ... <table class='bookinfo 121018261' style='width:100%;'>
    ...     <tr><td><b>Charge 1  <a href='http://apps.leg.wa.gov/rcw/default.aspx?cite=' target='_blank'>RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG <br /> <b>Report Number:</b> 120160423 <b style='margin-left:10px;'>Report Agency:</b> 002 - SPOKANE CITY</td></tr>
    ...     <tr><td><b>Charge 2 <a href='http://apps.leg.wa.gov/rcw/default.aspx?cite=' target='_blank'>RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG<br /><b>Report Number:</b> 120160423<b style='margin-left:10px;'>Report Agency:</b> 002 - SPOKANE CITY</td></tr>
    ... </table>
    ... 
    ... <div style='float:left; width:100%;border-top:solid 1px #666;height:5px;'>
    ...     &nbsp;
    ... </div>
    ... <div style='width:45%; float:left;'>
    ...     <h2 style='margin-top:0px;' rel=121037010>Case Number: 121037010</h2>
    ... </div>
    ... <div style='width:45%;float:right; text-align:right;'>
    ...     <div>Added: 10/21/2012</div>
    ... </div>
    ... <div style='width:100%;clear:both;'>
    ...     <b>Case Bond:</b> $150,000.00 <b style='margin-left:10px;'>Set By:</b> Spokane County Superior Court
    ... </div>
    ... <table class='bookinfo 121037010' style='width:100%;'>
    ...     <tr><td><b>Charge 1 <a href='http://apps.leg.wa.gov/rcw/default.aspx?cite=9A.44.050' target='_blank'>RCW: 9A.44.050(1)(A)</a>:</b> RAPE-2ND(FORCIBLE) <br /> <b>Report Number:</b> 120345597 <b style='margin-left:10px;'>Report Agency:</b> 001 - SPOKANE COUNTY</td></tr>
    ... </table>
    ... </body>
    ... '''
    >>> soup = BeautifulSoup(sample)
    >>> case_top = soup.find_all(style=re.compile("border-top:solid 1px #666"))
    >>> for case in case_top:
    ...     wrapper = soup.new_tag('div', **{'class': 'case'})
    ...     case.insert_before(wrapper)
    ...     while wrapper.next_sibling:
    ...         wrapper.append(wrapper.next_sibling)
    ...         if wrapper.find('table', class_='bookinfo'):
    ...             # moved over the bookinfo table, time to stop
    ...             break
    ... 
    >>> soup.body
    <body><div class="case"><div style="float:left; width:100%;border-top:solid 1px #666;height:5px;">
         
    </div>
    <div style="width:45%; float:left;">
    <h2 rel="121018261" style="margin-top:0px;">Case Number: 121018261</h2>
    </div>
    <div style="width:45%;float:right; text-align:right;">
    <div>Added: 10/22/2012</div>
    </div>
    <div style="width:100%;clear:both;">
    <b>Case Bond:</b> $2,000,000.00 <b style="margin-left:10px;">Set By:</b> Spokane County Superior Court
    </div>
    <table class="bookinfo 121018261" style="width:100%;">
    <tr><td><b>Charge 1  <a href="http://apps.leg.wa.gov/rcw/default.aspx?cite=" target="_blank">RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG <br/> <b>Report Number:</b> 120160423 <b style="margin-left:10px;">Report Agency:</b> 002 - SPOKANE CITY</td></tr>
    <tr><td><b>Charge 2 <a href="http://apps.leg.wa.gov/rcw/default.aspx?cite=" target="_blank">RCW: 9A.56.210</a>:</b> ROBBERY-2ND DEG<br/><b>Report Number:</b> 120160423<b style="margin-left:10px;">Report Agency:</b> 002 - SPOKANE CITY</td></tr>
    </table></div>
    <div class="case"><div style="float:left; width:100%;border-top:solid 1px #666;height:5px;">
         
    </div>
    <div style="width:45%; float:left;">
    <h2 rel="121037010" style="margin-top:0px;">Case Number: 121037010</h2>
    </div>
    <div style="width:45%;float:right; text-align:right;">
    <div>Added: 10/21/2012</div>
    </div>
    <div style="width:100%;clear:both;">
    <b>Case Bond:</b> $150,000.00 <b style="margin-left:10px;">Set By:</b> Spokane County Superior Court
    </div>
    <table class="bookinfo 121037010" style="width:100%;">
    <tr><td><b>Charge 1 <a href="http://apps.leg.wa.gov/rcw/default.aspx?cite=9A.44.050" target="_blank">RCW: 9A.44.050(1)(A)</a>:</b> RAPE-2ND(FORCIBLE) <br/> <b>Report Number:</b> 120345597 <b style="margin-left:10px;">Report Agency:</b> 001 - SPOKANE COUNTY</td></tr>
    </table></div>
    </body>
    

