Handling HTML characters in web scraped html using Python BS4
Question: This might be a duplicate question, but unable to find any answers searching
through stackoverflow..

Scraped some html files from the web, but they contain special characters like
'>', '<' in the text and BeautifulSoup is unable to handle it and throwing
BeautifulSoup.find erratic. Is there a way to escape the text before using
BeautifulSoup to parse the html?

EDIT: Thought this is generic enough, but adding html with issue:

    
    
    <HTML>
    <HEAD><TITLE>Title</TITLE>
    </HEAD><BODY>
    <p>
    <h2>Heading 2</h2>
    <hr align=left width=75%>
    <dl><h3>Heading 3</h3>
    <p>
    <dd><a href="./ref.pl?R1"><b>R1</b></a>
    <i><b>PP</b></i>:
    <a href="./refs.pl?R2">R2</a>
    <dl>
    <dd>
     Text1 <a href="./refs.pl?T1">T1</a>
    ; Text2 <a href="./refs.pl?T1">T1</a>
    <i>value<=500</i> <a href="./refs.pl?+T2">T2</a>
    ; Text3 <a href="./refs.pl?T3">T3</a>
    </dl>
    Sat Feb  14 23:36:59 EST 2016
    <p></body></html>
    

Trying to collect all text values, calling dd = soup.find('dd') and parsing
dd.contents misses out value<=500 and Text3..


Answer: Answering my own question, but is there a easier way to handle it directly
with BeautifulSoup?

    
    
    from tidylib import tidy_document
    doc, errors = tidy_document(htmlfile.read())
    soup = BeautifulSoup(doc, "lxml")
    

Now the HTML document has `<i>value&lt;=500</i>`, and this helps
`BeautifulSoup.find` from behaving erratically.

Calling `dd = soup.find('dd')` and parsing `dd.contents` now provides
`value<=500` and `Text3`.

