Python HTML to Text File UnicodeDecodeError?
Question: So I am writing a program to read a webpage using urllib, then using
"html2text", write the basic text to a file. However, the raw contents given
from urllib.read() has various characters, so it would continuously raise
_UnicodeDecodeError_.

I of course Googled this for 3 hours, got plenty of answers like using
HTMLParser, or reload(sys), using external modules like pdfkit or
BeautifulSoup, and of course .encode/.decode.

Reloading sys and then executing _sys.setdefaultencoding("utf-8")_ grants me
the desired results, but IDLE and the program becomes unresponsive after that.

I tried every variation of the .encode/.decode with 'utf-8' and 'ascii', with
arguments like 'replace', 'ignore', etc. For some reason, it raises the same
error everytime regardless of the arguments I supply in the encode/decode.

    
    
    def download(self, url, name="WebPage.txt"):
        ## Saves only the text to file
        page = urllib.urlopen(url)
        content = page.read()
        with open(name, 'wb') as w:
            HP_inst = HTMLParser.HTMLParser()
            content = content.encode('ascii', 'xmlcharrefreplace')
            if True: 
                #w.write(HTT.html2text( (HP_inst.unescape( content ) ).encode('utf-8') ) )
                w.write( HTT.html2text( content) )#.decode('ascii', 'ignore')  ))
                w.close()
                print "Saved!"
    

There has to be another method or encoding I am missing... Please help!

Side Quest: I sometimes have to write it to a file where the name includes
unsupported chars like _"G\u00e9za Teleki"+".txt"_. How do I filter those
characters out?

Note:

  * This function was stored inside a class (hint "self").
  * Using python2.7
  * Don't want to use BeautfiulSoup 
  * Windows 8 64-bit


Answer: You should decode the content get from urllib with the properly encoding eg,
utf-8 latin1 depends on the page you get.

The way to detect the encoding of the content are various. From headers or
meta in html. I'd like to use a encoding detective module which I forget the
name, you could google it.

Once you decode it properly, you can encode it to any encoding you like before
write to a file

======================================

Here's the example using [chardet](https://pypi.python.org/pypi/chardet)

    
    
    import urllib
    import chardet
    
    
    def main():
        page = urllib.urlopen('http://bbc.com')
        content = page.read()
    
        # detect the encoding
        try:
            encoding = chardet.detect(content)['encoding']
        except:
            # use utf-8 as default encoding
            encoding = 'utf-8'
    
        # decode the content into unicode
        content = content.decode(encoding)
    
        # write to file
        with open('test.txt', 'wb') as f:
            f.write(content.encode('utf-8'))
    

