Ascii codec cannot decode HTML when saving to file in python
Question: I have a text that is part of html. I would like to save it to a file.

This works fine in debug mode in Eclipse, but fails on runtime from shell. I
am using a short example of html that fails.

    
    
    xx = '<input type="hidden" name="charset_test" value="€,´,€,´,水,Д,Є" />'
    with codecs.open('myfile.htm'), 'wb', encoding="utf-8") as output:
        output.write(data)
    

and I get:

    
    
     Exception 'ascii' codec can't decode byte 0xe2 in position XXX: ordinal not in range(128)
    

where XXX is the position in the relevant file of the "strange" symbols, such
as the EURO sign.

Why is this working from Eclipse and not from shell? How do I solve this?

I tried

    
    
    HTMLParser.HTMLParser().unescape()
    unquote()
    unicode()
    

Nothing worked...


Answer: The following code works for me...

    
    
    # coding=utf-8
    
    import codecs
    
    data = '<input type="hidden" name="charset_test" value="€,´,€,´,水,Д,Є" />'
    with codecs.open('myfile.htm', 'wb', encoding="utf-8") as output:
        output.write(data.decode('utf-8'))
    

...but if the source data is already UTF-8 encoded, and you also want to write
UTF-8 data, there's no need to decode it to a Python `unicode` object, then
re-encode back to UTF-8. You can just do...

    
    
    # coding=utf-8
    
    data = '<input type="hidden" name="charset_test" value="€,´,€,´,水,Д,Є" />'
    with open('myfile.htm', 'wb') as output:
        output.write(data)
    

