Write to UTF-8 file in Python
Question: I'm really confused with the `codecs.open function`. When I do:

    
    
    file = codecs.open("temp", "w", "utf-8")
    file.write(codecs.BOM_UTF8)
    file.close()
    

It gives me the error

> UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0:
> ordinal not in range(128)

If I do:

    
    
    file = open("temp", "w")
    file.write(codecs.BOM_UTF8)
    file.close()
    

It works fine.

**Question** is why does the first method fail? And how do I insert the bom?

If the second method is the correct way of doing it, what the point of using
`codecs.open(filename, "w", "utf-8")`?


Answer: Disclaimer: I'm not a Python programmer.

I believe the problem is that
[`codecs.BOM_UTF8`](https://docs.python.org/library/codecs.html#codecs.BOM_UTF8)
is a byte string, not a Unicode string. I suspect the file handler is trying
to guess what you really mean based on "I'm meant to be writing Unicode as
UTF-8-encoded text, but you've given me a byte string!"

Try writing the Unicode string for the byte order mark (i.e. Unicode U+FEFF)
directly, so that the file just encodes that as UTF-8:

    
    
    import codecs
    
    file = codecs.open("lol", "w", "utf-8")
    file.write(u'\ufeff')
    file.close()
    

(That seems to give the right answer - a file with bytes EF BB BF.)

EDIT: S. Lott's [suggestion](http://stackoverflow.com/a/934203/12892) of using
"utf-8-sig" as the encoding is a better one than explicitly writing the BOM
yourself, but I'll leave this answer here as it explains what was going wrong
before.

