Title: Python 'ascii' codec can't encode character error
Tags: python;python-3.x;codec
Question: I was trying to read utf-16 file. This code is working fine on local machine but when I ran it on AWS Batch service then it's giving above error.
import codecs
file_to_split = codecs.open("file_utf_16.txt", 'r+', "UTF-16")
It throws an exception header 'ascii' codec can't encode character '\ufeff' in position 0: ordinal not in range(128)
I am using Python 3.6.
Comment: Thanks but it didn't work.

Comment: the file needs to be decoded with `utf-16le` not `utf-16`


Here is another answer: The first few bytes are the BOM (Byte Order Mark). This is a sequence of bytes at the beginning of a Unicode document that tells you the byte order, encoding, etc.
This Wikipedia article will help explain the technical details:
Byte order mark
You have a couple of choices. This answer shows various methods of handling BOM for different encodings: Stack Overflow answer
And this answer shows you how to properly read the file into memory, which is what I think that you want to do. Stack Overflow answer. This way you can then process the file as a normal Python string.

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2018:00:07/11/18', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python><python-3.x><codec>', 'Title': "Python 'ascii' codec can't encode character error", 'ViewCount': 1099, 'id': 51283161, 'user_AboutMe': '<p>I am a passionate Senior Software Engineer. Majorly work in Java and BigData. I completed my masters degree in computer science and since then I developed various business application on different domains including financial systems, digitalsignage, security etc. I am also a good web developer and worked on different websites such as blogs, shopping carts. I have good understanding of programming languages and software development pros and cons.</p>\n', 'user_AccountId': 184519.0, 'user_DisplayName': 'Muhammad Imran Tariq', 'user_Reputation': 21886.0, 'user_Views': 2645.0, 'user_id': 420613.0}}