Python script chokes on a downloaded file because of unicode encode error
Question: I run a script 4 times a day that uses the requests module to download a file,
which I then throw into a database. 9 times out of 10, the script works
flawlessly. But the times it does not work is because of a character in the
downloaded file that my script, as it is, does not like. For example, here's
the error I got today: `UnicodeEncodeError: 'ascii' codec can't encode
characters in position 379-381: ordinal not in range(128)`. I downloaded the
file another way and here's the character at position 380 which I believe is
responsible for stopping my script, "∞". And, here's the place in my script
where it chokes:

    
    
    ##### request file
    
    r = requests.get('https://resources.example.com/requested_file.csv')
    
    ##### create the database importable csv file
    
    ld = open('/requested_file.csv', 'w')
    print(r.text, file=ld)
    

I know this probably has to do with encoding the file somehow before printing
it to the .csv file, and is probably a simple thing for someone who knows what
they are doing but, after many hours of research, I'm about to cry. Thanks for
your help in advance!


Answer: You need to provide an _encoding_ for your file; currently it defaults to
ASCII, which is a very limited codec.

You could use UTF-8 instead, for example:

    
    
    with open('/requested_file.csv', 'w', encoding='utf8') as ld:
        print(r.text, file=ld)
    

However, since you are loading from a URL you are now decoding then encoding
again. A better idea is to just copy the data straight to disk as bytes. Make
a _streaming_ request and have `shutil.copyfileobj()` copy the data in chunks.
That way you can handle any size of response without loading everything into
memory:

    
    
    import requests
    import shutil
    
    r = requests.get('https://resources.example.com/requested_file.csv', stream=True)
    with open('/requested_file.csv', 'wb') as ld:
        r.raw.decode_content = True  # decompress gzip or deflate responses
        shutil.copyfileobj(r.raw, ld)
    

