Unicode and urllib.open
Question: I am creating an application in python that can parse weather data from
[yr.no](http://yr.no "yr.no") in Python. It works fine with regular ASCII
strings, but fails when I use unicode.

    
    
    def GetYRNOWeatherData(country, province, place):
    
        #Parse the XML file
    
        wtree = ET.parse(urllib.urlopen("http://www.yr.no/place/" + string.replace(country, ' ', '_').encode('utf-8') + "/" + string.replace(province, ' ', '_').encode('utf-8') + "/" + string.replace(place, ' ', '_').encode('utf-8') + "/forecast.xml"))
    

For example, when I try

    
    
    GetYRNOWeatherData("France", "Île-de-France", "Paris")
    

I get this error

    
    
    'charmap' codec can't encode character u'\xce' in position 0: character maps to <undefined>
    

Is it true that urllib doesn't handle unicode very well? Since I am using
Tkinter as a frontend to this function, would that be the source of the
problem (does the Tkinter Entry widget handle unicode well?)


Answer: You can handle this by keeping every string as a `unicode` right up until you
actually make the `urllib.urlopen` request, at which point you `encode` to
`utf-8`:

    
    
    #!/usr/bin/python
    # -*- coding: utf-8 -*-
    
    # This import makes all literal strings in the file default to
    # type 'unicode' rather than type 'str'. You don't need to use this,
    # but you'd need to do u"France" instead of just "France" below, and
    # everywhere else you have a string literal.
    from __future__ import unicode_literals
    
    import urllib
    import xml.etree.ElementTree as ET
    
    def do_format(*args):
        ret = []
        for arg in args:
            ret.append(arg.replace(" ", "_"))
        return ret 
    
    
    def GetYRNOWeatherData(country, province, place):
        country, province, place = do_format(country, province, place)
        url = "http://www.yr.no/place/{}/{}/{}/forecast.xml".format(country, province, place)
        wtree = ET.parse(urllib.urlopen(url.encode('utf-8')))
        return wtree
    
    
    if __name__ == "__main__":
        GetYRNOWeatherData("France", "Île-de-France", "Paris")
    

