Encoding in Python - non-English characters into a URL
Question: I’m trying bit for bit to write a geocoding script. There is a Danish
(official and free) web service, where I enter an address in the URL and get a
json file with all needed info.

I can’t find the right way to translate my Danish characters (æ,ø,å) when they
go into an URL. In the example I have included two different urls (containing
the address info). One – where the street is ‘Byvej’ works fine, and I get the
result I expect printed out in IDLE. (And I can get the lat/long too).

The other, where the street is ‘Bispegårdsvej’ gives nothing in IDLE. The
returned list is empty. The url works fine in a browser, and I know, I need to
add something to the script, I just can’t find anything that works.

I'm using ActivePython 2.7.2.5 Thanks, Tommy

    
    
    # -*- coding: cp1252 -*-
    import urllib2
    import json
    
    
    #url='http://geo.oiorest.dk/adresser.json?postnr=4682&vejnavn=Byvej&husnr=31'
    url='http://geo.oiorest.dk/adresser.json?postnr=4320&vejnavn=Bispegårdsvej&husnr=2'
    
    try:
        data = urllib2.urlopen(url).read()
        adresser = json.loads(data)
    
        for adresse in adresser:
            print "%s %s, %s %s" % \
                (adresse['vejnavn']['navn'],
                 adresse['husnr'],
                 adresse['postnummer']['nr'],
                 adresse['postnummer']['navn'])
    
    except urllib2.HTTPError, e:
        print "HTTP error: %d" % e.code
    except urllib2.URLError, e:
        print "Network error: %s" % e.reason.args[1]    
    


Answer: You need to encode the social characters with percent encoding, also known as
URL encode. After percent encoding, the URL should like like this:

    
    
    http://geo.oiorest.dk/adresser.json?postnr=4320&vejnavn=Bispeg%C3%A5rdsvej&husnr=2
    

Web services that complies with the IRI to URI mapping defined in RFC 3987
would use utf-8 for encoding after character normalization, but you should
need to check the documentation the service to be sure what encoding to use.

Python has urllib.quote() in the standard library to do percent encoding from
a string and urllib.urlencode() in the standard library to do percent encoding
from a dictionary or an iterable of two elements tuples to produce a string
for the query parameter.

