Query by non-ascii charachters
Question: I am using Python, on Google App Engine platform. Let's say I have in my Data
Store the following code :

    
    
    class names(db.Model):
        name = db.StringProperty(multiline=True)
    

and there are names like :

    
    
    name1 = Beyoncé
    name2 = El Súper Clásico
    

with non-ascii charachters.

When I make a query like :

    
    
    q_1 = names.all().filter('name =', name1)
    

It doesn't work, the comparison is wrong.

Do you have any idea how can I solve this problem? I tried encoding the "name"
to UTF-8, but it didn't work also.

Thanks, Gori


Answer: There should be no problems with exact matches when correctly decoding input
strings (that you get from web request parameters) and correctly encoding
output strings (that you save in GAE data storage) in Unicode.

I've tried this snippet in the GAE SDK Interactive Console and it works:

    
    
    from google.appengine.ext import db
    
    class names(db.Model):
        name = db.StringProperty(multiline=True)
    
    some_name = 'Beyonc\xc3\xa9'.decode('utf-8')
        # same as: some_name = u'Beyoncé'
        # same as: some_name = u'Beyonc\u00e9'
    
    n = names(name=some_name)
    n.put()
    
    q = names.all().filter('name =', some_name)
    print q.get().name.encode('utf-8')
        # prints Beyoncé
    

You should debug what is the raw value of the strings you are comparing, i.e.,
the string saved in the storage and the string passed to the query.

I recommend reading this [article about Unicode by Joel
Spolsky](http://www.joelonsoftware.com/articles/Unicode.html) and the [Python
Unicode HOWTO](http://docs.python.org/howto/unicode.html) if you're not
familiar with handling Unicode strings.

In addition to this, if you're running search queries that should match
Unicode characters like `u'é'` when input is `'e'`, consider comparing
normalized strings:

    
    
    some_name = u'El S\u00faper Cl\u00e1sico' # El Súper Clásico
    normalized_name = unicodedata.normalize('NFKD', some_name).encode('ascii', 'ignore') # El Super Clasico
    

