UTF-8 "inconsistency" when storing unicode key/values from CSV file into dict
Question: I always get very confused when dealing with Unicode and UTF-8 characters and
encodings in Python. There's probably a simple explanation to what I'm going
to detail below, but, so far, I can't wrap my head around it.

Let's say I have a very very simple `.csv` file that contains non-ascii
characters:

`tildes.csv`:

    
    
    Año,Valor
    2001,Café
    2002,León
    

I want to read that file using a
[csv.DictReader](https://docs.python.org/2/library/csv.html#csv.DictReader)
object and store its key/values as `unicode` strings, with tildes and such
handled properly (unescaped) in a python dict. I've seen Tornado and Django
handling unicode key/value sets properly, so I said to myself _Yep, I can do
that too!!_... But nopes... it looks like I can't.

    
    
    import csv
    
    with open('tildes.csv', 'r') as csv_f:
        reader = csv.DictReader(csv_f)
        for dct in reader:
            print "dct (original): %s" % dct
            for k, v in dct.items():
                print '%s: %s' % (unicode(k, 'utf-8'), unicode(v, 'utf-8'))
            utf_dct = dict((unicode(k, 'utf-8'), unicode(v, 'utf-8')) \
                      for k, v in dct.items())
            print utf_dct
    

So, I thought: _Ok, I read a`dict` from the file (Its keys being `Año` and
`Valor`) which will be loaded **ascii** with escaped characters, but then I
can encode those into unicode values and use them as keys..._ Wrong!

This is what I see when I run the code above:

    
    
    dct (original): {'A\xc3\xb1o': '2001', 'Valor': 'Caf\xc3\xa9'}
    Año: 2001
    Valor: Café
    {u'A\xf1o': u'2001', u'Valor': u'Caf\xe9'}
    dct (original): {'A\xc3\xb1o': '2002', 'Valor': 'Le\xc3\xb3n'}
    Año: 2002
    Valor: León
    {u'A\xf1o': u'2002', u'Valor': u'Le\xf3n'}
    

So the first line shows the dictionary 'as it is' (escaped). Good, nothing odd
here. Then I `print` all the key/values parsed to unicode. It shows the
characters the way I want it. Good too. But then, using the exact same
instruction I used to re-encode the strings when I printed them, I try to
create a `dict` (the `utf_dct` variable) and when I print it, I get the values
escaped again.

* * *

**EDIT 1** :

Actually, I don't think I even need a csv file to show what I mean. I just
tried this in my console:

    
    
    >>> print "Año"
    Año                      # Yeey!! There's hope!
    >>> print {"Año": 2001}
    {'A\xc3\xb1o': 2001}     # 2 chars --> Ascii, I think I get this part 
    >>> print {u"Año": 2001}
    {u'A\xf1o': 2001}        # What happened here? 
                             # Why am I seeing the 0x00F1 UTF-8 code 
                             # from the Latin-1 Supplement (wiki:
                             # http://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_block)
                             # instead of an ñ?
    

Why can't I just print a dict showing `{u'Año': 2001}`? My terminal clearly
accepts it. What's happening here?


Answer: When you print the string itself, it is printed "nicely", using its `str()`
representation. When you print a dictionary, its contents are printed using
their `repr()` representation, which always escapes. The contents of the
string are the same in both cases, it's just that Python displays them
differently. It is the same reason that no quote marks are printed around
`Año` in the first case, but quote marks are printed around `'A\xc3\xb1o'` in
the second case. It's just two different display formats.

Here is an even simpler example that may help illustrate the situation:

    
    
    >>> import unicodedata
    >>> unicodedata.name('\u00f1') # 00F1 is unicode code point for this character
    'LATIN SMALL LETTER N WITH TILDE'
    >>> print(str(u'\u00f1')) # str() gives a displayable character
    ñ
    >>> print repr(u'\u00f1') # repr() gives an escaped representation
    u'\xf1'
    >>> print repr(str(u'\u00f1')) # repr() of the str() shows the two characters in the UTF-8 encoding -- this is what happens when showing a dict
    '\xc3\xb1'
    >>> len(str(u'\u00f1')) # the str() is two bytes long (UTF-8 encoded)
    2
    >>> len(repr(u'\u00f1')) # the repr() is 7 bytes long (`u`, `'`, `\`, `x`, `f`, `1`, `'`)
    7
    

There is a [related bug report](http://bugs.python.org/issue2630) suggesting
to change this behavior so that `repr` doesn't escape non-ASCII characters.
According to that bug report, this change was made in Python 3, so tools that
you have seen doing this may be using Python 3.

It's also possible for individual tools to display anything however they like.
A tool doesn't have to just call `str(someDict)` and display the result; if it
wants, it can "manually" call `str` on the contents of the dict instead and
build up its own displayable version from that.

