Hashlib md5 in python returning incorrect digests for some unicode characters?
Question: I have been playing around with python's and Java's MD5 implementation and
encountered this quirk that's been puzzling me.

The following python script illustrates the problem:

    
    
    # -*- coding: utf-8 -*-
    import hashlib
    
    def md5hash(x):
      m = hashlib.md5()
      m.update(x)
      return m.hexdigest()
    
    print md5hash('\xdb')
    print md5hash('Û')
    

Output:

    
    
    98fd00d788afe2a5fa5e4f8e1666638b
    31ecfb09f120720a55d96a2034f5d00b
    

I expected the two digests to be equivalent, given that `Û` ought to be
equivalent to `\xdb`. I built an equivalent implementation in Java to get some
more insight:

    
    
    import java.security.MessageDigest;
    import java.security.NoSuchAlgorithmException;
    
    public class Test {
      public static void main(String[] args) throws Exception {
        MessageDigest m = MessageDigest.getInstance("MD5");
    
        m.update("\u00db".getBytes());
        System.out.println(bytesToHex(m.digest()));
    
        m.update("Û".getBytes());
        System.out.println(bytesToHex(m.digest()));
      }
    
      final protected static char[] hexArray = "0123456789abcdef".toCharArray();
      public static String bytesToHex(byte[] bytes) {
        char[] hexChars = new char[bytes.length * 2];
        for ( int j = 0; j < bytes.length; j++ ) {
          int v = bytes[j] & 0xFF;
          hexChars[j * 2] = hexArray[v >>> 4];
          hexChars[j * 2 + 1] = hexArray[v & 0x0F];
        }
        return new String(hexChars);
      }
    }
    

Output:

    
    
    31ecfb09f120720a55d96a2034f5d00b
    31ecfb09f120720a55d96a2034f5d00b
    

The output of Java is consistent, as expected. This is leading me to believe
that `md5hash('\xdb')` is outputting an incorrect result, but I am not sure
what it is that I'm missing. Any thoughts?


Answer: Your assumptions are incorrect. You started your Python source code with:

    
    
    # -*- coding: utf-8 -*-
    

`Û` is **not** equivalent to `\xdb` in that case; it is instead two bytes:

    
    
    >>> u'Û'.encode('utf8')
    '\xc3\x9b'
    

Python is entirely consistent here:

    
    
    >>> import hashlib
    >>> hashlib.md5('\xc3\x9b').hexdigest()
    '31ecfb09f120720a55d96a2034f5d00b'
    >>> hashlib.md5('\xdb').hexdigest()
    '98fd00d788afe2a5fa5e4f8e1666638b'
    

In Java you started with a Unicode codepoint instead, converting that to UTF-8
bytes:

    
    
    "\u00db".getBytes()
    

The Python equivalent would be to use a `unicode` string literal and either a
`\uhhhh` or `\xhh` escape sequence:

    
    
    >>> u'\u00db'.encode('utf8')
    '\xc3\x9b'
    >>> u'\xdb'.encode('utf8')
    '\xc3\x9b'
    

Note the `u` prefix to produce a `unicode` string. `\xdb` without a `u` prefix
is a _byte string_ , not a Unicode codepoint, and only if you were to decode
it as _Latin 1_ would you get the same Unicode string:

    
    
    >>> '\xdb'.decode('latin1')
    u'\xdb'
    >>> '\xdb'.decode('latin1').encode('utf8')
    '\xc3\x9b'
    

You probably want to study up on Python and Unicode; see:

  * The [Python Unicode HOWTO](http://docs.python.org/2/howto/unicode.html)

  * [Pragmatic Unicode](http://nedbatchelder.com/text/unipain.html) by Ned Batchelder

and for completeness sake:

  * [The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)](http://joelonsoftware.com/articles/Unicode.html) by Joel Spolsky

