title: Bytes in a unicode Python string
body: <p>In Python 2, Unicode strings may contain both unicode and bytes:</p>
  
  <pre><code>a = u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba'
  </code></pre>
  
  <p>I understand that this is absolutely <strong>not something one should write</strong> in his own code, but this is a string that I have to deal with.</p>
  
  <p>The bytes in the string above are UTF-8 for <code>ек</code> (Unicode <code>\u0435\u043a</code>).</p>
  
  <p>My objective is to get a unicode string containing everything in Unicode, which is to say <code>Русский ек</code> (<code>\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \u0435\u043a</code>).</p>
  
  <p>Encoding it to UTF-8 yields</p>
  
  <pre><code>&gt;&gt;&gt; a.encode('utf-8')
  '\xd0\xa0\xd1\x83\xd1\x81\xd1\x81\xd0\xba\xd0\xb8\xd0\xb9 \xc3\x90\xc2\xb5\xc3\x90\xc2\xba'
  </code></pre>
  
  <p>Which then decoded from UTF-8 gives the initial string with bytes in them, which is not good:</p>
  
  <pre><code>&gt;&gt;&gt; a.encode('utf-8').decode('utf-8')
  u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \xd0\xb5\xd0\xba'
  </code></pre>
  
  <p>I found a hacky way to solve the problem, however:</p>
  
  <pre><code>&gt;&gt;&gt; repr(a)
  "u'\\u0420\\u0443\\u0441\\u0441\\u043a\\u0438\\u0439 \\xd0\\xb5\\xd0\\xba'"
  &gt;&gt;&gt; eval(repr(a)[1:])
  '\\u0420\\u0443\\u0441\\u0441\\u043a\\u0438\\u0439 \xd0\xb5\xd0\xba'
  &gt;&gt;&gt; s = eval(repr(a)[1:]).decode('utf8')
  &gt;&gt;&gt; s
  u'\\u0420\\u0443\\u0441\\u0441\\u043a\\u0438\\u0439 \u0435\u043a'
  # Almost there, the bytes are proper now but the former real-unicode characters
  # are now escaped with \u's; need to un-escape them.
  &gt;&gt;&gt; import re
  &gt;&gt;&gt; re.sub(u'\\\\u([a-f\\d]+)', lambda x : unichr(int(x.group(1), 16)), s)
  u'\u0420\u0443\u0441\u0441\u043a\u0438\u0439 \u0435\u043a' # Success!
  </code></pre>
  
  <p>This works fine but looks very hacky due to its use of <code>eval</code>, <code>repr</code>, and then additional regex'ing of the unicode string representation. Is there a cleaner way?</p>