Convert escaped utf-8 string to utf in python 3
Question: I have a py3 string that includes escaped utf-8 sequencies, such as
"Company\\\ffffffc2\\\ffffffae", which I would like to convert to the correct
utf 8 string (which would in the example be "Company®", since the escaped
sequence is c2 ae). I've tried

    
    
    print (bytes("Company\\\\ffffffc2\\\\ffffffae".replace(
        "\\\\ffffff", "\\x"), "ascii").decode("utf-8"))
    

result: Company\xc2\xae

    
    
     print (bytes("Company\\\\ffffffc2\\\\ffffffae".replace (
         "\\\\ffffff", "\\x"), "ascii").decode("unicode_escape"))
    

result: CompanyÂ®

(wrong, since chracters are treated separately, but they should be treated
together.

If I do

    
    
    print (b"Company\xc2\xae".decode("utf-8"))
    

It gives the correct result. Company®

How can i achieve that programmatically (i.e. starting from a py3 str)


Answer: A simple solution is:

    
    
    import ast
    
    test_in = "Company\\\\ffffffc2\\\\ffffffae"
    test_out = ast.literal_eval("b'''" + test_in.replace('\\\\ffffff','\\x') + "'''").decode('utf-8')
    print(test_out)
    

However it will fail if there is a triple quote `'''` in the input string
itself.

* * *

Following code does not have this problem, but it is not as simple as the
first one.

In the first step the string is split on a regular expression. The odd items
are ascii parts, e.g. `"Company"`; each even item corresponds to one escaped
utf8 code, e.g. `"\\\\ffffffc2"`. Each substring is converted to bytes
according to its meaning in the input string. Finally all parts are joined
together and decoded from bytes to a string.

    
    
    import re
    
    REGEXP = re.compile(r'(\\\\ffffff[0-9a-f]{2})', flags=re.I)
    
    def convert(estr):
        def split(estr):
            for i, substr in enumerate(REGEXP.split(estr)):
                if i % 2:
                    yield bytes.fromhex(substr[-2:])
                elif substr:
                    yield bytes(substr, 'ascii')
        return b''.join(split(estr)).decode('utf-8')
    
    
    test_in = "Company\\\\ffffffc2\\\\ffffffae"
    print(convert(test_in))
    

The code could be optimized. Ascii parts do not need encode/decode and
consecutive hex codes should be concatenated.

