How to encode UTF-8 strings with only "A-Z","a-z","0-9", and "_" in Python
Question: I need to build a python encoder so that I can reformat strings like this:

    
    
    import codecs
    codecs.encode("Random  UTF-8 String ☑⚠⚡", 'name_of_my_encoder')
    

The reason this is even something I'm asking stack overflow is, the encoded
strings need to pass this validation function. This is a hard constraint,
there is no flexibility on this, its due to how the strings have to be stored.

    
    
    from string import ascii_letters
    from string import digits
    
    valid_characters = set(ascii_letters + digits + ['_'])
    
    def validation_function(characters):
        for char in characters:
            if char not in valid_characters:
                raise Exception
    

Making an encoder seemed easy enough, but I'm not sure if this encoder is
making it harder to build a decoder. Heres the encoder I've written.

    
    
    from codecs import encode
    from string import ascii_letters
    from string import digits
    
    ALPHANUMERIC_SET = set(ascii_letters + digits)
    
    def underscore_encode(chars_in):
        chars_out = list()
        for char in chars_in:
            if char not in ALPHANUMERIC_SET:
                chars_out.append('_{}_'.format(encode(char.encode(), 'hex').decode('ascii')))
            else:
                chars_out.append(char)
        return ''.join(chars_out)
    

This is the encoder I've written. I've only included it for example purposes,
theres probably a better way to do this.

Edit 1 - Someone has wisely pointed out just using base32 on the entire
string, which I can definitely use. However, it would be preferable to have
something that is 'somewhat readable', so an escaping system like
<https://en.wikipedia.org/wiki/Quoted-printable> or
<https://en.wikipedia.org/wiki/Percent-encoding> would be preferred.

Edit 2 - Proposed solutions must work on Python 3.4 or newer, working in
Python 2.7 as well is nice, but not required. I've added the python-3.x tag to
help clarify this a little.


Answer: This seems to do the trick. Basically, alphanumeric letters are left alone.
Any non-alphanumeric character in the ASCII set is encoded as a `\xXX` escape
code. All other unicode characters are encoded using the `\uXXXX` escape code.
However, you've said you can't use `\`, but you can use `_`, thus all escape
sequences are translated to start with a `_`. This makes decoding extremely
simple. Just replace the `_` with `\` and then use the `unicode-escape` codec.
Encoding is slightly more difficult as the `unicode-escape` codec leaves ASCII
characters alone. So first you have to escape the relevant ASCII characters,
then run the string through the `unicode-escape` codec, before finally
translating all `\` to `_`.

Code:

    
    
    from string import ascii_letters, digits
    
    # non-translating characters
    ALPHANUMERIC_SET = set(ascii_letters + digits)    
    # mapping all bytes to themselves, except '_' maps to '\'
    ESCAPE_CHAR_DECODE_TABLE = bytes(bytearray(range(256)).replace(b"_", b"\\"))
    # reverse mapping -- maps `\` back to `_`
    ESCAPE_CHAR_ENCODE_TABLE = bytes(bytearray(range(256)).replace(b"\\", b"_"))
    # encoding table for ASCII characters not in ALPHANUMERIC_SET
    ASCII_ENCODE_TABLE = {i: u"_x{:x}".format(i) for i in set(range(128)) ^ set(map(ord, ALPHANUMERIC_SET))}
    
    
    
    def encode(s):
        s = s.translate(ASCII_ENCODE_TABLE) # translate ascii chars not in your set
        bytes_ = s.encode("unicode-escape")
        bytes_ = bytes_.translate(ESCAPE_CHAR_ENCODE_TABLE)
        return bytes_
    
    def decode(s):
        s = s.translate(ESCAPE_CHAR_DECODE_TABLE)
        return s.decode("unicode-escape")
    
    s = u"Random UTF-8 String ☑⚠⚡"
    #s = '北亰'
    print(s)
    b = encode(s)
    print(b)
    new_s = decode(b)
    print(new_s)
    

Which outputs:

    
    
    Random UTF-8 String ☑⚠⚡
    b'Random_x20UTF_x2d8_x20String_x20_u2611_u26a0_u26a1'
    Random UTF-8 String ☑⚠⚡
    

This works on both python 3.4 and python 2.7, which is why the
`ESCAPE_CHAR_{DE,EN}CODE_TABLE` is a bit messy `bytes` on python 2.7 is an
alias for `str`, which works differently to `bytes` on python 3.4. This is why
the table is constructed using a `bytearray`. For python 2.7, the `encode`
method expects a `unicode` object not `str`.

