How can I get a random unicode string
Question: I am testing a REST based service and one of the inputs is a text string. So I
am sending it random unicode strings from my python code. So far the unicode
strings that I sent were in the ascii range, so everything worked.

Now I am attempting to send characters beyond the ascii range and I am getting
an encoding error. Here is my code. I have been through this
[link](http://nedbatchelder.com/text/unipain.html) and still unable to wrap my
head around it.

    
    
    # coding=utf-8
    
    import os, random, string
    import json
    
    junk_len = 512
    junk =  (("%%0%dX" % junk_len) % random.getrandbits(junk_len * 8))
    
    for i in xrange(1,5):
        if(len(junk) % 8 == 0):
            print u'decoding to hex'
            message = junk.decode("hex")
    
        print 'Hex chars %s' %message
        print u' '.join(message.encode("utf-8").strip())
    

The first line prints without any issues, but I can't send it to the REST
service without encoding it. Hence the second line where I am attempting to
encode it to utf-8. This is the line of code that fails with the following
message.

> UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position 7:
> ordinal not in range(128)


Answer: UTF-8 only allows certain bit patterns. You appear to be using UTF-8 in your
code, so you will need to conform to the allowed UTF-8 patterns.

    
    
    1 byte: 0b0xxxxxxx
    
    2 byte: 0b110xxxxx 0b10xxxxxx
    
    3 byte: 0b1110xxxx 0b10xxxxxx 0b10xxxxxx
    
    4 byte: 0b11110xxx 0b10xxxxxx 0b10xxxxxx 0b10xxxxxx
    

In the multi-byte patterns, the first byte indicates the number of bytes in
the whole pattern with leading 1s followed by 0 and data bits `x`. The non-
leading bytes all follow the same pattern: 0b10xxxxxx with two leading
indicator bits `10` and six data bits `xxxxxx`.

In general, randomly generated bytes will not follow these patterns. You can
only generate the data bits `x` randomly.

