why output list is empty in my code in Python 2.7
Question: Using Python 2.7 and trying to do simple tokenization on UTF-8 encoded files.
The output of `a` seems a byte string, which is expected, since after
`tk[0].encode('utf-8')`, it converts from Python `unicode` type to `str/byte`.
My major confusion is why output of `b` is empty list? I think without
encoding (I mean without calling `.encode('utf-8')`), it should be raw unicode
character (e.g. I expect some Chinese character printed, as `1.txt` is UTF-8
encoded Chinese character file).

**Source code** ,

    
    
    import jieba
    
    if __name__ == "__main__":
    
        with open('1.txt', 'r') as content_file:
            content = content_file.read()
    
        segment_list = jieba.tokenize(content.decode('utf-8'), mode='search')
        if segment_list is None:
            print 'segment is None'
        else:
            a = [tk[0].encode('utf-8') for tk in segment_list]
            b = [tk[0] for tk in segment_list]
    
            print a
            print b
    

** Output **,

    
    
    ['\xe4\xb8\x8a\xe6\xb5\xb7', '\xe6\xb5\xb7\xe5\xb8\x82', '\xe4\xb8\x8a\xe6\xb5\xb7\xe5\xb8\x82', '\xe6\xb7\xb1\xe5\x9c\xb3', '\xe6\xb7\xb1\xe5\x9c\xb3\xe5\xb8\x82', '\xe7\xa6\x8f\xe7\x94\xb0', '\xe7\xa6\x8f\xe7\x94\xb0\xe5\x8c\xba', '\xe6\xa2\x85\xe6\x9e\x97', '\xe6\x9e\x97\xe8\xb7\xaf', '\xe6\xa2\x85\xe6\x9e\x97\xe8\xb7\xaf', '\xe4\xb8\x8a\xe6\xb5\xb7', '\xe6\xb5\xb7\xe5\xb8\x82', '\xe6\xb5\xa6\xe4\xb8\x9c', '\xe6\x96\xb0\xe5\x8c\xba', '\xe4\xb8\x8a\xe6\xb5\xb7\xe5\xb8\x82', '\xe4\xb8\x8a\xe6\xb5\xb7\xe5\xb8\x82\xe6\xb5\xa6\xe4\xb8\x9c\xe6\x96\xb0\xe5\x8c\xba', '\xe8\x80\x80\xe5\x8d\x8e', '\xe8\xb7\xaf', '\r\n']
    []
    


Answer: It appears that `jieba.tokenize()` returns a generator. A generator can be
iterated over only once. Better do

    
    
        b = [tk[0] for tk in segment_list]
        a = [tk.encode('utf-8') for tk in b]
    

