title: How to itereate over special characters in a python string
body: <p>Consider the following string of special characters:</p>
  
  <pre><code>x = "óیďÚÚ懇償燥績凡壇壁曇ÏエÀэүウーー」ÆØøæგბთლõшүжҮÿதணடஇஉுூொெௌДВБйЫСچخرسسبŞÛşکلںغখঙঝডইঊওোéñÑÜßẞÖÄäöÜĦĦ"
  </code></pre>
  
  <p>when printed in ipython:</p>
  
  <pre><code>In [11]: x = "óیďÚÚ懇償燥績凡壇壁曇ÏエÀэүウーー」ÆØøæგბთლõшүжҮÿதணடஇஉுூொெௌДВБйЫСچخرسسبŞÛşکلںغখঙঝডইঊওোéñÑÜßẞÖÄäöÜĦĦ"
  In [12]: x
  
  Out[12]: '\xc3\xb3\xdb\x8c\xc4\x8f\xc3\x9a\xc3\x9a\xe6\x87\x87\xe5\x84\x9f\xe7\x87\xa5\xe7\xb8\xbe\xe5\x87\xa1\xe5\xa3\x87\xe5\xa3\x81\xe6\x9b\x87\xc3\x8f\xe3\x82\xa8\xc3\x80\xd1\x8d\xd2\xaf\xe3\x82\xa6\xe3\x83\xbc\xe3\x83\xbc\xe3\x80\x8d\xc3\x86\xc3\x98\xc3\xb8\xc3\xa6\xe1\x83\x92\xe1\x83\x91\xe1\x83\x97\xe1\x83\x9a\xc3\xb5\xd1\x88\xd2\xaf\xd0\xb6\xd2\xae\xc3\xbf\xe0\xae\xa4\xe0\xae\xa3\xe0\xae\x9f\xe0\xae\x87\xe0\xae\x89\xe0\xaf\x81\xe0\xaf\x82\xe0\xaf\x8a\xe0\xaf\x86\xe0\xaf\x8c\xd0\x94\xd0\x92\xd0\x91\xd0\xb9\xd0\xab\xd0\xa1\xda\x86\xd8\xae\xd8\xb1\xd8\xb3\xd8\xb3\xd8\xa8\xc5\x9e\xc3\x9b\xc5\x9f\xda\xa9\xd9\x84\xda\xba\xd8\xba\xe0\xa6\x96\xe0\xa6\x99\xe0\xa6\x9d\xe0\xa6\xa1\xe0\xa6\x87\xe0\xa6\x8a\xe0\xa6\x93\xe0\xa7\x8b\xc3\xa9\xc3\xb1\xc3\x91\xc3\x9c\xc3\x9f\xe1\xba\x9e\xc3\x96\xc3\x84\xc3\xa4\xc3\xb6\xc3\x9c\xc4\xa6\xc4\xa6'
  </code></pre>
  
  <p>This string is passed to the bellow code from another service as a list:</p>
  
  <pre><code>value_list = []
  value_list.append(x) 
  </code></pre>
  
  <p>The goal of the bellow code is to find all the special characters in the given string and return them as a list. this list is to be parsed as a text in utf-8</p>
  
  <pre><code>In [33]: value_list
  
  Out[33]: ['\xc3\xb3\xdb\x8c\xc4\x8f\xc3\x9a\xc3\x9a\xe6\x87\x87\xe5\x84\x9f\xe7\x87\xa5\xe7\xb8\xbe\xe5\x87\xa1\xe5\xa3\x87\xe5\xa3\x81\xe6\x9b\x87\xc3\x8f\xe3\x82\xa8\xc3\x80\xd1\x8d\xd2\xaf\xe3\x82\xa6\xe3\x83\xbc\xe3\x83\xbc\xe3\x80\x8d\xc3\x86\xc3\x98\xc3\xb8\xc3\xa6\xe1\x83\x92\xe1\x83\x91\xe1\x83\x97\xe1\x83\x9a\xc3\xb5\xd1\x88\xd2\xaf\xd0\xb6\xd2\xae\xc3\xbf\xe0\xae\xa4\xe0\xae\xa3\xe0\xae\x9f\xe0\xae\x87\xe0\xae\x89\xe0\xaf\x81\xe0\xaf\x82\xe0\xaf\x8a\xe0\xaf\x86\xe0\xaf\x8c\xd0\x94\xd0\x92\xd0\x91\xd0\xb9\xd0\xab\xd0\xa1\xda\x86\xd8\xae\xd8\xb1\xd8\xb3\xd8\xb3\xd8\xa8\xc5\x9e\xc3\x9b\xc5\x9f\xda\xa9\xd9\x84\xda\xba\xd8\xba\xe0\xa6\x96\xe0\xa6\x99\xe0\xa6\x9d\xe0\xa6\xa1\xe0\xa6\x87\xe0\xa6\x8a\xe0\xa6\x93\xe0\xa7\x8b\xc3\xa9\xc3\xb1\xc3\x91\xc3\x9c\xc3\x9f\xe1\xba\x9e\xc3\x96\xc3\x84\xc3\xa4\xc3\xb6\xc3\x9c\xc4\xa6\xc4\xa6']
  
  In [34]: separator = re.compile('[.,;:!?&amp;()]+', re.MULTILINE | re.UNICODE)
  
  In [35]: value_list = [" ".join([word for word in separator.sub(' ', value).split()]).strip() for value in value_list]
  
  In [36]: word_found = []
  
  In [37]: for value in value_list:
               word_found.extend([i for i in value if 31 &gt; ord(i) or ord(i) &gt; 127])
           ....: 
  
  In [39]: word_found.pop().encode('utf-8')
  ---------------------------------------------------------------------------
  UnicodeDecodeError                        Traceback (most recent call last)
  &lt;ipython-input-39-61e9eca29caa&gt; in &lt;module&gt;()
  ----&gt; 1 word_found.pop().encode('utf-8')
  
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 0: ordinal not in range(128)
  </code></pre>
  
  <p>It is clear, that python is reading x as a python string (which has each \x character showing the higher and lower byte). While iterating over the characters in the string, we are actually iterating over the bytes instead of the character in the original string. Because of this, ord is giving them bytes as special characters and putting in the list. When encoded to utf-8 the out of range error is coming because we are trying to decode a part of the original character in utf-8. </p>
  
  <p>I need to understand, how can i iterate over this python string without changing the way in which the value is passed into value_list or read from words_found</p>
  
  <p>Please help.</p>