title: Encoding a unicode string to utf-8 and getting question marks
body: <p>System:</p>
  
  <pre><code>Python 2.7.5 , IPython 2.3.1 , OSX terminal (local), sys.stdout.encoding : 'UTF-8'
  (venv) toz$ locale LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL= –
  </code></pre>
  
  <p>I am expecting both commands print the same but the latter prints question marks for ord(c)>128. Why is that? How can I encode this unicode string and iterate through without getting the question marks?</p>
  
  <pre><code>In [77]: for c in u'!"#%\'()*+,-./:;&lt;=&gt;?@[\]^_`{|}~’”“': print c.encode('utf-8'),
  ! " # % ' ( ) * + , - . / : ; &lt; = &gt; ? @ [ \ ] ^ _ ` { | } ~ ’ ” “
  
  In [78]: for c in u'!"#%\'()*+,-./:;&lt;=&gt;?@[\]^_`{|}~’”“'.encode('utf-8'): print c,
  ! " # % ' ( ) * + , - . / : ; &lt; = &gt; ? @ [ \ ] ^ _ ` { | } ~ ? ? ? ? ? ? ? ? ?
  </code></pre>
  
  <p>Lets print the ord values:</p>
  
  <pre><code>In [92]: for c in u'!"#%\'()*+,-./:;&lt;=&gt;?@[\]^_`{|}~’”“': print c.encode('utf-8'),ord(c),
  ! 33 " 34 # 35 % 37 ' 39 ( 40 ) 41 * 42 + 43 , 44 - 45 . 46 / 47 : 58 ; 59 &lt; 60 = 61 &gt; 62 ? 63 @ 64 [ 91 \ 92 ] 93 ^ 94 _ 95 ` 96 { 123 | 124 } 125 ~ 126 ’ 8217 ” 8221 “ 8220
  
  In [93]: for c in u'!"#%\'()*+,-./:;&lt;=&gt;?@[\]^_`{|}~’”“': print c.encode('utf-8'),ord(c.encode('utf-8')),
  ! 33 " 34 # 35 % 37 ' 39 ( 40 ) 41 * 42 + 43 , 44 - 45 . 46 / 47 : 58 ; 59 &lt; 60 = 61 &gt; 62 ? 63 @ 64 [ 91 \ 92 ] 93 ^ 94 _ 95 ` 96 { 123 | 124 } 125 ~ 126 ’---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-93-1eb3985b825b&gt; in &lt;module&gt;()
  ----&gt; 1 for c in u'!"#%\'()*+,-./:;&lt;=&gt;?@[\]^_`{|}~’”“': print c.encode('utf-8'),ord(c.encode('utf-8')),
  
  TypeError: ord() expected a character, but string of length 3 found
  </code></pre>
  
  <p>I can relate this length of three to nine question marks as well as the required (three) bytes for each of the three non-ascii characters.</p>