title: Python: convert Chinese characters into pinyin with CJKLIB
body: <p>I'm trying to convert a bunch of Chinese characters into pinyin, reading the characters from one file and writing the pinyin into another. I'm working with the CJKLIB functions to do this. </p>
  
  <p>Here's the code,</p>
  
  <pre><code>from cjklib.characterlookup import CharacterLookup
  
  source_file = 'cities_test.txt'
  dest_file = 'output.txt'
  
  s = open(source_file, 'r')
  d = open(dest_file, 'w')
  
  cjk = CharacterLookup('T')
  
  for line in s:
      p = line.split('\t')
      for p_shard in p:
          for c in p_shard:
              readings = cjk.getReadingForCharacter(c.encode('utf-8'), 'Pinyin')
              d.write(readings[0].encode('utf-8'))
          d.write('\t')
      d.write('\n')
  
  s.close()
  d.close()
  </code></pre>
  
  <p>My problem is that I keep running into Unicode-related errors, the error comes up when I call the <code>getReadingForCharacter</code> function. If I called it as written, </p>
  
  <pre><code>readings = cjk.getReadingForCharacter(c.encode('utf-8'), 'Pinyin')
  </code></pre>
  
  <p>I get: UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range (128). </p>
  
  <p>If I call it like this, without the <code>.encoding()</code>, </p>
  
  <pre><code>readings = cjk.getReadingForCharacter(c, 'Pinyin')
  </code></pre>
  
  <p>I get an error thrown by sqlalchemy (the CJKLIB uses sqlalchemy and sqlite): You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings ... etc. </p>
  
  <p>Can someone help me out? Thanks! </p>
  
  <p>Oh also, is there a way for CJKLIB to return the pinyin without any tones? I think by default it's returning pinyin with these weird characters to represent tones, I just want the letters without these tones. </p>