title: encoding error while using np.matrix and np.chararray
body: <p>I am building a Flask website in Spanish for people to send via mail encoded messages. Basically, you paste your text in a textfield and it returns its encoded version. The functions encode() and decode() below function OK until it deals with accentuated and other non-standard characters. My default system encoding is 'ascii', and I believe the problem may arise by me using numpy.matrix and numpy.chararray which might change the encoding of my string. </p>
  
  <p>When I build the code and test below in Sublime Text 2 I get a:</p>
  
  <pre><code>SyntaxError: Non-ASCII character '\xc3'... but no encoding declared;
  see http://www.python.org/peps/pep-0263.html for details
  </code></pre>
  
  <p>When I add</p>
  
  <pre><code>#!/usr/bin/env python
  #-*- coding: utf-8 -*-
  </code></pre>
  
  <p>to the code it runs in ST2, but it also spits an error and the decoded message is missing certain characters like so:</p>
  
  <pre><code>[Decode error - output not utf-8]
  
  La cr  a del le  n tiene dos a  os. 
  </code></pre>
  
  <p>When I run it in a local server with Flask i get:</p>
  
  <pre><code>UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 0: ordinal not in range(128)
  </code></pre>
  
  <p>I tried the chardet package and the items in the matrix are recognized as 'windows-1252'. I decode the items in the matrix using 'windows-1252' and 'cp1252' but the problems persists. I tried encoding with 'utf-8' after the previous decoding (i.e. with 'windows-1252') but it doesn't work. I suspect this is an encoding problem, but I am not entirely sure. Any leads on how to solve this are very much appreciated.</p>
  
  <p>This is the code:</p>
  
  <pre><code>import numpy as np
  import random, string, re
  
  def encode(message, size, token):
          """Assumes message is a string, size is the size limit of the message,
          and token is a string with unique characters, i.e. bufalo but not rana"""
  
          message = list(message)
  
          while len(message) &lt; size:
              sgn = random.choice(['*', '?', '&amp;', '@'])
              message.append(sgn)
  
          matrix = np.matrix(message)
          cols = size/5
  
          matrix = matrix.reshape((cols, 5)).T
          encoded = np.chararray(shape=(cols,5)).T
  
          token = token.lower()
          token = list(token)
          new = []
          for i in token:
              new.append(sorted(token).index(i))
  
          while len(new) &gt; 5:
              for i in new:
                  if i &gt;= (5):
                      new.remove(i)
  
          old = range(0,5)
  
          for o, n in zip(old, new):
              encoded[np.ix_([n], range(0, matrix.shape[1]))] = matrix[np.ix_([o], range(0, matrix.shape[1]))]
  
          encoded_str = ''
          for i in range((encoded.size)):
              encoded_str += encoded.item(i)
  
          return encoded_str
  
  #########################################
  #THIS IS A TEST
  #########################################
  mssg = "La cría del león tiene dos años."
  print encode(mssg, 120, 'bufalo')
  #########################################
  
  def decode(message, size, token):
          message = list(message)
  
          while len(message) &lt; size:
              sgn = random.choice(['*', '?', '&amp;', '@'])
              message.append(sgn)
  
          matrix = np.matrix(message)
  
          cols = size/5
          matrix = matrix.reshape((5, cols))
  
          token = token.lower()
          token = list(token)
          new = []
          for i in token:
              new.append(sorted(token).index(i))
          while len(new) &gt; 5:
              for i in new:
                  if i &gt;= (5):
                      new.remove(i)
          old = range(0,5)
  
          decoded = np.chararray(shape=(cols,5)).T
          for n, o in zip(old, new):
              decoded[np.ix_([n], range(0, matrix.shape[1]))] = matrix[np.ix_([o], range(0, matrix.shape[1]))]
  
          decoded =decoded.T
  
          decoded_str = ''
          for i in range((decoded.size)):
              decoded_str += decoded.item(i)
  
          decoded_str = re.sub('[^a-zA-Z0-9\n\.]', ' ', decoded_str)
          return decoded_str
  </code></pre>