title: UnicodeDecodeError error when loading word2vec
body: <p><strong>Full Description</strong></p>
  
  <p>I am starting to work with word embedding and found a great amount of information about it. I understand, this far, that I can train my own word vectors or use previously trained ones, such as Google's or Wikipedia's, which are available for the English language and aren't useful to me, since I am working with texts in <em>Brazilian Portuguese</em>. Therefore, I went on a hunt for pre-trained word vectors in Portuguese and I ended up finding <a href="http://ahogrammer.com/2017/01/20/the-list-of-pretrained-word-embeddings/" rel="nofollow noreferrer">Hirosan's List of Pretrained Word Embeddings</a> which led me to Kyubyong's <a href="https://github.com/Kyubyong/wordvectors" rel="nofollow noreferrer">WordVectors</a> from which I learned about Rami Al-Rfou's <a href="https://sites.google.com/site/rmyeid/projects/polyglot" rel="nofollow noreferrer">Polyglot</a>. After downloading both, I unsuccessfully have been trying to simply load the word vectors.</p>
  
  <p><strong>Short Description</strong></p>
  
  <p>I can't load pre-trained word vectors; I am trying <a href="https://github.com/Kyubyong/wordvectors" rel="nofollow noreferrer">WordVectors</a> and <a href="https://sites.google.com/site/rmyeid/projects/polyglot" rel="nofollow noreferrer">Polyglot</a>.</p>
  
  <p><strong>Downloads</strong></p>
  
  <ul>
  <li><a href="https://drive.google.com/open?id=0B0ZXk88koS2KRDcwcV9IVWFTeUE" rel="nofollow noreferrer">Kyubyong's pre-trained word2vector format word vectors for Portuguese</a>;</li>
  <li><a href="https://doc-0g-54-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/c1ch6rdnp89glqmi8g81ev2somslu7cs/1527537600000/10341224892851088318/*/0B5lWReQPSvmGNEh0VTdmSHlHZ1k?e=download" rel="nofollow noreferrer">Polyglot's pre-trained word vectors for Portuguese</a>;</li>
  </ul>
  
  <p><strong>Loading attempts</strong></p>
  
  <p><em>Kyubyong's <a href="https://github.com/Kyubyong/wordvectors" rel="nofollow noreferrer">WordVectors</a></em>
  First attempt: using Gensim as suggested by <a href="http://ahogrammer.com/2017/01/20/the-list-of-pretrained-word-embeddings/" rel="nofollow noreferrer">Hirosan</a>;</p>
  
  <pre><code>from gensim.models import KeyedVectors
  kyu_path = '.../pre-trained_word_vectors/kyubyong_pt/pt.bin'
  word_vectors = KeyedVectors.load_word2vec_format(kyu_path, binary=True)
  </code></pre>
  
  <p>And the error returned:</p>
  
  <pre><code>[...]
  File "/Users/luisflavio/anaconda3/lib/python3.6/site-packages/gensim/utils.py", line 359, in any2unicode
  return unicode(text, encoding, errors=errors)
  
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
  </code></pre>
  
  <p>The zip downloaded also contains other files but all of them return similar errors.</p>
  
  <p><em><a href="https://sites.google.com/site/rmyeid/projects/polyglot" rel="nofollow noreferrer">Polyglot</a></em>
  First attempt: following <a href="http://nbviewer.jupyter.org/gist/aboSamoor/6046170" rel="nofollow noreferrer">Al-Rfous's instructions</a>;</p>
  
  <pre><code>import pickle
  import numpy
  pol_path = '.../pre-trained_word_vectors/polyglot/polyglot-pt.pkl'
  words, embeddings = pickle.load(open(pol_path, 'rb'))
  </code></pre>
  
  <p>And the error returned:</p>
  
  <pre><code>File "/Users/luisflavio/Desktop/Python/w2v_loading_tries.py", line 14, in &lt;module&gt;
      words, embeddings = pickle.load(open(polyglot_path, "rb"))
  
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xd4 in position 1: ordinal not in range(128)
  </code></pre>
  
  <p>Second attempt: using <a href="https://polyglot.readthedocs.io/en/latest/Embeddings.html" rel="nofollow noreferrer">Polyglot's word embedding load function</a>;</p>
  
  <p>First, we have to install polyglot via pip:</p>
  
  <pre><code>pip install polyglot
  </code></pre>
  
  <p>Now we can import it:</p>
  
  <pre><code>from polyglot.mapping import Embedding
  pol_path = '.../pre-trained_word_vectors/polyglot/polyglot-pt.pkl'
  embeddings = Embedding.load(polyglot_path)
  </code></pre>
  
  <p>And the error returned:</p>
  
  <pre><code>File "/Users/luisflavio/anaconda3/lib/python3.6/codecs.py", line 321, in decode
  (result, consumed) = self._buffer_decode(data, self.errors, final)
  
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
  </code></pre>
  
  <p><strong>Extra Information</strong></p>
  
  <p>I am using python 3 on MacOS High Sierra.</p>
  
  <p><strong>Solutions</strong></p>
  
  <p><em>Kyubyong's <a href="https://github.com/Kyubyong/wordvectors" rel="nofollow noreferrer">WordVectors</a></em>
  As pointed out by <a href="https://stackoverflow.com/a/50579950?noredirect=1">Aneesh Joshi</a>, the correct way to load Kyubyong's model is by calling the native load function of Word2Vec.</p>
  
  <pre><code>from gensim.models import Word2Vec
  kyu_path = '.../pre-trained_word_vectors/kyubyong_pt/pt.bin'
  model = Word2Vec.load(kyu_path)
  </code></pre>
  
  <p>Even though I am more than grateful for Aneesh Joshi solution, polyglot seems to be a better model for working with Portuguese. Any ideas about that one?</p>