Why am I getting this error from having nonvalid unicode in source (But only when importing matplotlib)
Question: So it took me a while before I pinned the cause of this one but still am at
loss off how it is possible. I have recently switched to python3 and I got
this huge error when trying to import matplotlib:

    
    
    Traceback (most recent call last):
      File "C:/Users/y2kbugger/Desktop/test.py", line 6, in <module>
      File "C:\Anaconda2\envs\mypackage\lib\site-packages\matplotlib\__init__.py", lin
    e 124, in <module>
        from matplotlib.rcsetup import (defaultParams,
      File "C:\Anaconda2\envs\mypackage\lib\site-packages\matplotlib\rcsetup.py", line
     30, in <module>
        from matplotlib.fontconfig_pattern import parse_fontconfig_pattern
      File "C:\Anaconda2\envs\mypackage\lib\site-packages\matplotlib\fontconfig_patter
    n.py", line 25, in <module>
        from pyparsing import Literal, ZeroOrMore, \
      File "C:\Anaconda2\envs\mypackage\lib\site-packages\pyparsing.py", line 3539, in
     <module>
        _escapedPunc = Word( _bslash, r"\[]-*.$+^?()~ ", exact=2 ).setParseAction(la
    mbda s,l,t:t[0][1])
      File "C:\Anaconda2\envs\mypackage\lib\site-packages\pyparsing.py", line 966, in
    setParseAction
        self.parseAction = list(map(_trim_arity, list(fns)))
      File "C:\Anaconda2\envs\mypackage\lib\site-packages\pyparsing.py", line 813, in
    _trim_arity
        this_line = extract_stack()[-1]
      File "C:\Anaconda2\envs\mypackage\lib\site-packages\pyparsing.py", line 797, in
    extract_stack
        frame_summary = traceback.extract_stack()[offset]
      File "C:\Anaconda2\envs\mypackage\lib\traceback.py", line 207, in extract_stack
        stack = StackSummary.extract(walk_stack(f), limit=limit)
      File "C:\Anaconda2\envs\mypackage\lib\traceback.py", line 358, in extract
        f.line
      File "C:\Anaconda2\envs\mypackage\lib\traceback.py", line 282, in line
        self._line = linecache.getline(self.filename, self.lineno).strip()
      File "C:\Anaconda2\envs\mypackage\lib\linecache.py", line 16, in getline
        lines = getlines(filename, module_globals)
      File "C:\Anaconda2\envs\mypackage\lib\linecache.py", line 47, in getlines
        return updatecache(filename, module_globals)
      File "C:\Anaconda2\envs\mypackage\lib\linecache.py", line 137, in updatecache
        lines = fp.readlines()
      File "C:\Anaconda2\envs\mypackage\lib\codecs.py", line 321, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 308: invali
    d start byte
    

Commenting out the `import matplotlib as mpl` causes the error not to occur.
This led me astray trying different combinations of matplot, numpy, etc. The
part that confuses me is that if I delete the comments (that I pasted from the
web) the error is actually fixed. My editor is vim. I guess utf-8 was not the
encoding that vim used to write the file.

Minimal error producing example:

    
    
    # -*- coding: utf-8 -*-
    import matplotlib as mpl
    # Bad character pasted into vim from chrome: –
    

To fix just remove the "EN DASH" (or the entire line 3) and the matplotlib
imports correctly.

So why does the invalid(?) unicode in the comments cause an error only when
trying to `import matplotlib`(and before it even reaches the comment in
question)

    
    
    python==3.5.2
    
    colorama==0.3.7
    comtypes==1.1.2
    cycler==0.10.0
    matplotlib==1.5.1
    numpy==1.11.1
    pandas==0.18.1
    py==1.4.31
    pyparsing==2.1.4
    pytest==2.9.2
    python-dateutil==2.5.3
    pytz==2016.6.1
    pywin32==220
    scikit-learn==0.17.1
    scipy==0.18.0
    six==1.10.0
    


Answer: The problem is in [pyparsing](http://pythonhosted.org/pyparsing/):

> The pyparsing module is an alternative approach to creating and executing
> simple grammars, vs. the traditional lex/yacc approach, or the use of
> regular expressions. With pyparsing, you don't need to learn a new syntax
> for defining grammars or matching expressions - the parsing module provides
> a library of classes that you use to construct the grammar directly in
> Python.

In order to "construct the grammar directly in Python", pyparsing needs to
read the source file (in this case a matplotlib source file) where the grammar
is defined. In what would usually just be a bit of harmless extra work,
pyparsing is reading not just the matplotlib source file but everything in the
stack at the point the grammar is defined, all the way down to the source file
where you have your `import matplotlib`. When it reaches your source file it
chokes, because your file indeed is not in UTF-8; 0x96 is the Windows-1252
(and/or Latin-1) encoding for the en dash. This issue (reading too much of the
stack) has [already been fixed](https://sourceforge.net/p/pyparsing/code/415/)
by [the author of pyparsing](http://stackoverflow.com/users/165216/paul-
mcguire) so the fix should be in the next [release of
pyparsing](https://pypi.python.org/pypi/pyparsing) (probably 2.1.8).

By the way, matplotlib is defining a pyparsing grammar in order to be able to
read fontconfig files, which are a way of configuring fonts used mainly on
Linux. So on Windows pyparsing is probably not even required to use
matplotlib!

