Doctests fail with UnicodeDecodeError on C-extension and Python3
Question: I am having difficulty getting my testing framework to work for a C-extension
module for both Python2 and Python3. I like to run my docstrings through
`doctest` to make sure that I am not feeding my users bad information, so I
want to run `doctest` as part of my testing.

I don't believe that the source of my problem is the docstrings themselves,
but rather how the `doctest` module is trying to read my extension module. If
I run `doctest` with Python2 (on the module compiled against Python2), I get
the output that I expect:

    
    
    $ python -m doctest myext.so -v
    ...
    1 items passed all tests:
    98 tests in myext.so
    98 tests in 1 items.
    98 passed and 0 failed.
    Test passed.
    

However, when I do the same but with Python3, I get a `UnicodeDecodeError`:

    
    
    $ python3 -m doctest myext3.so -v
    Traceback (most recent call last):
    ...
      File "/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/doctest.py", line 223, in _load_testfile
        return f.read(), filename
      File "/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/codecs.py", line 301, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 0: invalid continuation byte
    

To get some more info, I ran it through `pytest` with full traceback:

    
    
    $ python3 -m pytest --doctest-glob "*.so" --full-trace
    ...
    self = <encodings.utf_8.IncrementalDecoder object at 0x102ff5110>
    input = b'\xcf\xfa\xed\xfe\x07\x00\x00\x01\x03\x00\x00\x00\x08\x00\x00\x00\r\x00\x00\x00\xd0\x05\x00\x00\x85\x00\x00\x00\x00\x...edString\x00_PyUnicode_FromString\x00_Py_BuildValue\x00__Py_FalseStruct\x00__Py_TrueStruct\x00dyld_stub_binder\x00\x00'
    final = True
    
        def decode(self, input, final=False):
            # decode input (taking the buffer into account)
            data = self.buffer + input
    >       (result, consumed) = self._buffer_decode(data, self.errors, final)
    E       UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 0: invalid continuation byte
    
    /usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/codecs.py:301: UnicodeDecodeError    
    

* * *

It looks like `doctest` is actually _reading_ the `.so` file to get the
docstrings (rather than importing the module), but Python3 doesn't know how to
decode the input. I can confirm this by replicating the byte string and
traceback by trying to read the `.so` file myself:

    
    
    $ python3
    Python 3.3.3 (default, Dec 10 2013, 20:13:18) 
    [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> open('myext3.so').read()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/codecs.py", line 301, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 0: invalid continuation byte
    >>> open('myext3.so', 'rb').read()
    b'\xcf\xfa\xed\xfe\x07\x00\x00\x01\x03\x00\x00\x00\x08\x00\x00\x00\r\x00\x00\x00\xd0\x05...'
    

* * *

Has anyone else run into this problem before? Is there a standard (or not-so-
standard) way to get `doctest` to execute tests on C extension modules on
python3?

**Update** : I should also add that I get identical results on Travis-CI ([see
here](https://travis-ci.org/SethMMorton/fastnumbers/jobs/31680811)), so it's
not specific to my local build.


Answer: I have found a workaround to this problem so I will post it, but I find it
rather unsatisfying. I am still looking for more elegant/less hacky solutions
to this.

* * *

There are **three** problems with `doctest.py` that need to be overcome to
make this work:

**1) Get doctest to consider .so files as python modules.**

If you look at the `doctest.py` source, you will notice in the test runner a
block that looks similar to this (depending on the python version you are
running):

    
    
    if filename.endswith(".py"):
        # It is a module -- insert its dir into sys.path and try to
        # import it. If it is part of a package, that possibly
        # won't work because of package imports.
        dirname, filename = os.path.split(filename)
        sys.path.insert(0, dirname)
        m = __import__(filename[:-3])
        del sys.path[0]
        failures, _ = testmod(m)
    else:
        failures, _ = testfile(filename, module_relative=False)
    

What is happening here is `doctest.py` is checking for the ".py" extension,
and if so the file is loaded as a python module, but otherwise the file is
read as if it were text (like a README.rst might be). We need to get
`doctest.py` to acknowledge that a file with ".so" extension is a python
module. To do this, simply add a check for the ".so" extension by modifying
this `if` block to read

    
    
    if filename.endswith(".py") or filename.endswith(".so"):
        ...
    

**2) Get doctest to identify the functions in the C-extension module**

`doctest.py` uses the
[inspect.isfunction](https://docs.python.org/3/library/inspect.html#inspect.isfunction)
function to determine what objects are functions when recursively searching
for docstrings within a module object. The problem with this function is that
it only identifies functions written in python, not in C (python identifies
C-extension functions as builtin). So, to identify our functions when
recursing through the module, we need to use
[inspect.isbuiltin](https://docs.python.org/3/library/inspect.html#inspect.isbuiltin)
instead.

To rectify this, we need to locate the `DocTestFinder._find` method in
`doctest.py` and change how it looks for functions. I converted

    
    
    # Recurse to functions & classes.
    if ((inspect.isfunction(val) or inspect.isclass(val)) and
        self._from_module(module, val)):
        self._find(tests, val, valname, module, source_lines,
                   globs, seen)
    

to

    
    
    # Recurse to functions & classes.
    if ((inspect.isbuiltin(val) or inspect.isclass(val)) and
        self._from_module(module, val)):
        self._find(tests, val, valname, module, source_lines,
                   globs, seen)
    

**3) Properly remove the version tag on the .so file (Python3 only).**

On Python3, C-extensions can be tagged with a version identifier (i.e.
"myext.cpython-3mu.so", please see [PEP
3149](http://legacy.python.org/dev/peps/pep-3149/)). We need to know how to
remove this when doing the initial import in the `doctest.py` test runner.

To do this, I converted the line

    
    
    m = __import__(filename[:-3])
    

to

    
    
    from sysconfig import get_config_var
    m = __import__(filename[:-3] if filename.endswith(".py") else filename.replace(get_config_var("EXT_SUFFIX"), ""))
    

This is only needed for Python3.

* * *

After making these modifications, I can get doctest to work as expected on
both Python2 and Python3. Since these modifications are rather annoying, I
have made a `patch_doctest.py` script that does this automatically and puts
the patched `doctest.py` in your current directory. You can get this file
[here](https://github.com/SethMMorton/fastnumbers/blob/develop/patch_doctest.py)
if you want to use it. You can then run the tests on the extension modules
like this

    
    
    $ python2 patch_doctest.py
    $ python2 -m doctest myext2.so
    $ rm doctest.py
    $ python3 patch_doctest.py
    $ python3 -m doctest myext3.so
    

As evidence that this works, [here are the new Travis-CI
results](https://travis-ci.org/SethMMorton/fastnumbers/builds/31772265).

