packaging with numpy and test suite
Question: # Introduction

Disclaimer: I'm very new to python packaging with distutils. So far I've just
stashed everything into modules, and packages manually and developed on top of
that. I never wrote a `setup.py` file before.

I have a Fortran module that I want to use in my python code with numpy. I
figured the best way to do that would be f2py, since it is included in numpy.
To automate the build process I want to use distutils and the corresponding
numpy enhancement, which includes convenience functions for f2py wrappers.

I do not understand how I should organize my files, and how to include my test
suite.

What I want is the possibility to use `./setup.py` for building, installing,
and testing, and developing.

My directory structure looks as follows:

    
    
    volterra
    ├── setup.py
    └── volterra
        ├── __init__.py
        ├── integral.f90
        ├── test
        │   ├── __init__.py
        │   └── test_volterra.py
        └── volterra.f90
    

And the `setup.py` file contains this:

    
    
    def configuration(parent_package='', top_path=None):
        from numpy.distutils.misc_util import Configuration
        config = Configuration('volterra', parent_package, top_path)
        config.add_extension('_volterra',
                             sources=['volterra/integral.f90', 'volterra/volterra.f90'])
        return config
    
    
    if __name__ == '__main__':
        from numpy.distutils.core import setup
        setup(**configuration(top_path='').todict())
    

After running `./setup.py build` I get.

    
    
    build/lib.linux-x86_64-2.7/
    └── volterra
        └── _volterra.so
    

Which includes neither the `__init__.py` file, nor the tests.

# Questions

  * Is it really necessary to add the path to every single source file of the extension? (I.e. `volterra/integral.f90`) Can't I give a parameter which says, look for stuff in `volterra/`? The `top_path`, and `package_dir` parameters didn't do the trick.
  * Currently, the `__init__.py` file is not included in the build. Why is that?
  * How can I run my tests in this setup?
  * What's the best workflow for doing development in such an environment? I don't want to _install_ my package for every single change I do. How do you do development in the source directory when you need to compile some extension modules?


Answer: Here is setup.py from a project I made. I have found figuring out setup.py /
packaging to be frustrating with no solid answers and definitely not pythonic
in the sense of having one and only one obvious way to do something. Hopefully
this will help a little.

The points you may find useful are:

  * `find_packages` which removes the drudgery of including lots of files or messing around with generating manifest.
  * `package_data` which allows you to easily specify non .py files to be included
  * `install_requires` / `tests_require`

You'll need to find the source for distribute_setup.py if you don't have it
already.

>   * Is it really necessary to add the path to every single source file of
> the extension? (I.e. volterra/integral.f90) Can't I give a parameter which
> says, look for stuff in volterra/? The top_path, and package_dir parameters
> didn't do the trick.
>   * Currently, the **init**.py file is not included in the build. Why is
> that?
>

Hopefully `find_packages()` will solve both of those. I don't have much
experience packaging but I haven't had to go back to manual inclusion yet.

>   * How can I run my tests in this setup?
>

I think this is probably a different question with many answers depending on
how you are doing tests. Maybe you can ask it separately?

As a side note, I am under the impression that the standard is to put your
tests directory at the top level. I.e. `volterra/volterra` and
`volterra/tests`.

>   * What's the best workflow for doing development in such an environment? I
> don't want to install my package for every single change I do. How do you do
> development in the source directory when you need to compile some extension
> modules?
>

This might be worth another question as well. I don't see why you would need
to install your package for every single change. If you are uploading the
package, just don't install it on your dev system (except to test
installation) and work directly from your development copy. Maybe I'm missing
something though since I don't work with compiled extensions.

* * *

Here is the example

    
    
    try:
        from setuptools import setup, find_packages
    except ImportError:
        from distribute_setup import use_setuptools
        use_setuptools()
        from setuptools import setup, find_packages
    
    
    setup(
        # ... other stuff
        py_modules=['distribute_setup'],
        packages=find_packages(),
        package_data={'': ['*.png']},  # for me to include anything with png
        install_requires=['numpy', 'treenode', 'investigators'],
        tests_require=['mock', 'numpy', 'treenode', 'investigators'],
    )
    

