Using code from a dependency in setup.py
Question: I have a little web framework, let's call it Bread, which is used to build
applications like Jam, Marmalade, PeanutButter, and other toppings. Bread both
_builds_ and _serves_ these applications.

I'm trying to figure out how to make the applications' `setup.py`s work, given
the following requirements:

  * The apps depend upon Bread, via [setuptool's install_requires](http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies)
  * To build an application at development time, Bread reads some config and then emits assets (HTML, JS, CSS, images, etc) to the application's `output` directory. In other words, `bread devserver` reads `Jam/bread.yaml` and assembles assets in `Jam/output`, then serves the application (via Flask, but that's not otherwise pertinent).
  * In order to build a _deployable_ Jam application, I want to invoke Bread during `python setup.py install` of Jam, to build `Jam/output`. In production, Jam should not need to build anything.
  * I've defined a custom `bdist_egg` setup command where `initialize_options` imports Bread, invokes the builder, then sets `self.distribution.data_files` with the appropriate tuples, before calling the base class. (And _that_ was no fun to figure out.)
  * Right now, the `bdist_egg` is defined in Jam's `setup.py`. I want to move this and other boilerplate code into `bread.setup`, so that I can reuse it in Marmalade, PeanutButter, etc.
  * Potentially, this means that I'm now importing Bread code before Bread has been installed. This will surely arise in a clean install, such as a fresh virtualenv on a build machine.

Can this be done with Distutils / setuptools / Distribute?


Answer: I also asked this question in [Distutils-
SIG](http://mail.python.org/pipermail/distutils-sig/2013-May/020963.html). A
mention there of `setup_requires` led me to
<http://stackoverflow.com/a/12061891/6364>, which gave me the hint I needed:
create a separate `Distribution` object before calling `setup` which defines
the `setup_requires` entries.

Jam's `setup.py` now looks like:

    
    
    from setuptools import setup, dist
    
    dist.Distribution(dict(setup_requires='Bread'))
    
    from bread.setup_topping import *
    
    setup(
        name='Jam',
        version='0.2',
        long_description=open('README.md').read(),
        **topping_setup_options
    )
    
    # Remove *.egg left by bootstrapping Bread
    cleanup_bread_bootstrap()
    

* * *

EDIT: a better explanation of what's happening in Jam's `setup.py` is needed:

  * The initial `Distribution(setup_requires='Bread')` is using `easy_install` to install Bread and its dependencies in the _current directory_.
  * The call to `setup()` triggers the `bdist_egg` below, which uses Bread to build Jam's `output`. Bread is found in the current directory.
  * `setup()` later installs Jam, Bread, and all the dependencies, in the correct location.
  * The call to `cleanup_bread_bootstrap()` removes all the Eggs that were laid in the current directory by the initial `Distribution`.

* * *

And `bread/setup_topping.py` looks like:

    
    
    from setuptools.command.bdist_egg import bdist_egg as _bdist_egg
    import os, fnmatch, glob, shutil
    
    def recursive_data_files(treeroot, pattern):
        results = []
        for base, dirs, files in os.walk(treeroot):
            goodfiles = fnmatch.filter(files, pattern)
            if goodfiles:
                results.append((base, [os.path.join(base, f) for f in goodfiles]))
        return results
    
    def make_data_files(output='output'):
        return (
            [('', ['bread.yaml'])]
            + recursive_data_files(output, '*')
        )
    
    class bdist_egg(_bdist_egg):
        def initialize_options(self):
            bake_bread()    # build files to './output'
            self.distribution.data_files = make_data_files()
            _bdist_egg.initialize_options(self)
    
    topping_setup_options = dict(
        cmdclass={
            'bdist_egg': bdist_egg,
        },
        install_requires=[
            'Bread',
        ],
        zip_safe=False,
    )
    
    def cleanup_bread_bootstrap(root='.'):
        for f in glob.glob(os.path.join(os.path.abspath(root), '*.egg')):
            if os.path.isdir(f):
                shutil.rmtree(f)  # Egg directory
            else:
                os.remove(f)      # Zipped Egg
    

