Good way to collect programmatically generated test suites in nose or pytest
Question: Say I've got a test suite like this:

    
    
    class SafeTests(unittest.TestCase):
        # snip 20 test functions
    
    class BombTests(unittest.TestCase):
        # snip 10 different test cases
    

I am currently doing the following:

    
    
    suite = unittest.TestSuite()
    loader = unittest.TestLoader()
    safetests = loader.loadTestsFromTestCase(SafeTests)
    suite.addTests(safetests)
    
    if TARGET != 'prod':
        unsafetests = loader.loadTestsFromTestCase(BombTests)
        suite.addTests(unsafetests)
    
    
    unittest.TextTestRunner().run(suite)
    

I have major problem, and one interesting point

  * I would like to be using nose or py.test (doestn't really matter which)
  * I have a large number of different applications that are exposing these test suites via entry points.

I would like to be able to aggregate these custom tests across all installed
applications so I can't _just_ use a clever naming convention. I don't
_particularly_ care about these being exposed through entry points, but I
**do** care about being able to run tests across applications in site-
packages. (Without just importing... every module.)

I do _not_ care about maintaining the current dependency on
`unittest.TestCase`, trashing that dependency is practically a goal.

* * *

**EDIT** This is to confirm that @Oleksiy's point about passing args to
`nose.run` does in fact work with some caveats.

Things that _do not_ work:

  * passing all the files that one wants to execute (which, _weird_)
  * passing all the _modules_ that one wants to execute. (This either executes nothing, the wrong thing, or too many things. Interesting case of 0, 1 or many, perhaps?)
  * Passing in the modules _before_ the directories: the directories have to come first, or else you will get duplicate tests.

This fragility is absurd, if you've got ideas for improving it I welcome
comments, or I set up [a github repo with my experiments trying to get this to
work](https://github.com/quodlibetor/test-loading-experiment).

All that aside, The following works, including picking up multiple projects
installed into site-packages:

    
    
    #!python
    import importlib, os, sys
    import nose
    
    def runtests():
        modnames = []
        dirs = set()
        for modname in sys.argv[1:]:
            modnames.append(modname)
    
            mod = importlib.import_module(modname)
            fname = mod.__file__
            dirs.add(os.path.dirname(fname))
    
        modnames = list(dirs) + modnames
    
        nose.run(argv=modnames)
    
    if __name__ == '__main__':
        runtests()
    

which, if saved into a `runtests.py` file, does the right thing when run as:

    
    
    runtests.py project.tests otherproject.tests
    


Answer: For nose you can have both tests in place and select which one to run using
[attribute](http://nose.readthedocs.org/en/latest/plugins/attrib.html "attrib
plugin") plugin, which is great for selecting which tests to run. I would keep
both tests and assign attributes to them:

    
    
    from nose.plugins.attrib import attr
    
    @attr("safe")
    class SafeTests(unittest.TestCase):
        # snip 20 test functions
    
    class BombTests(unittest.TestCase):
        # snip 10 different test cases
    

For you production code I would just call nose with `nosetests -a safe`, or
setting `NOSE_ATTR=safe` in your os production test environment, or call run
method on nose object to run it natively in python with `-a` command line
options based on your `TARGET`:

    
    
    import sys
    import nose
    
    if __name__ == '__main__':
        module_name = sys.modules[__name__].__file__
        argv = [sys.argv[0], module_name]
        if TARGET == 'prod':
            argv.append('-a slow')
    
        result = nose.run(argv=argv)
    

Finally, if for some reason your tests are not discovered you can explicitly
mark them as test with `@istest` attribute (`from nose.tools import istest`)

