text: At 06:31 PM 11/11/2005 +0000, Richard Cooper wrote: >From: Phillip J. Eby [mailto:pje at telecommunity.com] > > What you probably want is a layout like this: > > > > app/ > > setup.py > > My/ > > Project/ > > app > > Plugin1/ > > setup.py > > My/ > > Project/ > > plugin1/ > > > > etc. > >My knee-jerk reaction is 'Ewww!' Is this what PEAK (or ant other large >setuptools based project) does? Yes, actually. More precisely, it's what PEAK *will* do, since I haven't actually started breaking out individual projects yet. >How do you handle My.Project.utils (for >example) and the half dozen or so other packages at different levels of >the hierarchy which are used in multiple "products"? You make them individual projects. For example, I have a module, peak.util.imports, that will be spun off as its own project. Its layout will be: setup.py peak/ __init__.py util/ __init__.py imports.py The __init__.py files will be empty, of course, since as namespace packages they don't need any contents. Any project (including PEAK itself) that needs to use this module will simply depend on this project as a whole. My original plan for setuptools was to use an approach that could section out parts of a large distribution like this. However, I quickly realized that this was counterproductive to the Python "ecosystem"; it would be more beneficial to have lots of small projects that can be used by others (without buying into the entire PEAK system), than it would be for me to keep using the same "all-in-one" source tree layout that I'm currently using for PEAK. I suspect that you may like Zope's "zpkg" system better for your purposes, since it is designed to extract subsets of a larger distribution and automatically generate setup scripts for them. I don't know if you can use it to create eggs, but it might be possible to modify it to do so. >My needs are a bit specialised, this is a commercial app so source >distribution, easy_install, etc will never be on the cards. Distribution >is strictly "frozen" products (py2exe and eggs) only. easy_install is useful as an in-house tool for automatically building your projects. For example, you could create a setup.py that lists all your plugin project names as requirements, and a setup.cfg that lists their subversion URLs, and you could run easy_install against that directory to check out and build all the latest plugin eggs, dropping them in a distribution directory. Your users don't ever have to see easy_install, for it to be useful in your project. Of course, you may not be using Subversion, etc., and this may be moot, but I'm not psychic and don't know anything about what you're doing besides what you tell me. :) > > If you are working on these projects at the same > > time, you need only run "setup.py develop" in each one to set > > up a sys.path structure that merges all the packages into one. > >We would probably end up with 20 or so "projects" if we did things that >way as opposed to the single code tree we have now. Given that, >"setup.py develop"*20 seems like more work than our current approach >which is "add the root of the code tree to PYTHONPATH" Does that mean you have 19 plugins? Perhaps I'm misunderstanding something about your intentions. I assume that your projects would be one for the app, and one for each plugin, so that you can create a py2exe for the app and an egg for each plugin. If that's the case, then you would only have 20 projects if you had 19 plugins. Also, if the plugins don't depend on each other, then to work on any one plugin you only run "develop" twice: once for the app and once for the plugin. Also, this may not be especially relevant to your project at the moment, but "develop" also makes sure that your environment is synchronized with any external requirements. For example, in a multi-developer scenario where developer Aaron changes a dependency on the FooBar package from 1.1 to 1.2, then when developer Bob updates his source checkout and runs "develop", it will find/fetch/build/install FooBar 1.2, thereby helping to reduce the pain of adding or removing dependencies. You could probably quite rightly say that this doesn't affect your project because you don't have external dependencies, but a significant part of *why* few projects have external dependencies is because they're a costly pain -- if you're not using setuptools. I point this out because setuptools is intended primarily to enable a new way of developing projects, and enabling people to do things that were previously unthinkable. If you aren't trying to do what was impossible before, then certainly setuptools will not appear to help your current situation much. :) > > Note, by the way, that you don't need namespace packages to > > have plugins for an application. Plugins can be in any > > package structure you like, as eggs' "entry points" system > > can be used by plugins to advertise the functions they > > provide, and the importing can be done automatically for you. > >Yeah I know. The entry point system is the main reason I started playing >with setuptools. However I still wanted the plugins we produce to all >live in the My.project.plugins package. Note that giving each plugin its own top-level package would completely eliminate any duplication of directory structure between plugins, if that's your concern. But note that in any case, all that you'd be duplicating are empty directories and zero-length __init__.py files. >It looks like my problem is that I'm mingling separate products in the >same code tree and setuptools/distutils doesn't like that. Fair enough. >I don't think I'm quite ready to drink the "one tree per product" >kool-aid just yet. Firstly, it seems weird to me to split up the source >like that and secondly, my colleagues would probably kill me ;-) Here's another possible layout, that will work nicely for eggs, but won't work for "setup.py develop": TheProject/ MyApp/ __init__.py setup.py *all app code here* Plugin1/ __init__.py setup.py *plugin code here* The idea here is that you have a MyApp top-level package, and a top-level package for each plugin. Each setup script is *inside* the corresponding package directory. Adding TheProject to PYTHONPATH makes everything importable, just like you have now. But each project corresponds to a top level package, and uses package_dir = {'MyApp':'.'}, or {'Plugin1':'.'} etc. so that each project directory is also a package directory. This compromise layout preserves the single-checkout+PYTHONPATH approach, and should allow you to generate eggs without any special hacks, at the cost of flattening your hierarchy a bit, and making it impossible to use setuptools' "develop" or "test" commands (which need the package directories to be somewhere *under* the directory containing setup.py). Personally, I find that deeply nested foo.bar.baz hierarchies are a Java-ism; Python doesn't need them and is usually better off without them; "flat is better than nested", and all that. But I can also understand that reorganizing your packages just to support eggs might be considered too costly. Just make sure you factor in the cost of *maintaining* a hack to do this, since the reorganization cost would be a one-time thing, but the hack would be something you'd have to maintain forever. >So what I will probably end up doing is hacking up setuptools to insert >the __init__.py files I need. Which begs the question - Is this: > >a) A dirty, dirty hack I should never speak of again OR >b) Not a bad idea and potentially useful to other people. Yes. :) It's both, really. Consider Chandler, for example, which is currently very similar to your project. Actually, it's worse, because it doesn't even have a setup script yet. In the next release cycle, we'll be moving to eggs and we will have to do some source tree splitting, and there will be some pain, and questioning, and maybe even some griping about it. *But*, we spent a lot of time during the current release cycle doing quite a lot of *flattening* of the source tree, turning deeply nested modules like "osaf.contentmodel.contacts.Contacts" into "osaf.pim.contacts", and consolidating APIs so that most code can now do: from osaf import pim aContact = pim.Contact() instead of the former: import osaf.pim.contacts.Contacts as Contacts aContact = Contacts.Contact() Flat really *is* better than nested. Much better. And as a bonus, anything that we choose to split into separate eggs will not have as much duplication. In the general case, the 'osaf' top-level package will be the only thing that gets duplicated, and since it's a namespace package, the __init__.py will be an empty file anyway. So, I would encourage you to urge those you share the hack with, to consider all of the issues I've brought out here, and use it only as a crutch to allow a project to get past a temporary issue with its layout, not as a cane to perpetually lean on. Putting on my software development manager hat, I would tell you flatly that the time to manage the directory structure duplication and use of "develop" is nothing compared to the time saved by being able to intelligently manage external dependencies. And if you don't have any external dependencies, it's only because they cost too much before. Setuptools changes the cost equation, and therefore changes what the ideal development patterns are. The reality may be that in your particular environment you don't have the authority or reputation to lead such a change, but we shouldn't let that stop other people from getting that awareness. Okay, I'll get off the soapbox now. :) Hack away, just make sure to point people to this thread and especially this message. One other idea, by the way... Have you tried adding: py_modules = ['My.__init__', 'My.App.__init__', ...] to the setup() arguments? It's just a guess on my part, but I think it might actually work, without doing any hacking on distutils or setuptools. Just a thought. Happy hacking, either way. :)