text: Created on 2013-08-28 06:56 by eric.snow, last changed 2016-05-12 14:37 by eric.snow. This issue is now closed.
  This ticket will track the implementation for PEP 451 (ModuleSpec). I'll have a patch up in the next couple days.
  Here's an initial patch. Some key things left to do:
  * unit tests
  * documentation
  * implement exec_module() for the various importlib loaders.
  Once that's squared away there are further things that will be addressed in new tickets (or at least separate patches). This includes:
  * remove init_module_attrs() and module_to_load()
  * deprecations
  * clear a bunch of helper functions out of _bootstrap.py
  * use ModuleSpec with __main__
  * address impact on stdlib (pkgutil, pickle, etc.)
  Here's a patch just for the changes to the import page in the language reference. It's not perfect, but should represent the target pretty closely.
  Let's try that again with the proper diff. :)
  Thanks, that looks pretty good - so much less work for loaders to do.
  After this, we may even be able to finally fix the circular import bug and the one where submodule imports succeed but the overall package import fails.
  I've added a review
  Something else we may be able to fix (albeit probably not in 3.4): several of the issues noted in PEP 395 (since __main__.__spec__.name will give the real module name, even when __name__ is set to "__main__" or otherwise modified)
  Here's an update of the language reference patch based on Brett's comments.
  Here's an updated patch that implements the meat of the current PEP. Docs are still lacking and I could probably add a few more tests. Other key things left to do:
  * deprecations and removals
  * refactor importlib finders and loaders to use the new Finder/Loader APIs
  * refactor pythonrun.c to make use of __spec__
  * adjust other APIs to use __spec__ (pickle, etc.)
  This time against tip. :)
  I should also point out that this patch is on top of the path in issue19413.
  Here's a new patch that is mostly up to date with the PEP. I still need to work on Docs and add more tests. There are a few failing tests (due to the recent reload patch) that I need to fix when I get a minute.
  This patch also implements find_spec() on FileFinder and PathFinder. Left to do:
  * deprecations and removals
  * refactor importlib loaders to use the new Finder/Loader APIs
  * refactor pythonrun.c to make use of __spec__
  * adjust other APIs to use __spec__ (pickle, etc.)
  I've created a server-side clone for the implementation:
  Okay, I've updated the pep451 branch in the clone to include as much of the implementation as I've completed, which is the bulk of the functional changes. It's enough to pass the test suite.
  Here's what I'd like to get done before the feature freeze (in this priority order):
  1. wrap up the current functional changes in the clone;
  2. change module.__initializing__ to module.__spec__._initializing;
  3. refactor importlib loaders to use the new Finder/Loader APIs;
  4. refactor pythonrun.c to make use of __spec__;
  5. implement the deprecations and removals;
  6. adjust other APIs to use __spec__ (pickle, etc.);
  7. add rudimentary doc additions for the new APIs.
  Other things that can (but don't have to) wait until after the beta release:
  * finish doc changes;
  * fill in gaps in test coverage (there shouldn't be much due to Brett's mother of all test suites for importlib)
  I haven't had a chance yet to make any changes to Doc/reference/import.rst in response to Brett's review, but I did make the Doc/library/importlib.rst changes he recommended.
  I don't quite know what you mean by "current functional changes in the clone". Is this changes to import to use exec_module() and such? Just a guess since step 3 suggests there are no loaders implementing the new API currently.
  As for the tests, I'm hoping that simply refactoring some of them will allow reusing a bunch of the finder & loader tests instead of having to do it from scratch.
  I.
  Sorry that wasn't more clear. I committed the changes from the modulespec-primary-changes.diff patch to the pep451 branch in the server-side clone. Those changes are:
  Step 1
  ------
  1. added ModuleSpec class
  2. added _SpecMethods wrapper class
  3. added ModuleSpec factory functions
  4. added tests for ModuleSpec, _SpecMethods, and the factories
  5. exposed ModuleSpec in importlib.machinery
  6. exposed ModuleSpec factories in importlib.util
  7. added basic docs for ModuleSpec and the factory functions
  Step 2
  ------
  1. changed _find_module() to _find_spec()
  2. changed _find_and_load_unlocked() to use _find_spec() and _SpecMethods
  3. changed _setup() to use specs
  4. changed pydoc to recognize __spec__
  Step 3
  ------
  1. updated the import reference doc
  2. changed importlib.reload() to use specs
  3. added importlib.find_spec()
  4. changed importlib.find_loader() to wrap find_spec()
  5. updated importlib.abc to reflect the new APIs
  6. changed pkgutil to use specs
  7. changed imp to use specs
  8. fixed a bunch of broken tests to use spec
  Step 3
  ------
  1. implemented find_spec() on PathFinder
  2. implemented find_spec() on FileFinder
  3. re-implemented FileFinder.find_loader() to wrap find_spec()
  4. re-implemented PathFinder.find_module() to wrap find_spec()
  5. changed _NamespacePath to use specs
  Step 5
  ------
  1. added _module_repr function
  2. changed ModuleType.__repr__ to wrap _module_repr
  Others
  ------
  * removed _NamespaceLoader
  * added comments indicating deprecations and removals
  At this point, the test suite passes and the fundamental changes of the PEP are implemented (on the server-side clone). Here's what's left to do before the feature freeze:
  1.
  2. change module.__initializing__ to module.__spec__._initializing
  3. refactor importlib loaders to use the new Finder/Loader APIs
  4. refactor pythonrun.c to make use of specs
  5. check pkgutil for any missed changes
  6. implement the deprecations and removals
  7. adjust other APIs to use __spec__ (pickle, runpy, inspect, pydoc, others?)
  8. evaluate any impact on setuptools
  Other things that can (but don't have to) wait until after the beta release:
  * finish doc changes
  * fill in any gaps in test coverage
  * ensure new docstrings exist and are correct
  * ensure existing docstrings are still correct
  "1." was intentionally left blank <wink>
  Regarding tests, a bunch of importlib (and other?) tests make direct calls to find_module(), find_loader(), or load_module(). These are still working, as expected. However, we should probably either replace them or supplement them with equivalent tests that make use of specs directly.
  Yeah, don't replace any tests, add new ones for the new APIs. Given the
  needs of 2/3 compatible loader implementations, the deprecations referred
  to in the PEP should also be documentation-only for 3.4..
  > Yeah,....
  Although, a boolean query method would bring back the problem of the loader
  not reporting any details on *why* it can't load into a particular target
  module. So it may be better to have an optional loader
  "check_existing_target" API that throws a suitable exception if the target
  is unacceptable. Loaders that don't support reloading at all would just
  always raise an exception, while those that don't care would just not
  implement the method.
  That would address my concern about the lack of useful error information in
  Eric's original boolean check idea, without bothering finders with loader
  related details as the accepted PEP does (I still like the idea of passing
  a target to importlib.find_spec - I just no longer think we should be
  passing that down to the finders themselves).
  Another thing we need to check we have a test for: ensuring reloading a
  namespace module picks up new directories added since it was first loaded.
  This would all be so much easier if reloading wasn't supported in the first
  place :)
  On 12 Nov 2013 09:36, "Eric Snow" <report@bugs.python.org> wrote:
  >
  >.
  Yes, that's an option, too, and probably a good one. To go down that path,
  we would drop the various "target" parameters and say loaders that need to
  check for the reloading case should continue to provide load_module without
  exec_module (at least for 3.4). runpy would use the rule that it supports
  anything that exposes exec_module without create_module.
  > (I still like the idea of passing
  > a target to importlib.find_spec - I just no longer think we should be
  > passing that down to the finders themselves).
  Passing the target to the finders isn't just for the sake of any implicit "check_target" test, though that was the original motivator. It also allows the finder to decide between multiple loaders based on other criteria related to the target (but not necessarily the loader). I think it was a good addition to the API regardless.
  > Another thing we need to check we have a test for: ensuring reloading a
  > namespace module picks up new directories added since it was first loaded.
  Agreed. Furthermore, such a test is worthwhile outside the context of PEP 451. I'm tempted to say we're already covered with existing tests, but reload is goofy enough that an explicit test is worth it.
  > This would all be so much easier if reloading wasn't supported in the first
  > place :)
  So very true. :)
  And coming full circle: there's no *harm* in letting finders reject loading
  into a target module, and that's orthogonal to having loaders reject it.
  It's just that loaders that want to do that will currently still need to
  implement load_module.
  That means the only thing we need to postpone is the load_module
  deprecation, since it still covers at least one advanced use case the new
  API doesn't handle.
  Sounds good. It will be worth adding a note to the load_module() docs indicating the limited cases where it is still appropriate, and encouraging the use of exec_module() instead.
  I'm running the pep-451 clone against 2 buildbots for now:
  *
  *
  The ubuntu one is happy with the feature clone, but the windows buildbot keeps finding problems. At first it was issue17116 all over again. I've now set __spec__ to None by default (a la issue17115). Then it fussed about test_everyone_has___loader__. I'm sure something else will turn up.
  Any thoughts on why import-related tests are responding differently depending on platform? FWIW, I think the Windows failures should have also failed on the Ubuntu buildbot.
  I'm also going to run on a Windows 7 and OS X to see where things stand.
  The OS X buildbot I used [1] did not exhibit any of the failures that Windows server 2003 showed.
  [1]
  The Windows 7 buildbot I tried [1] shows the exact same failures that the server 2003 does.
  [1]
  Naturally the other Windows buildbot I tried did not fail any import-related tests. So much for consistency. :(
  The three Windows machines look like they have very different system
  versions of Python (ActiveState, cygwin, vanilla CPython). Any chance we're
  accidentally invoking the system Python instead of the built one somewhere?
  I.
  > The three Windows machines look like they have very different system
  > versions of Python (ActiveState, cygwin, vanilla CPython). Any chance we're
  > accidentally invoking the system Python instead of the built one somewhere?
  Good point. I'll check that out. That said, on my local clone (Ubuntu 12.10) I'm also getting no failures. It could also be related to the speed of the buildbot, but I doubt it. Anyway, I'll take a closer look at the stdout artifacts.
  > I think worrying about more API expansions to handle edge case/advanced
  > reload scenarios is not worth it (or at least not in Python 3.4).
  Agreed.
  Regarding the to-do list, I'm going to add a TODO file to the root of the feature clone (in the pep451 branch) so we don't have to keep updating this ticket. :)
  FYI, as of cc7ebd952777 the bulk of the implementation in features/pep-451 is working.
  There is a failing test on the Windows Server 2003 buildbot (which I'm trying to track down). I haven't been able to reproduce the failure on other buildbots, on my local Ubuntu 12.10 workstation, or on my local Windows laptop. See.
  Cool. Roundup linked the revision hash to the feature clone. :)
  Just a heads up - since the two of you appear to have this well in hand,
  don't wait for a review from me before committing it. I have plenty of
  other things to look at before the beta deadline :)
  Brett: looks like something frozen-related broke due to 6d1656ab2c85a527c.
  test_frozen in test_frozen.py is failing now because frozen modules no longer have a __cached__ attribute (which was previously set to None). Previously it was set in PyImport_ExecCodeModuleObject() in import.c. I'm guessing that simply setting __cached__ (and __file__?) in FrozenImporter.exec_module() should restore the previous behavior. Doing so makes the test pass at least.
  That.
  I?
  Semantic-affecting changes first, then fluffy bits in terms of priority. Basically go based on what is most critical to be in 3.4.
  And yes, basically can you comment out find_module/find_loader/load_module and execute the test suite (sans import-related tests) without import-related failures.
  No longer blocking b1, now just blocking rc1.
  Your commit doesn't compile on Windows.
  Category None
  Changed by Eric Snow <ericsnowcurrently@gmail.com>
  Changed at Fri 22 Nov 2013 16:17:09
  Branch default
  Revision 07229c6104b16d0ab7cc63f3306157d3d2819fed
  Implement PEP 451 (ModuleSpec).
  There is a commented-out line in Lib/importlib/_bootstrap.py:
  +# if hasattr(loader, 'get_data'):
  + if hasattr(loader, 'get_filename'):
  Removed the commented-out line in c8a84eed9155
  Failing buildbot:
  test.test_module.ModuleTests.test_module_repr_source
  test.test_pkgutil.ExtendPathTests.test_iter_importers (passed when retried)
  Same failures on
  The test_module failure can be triggered by running test_importlib test_module. For some reason unittest's __spec__ gets set to one for the BuiltinImporter (and nothing else).
  New changeset 98bab4a03120 by Brett Cannon in branch 'default':
  Issue #18864: Don't try and use unittest as a testing module for
  I just realized that there is no way to make ModuleSpec.has_location return True if the spec is created by simply instantiating ModuleSpec. As it stands now you **must** go through importlib.util.spec_from_file_location(), which makes it not a helper but a **required** API to use. I think this is the only thing that cannot be stated directly through ModuleSpec instantiation or public attribute assignment.
  This means either (a) the docs on ModuleSpec need to be **very** clear that you must use spec_from_file_location() to make spec.has_location be true, (b) add another keyword-only argument to ModuleSpec.__init__(), or (c) make has_location settable (e.g. be able to only set it to True/False or alternatively assigning it sets both has_location to True and origin to the assigned value).
  Good catch. My preference would be for (c) simply add a setter for the property that sets the underlying private variable to True or False.
  (c) it is then! I would just take the time to call bool() on the arg to coalesce it into a True/False thing (which maybe makes sense in __init__() as well).
  Actually, ignore the __init__() part of my last comment since it's set to False directly.
  Here's a patch for adding a setter for ModuleSpec.has_location.
  LGTM
  New changeset e961a166dc70 by Eric Snow in branch 'default':
  Issue #18864: Add a setter for ModuleSpec.has_location.
  About the only thing left for this ticket is to finish up writing a few tests and round out some documentation.
  There are a few lingering to-do comments that will need to be addressed as part of the remaining work:
  Lib/test/test_importlib/extension/test_case_sensitivity.py:# XXX find_spec tests
  Lib/test/test_importlib/extension/test_finder.py:# XXX find_spec tests
  Lib/test/test_importlib/source/test_file_loader.py: mod = loader.load_module('_temp') # XXX
  Lib/test/test_importlib/source/test_file_loader.py: # XXX Change to use exec_module().
  Lib/test/test_importlib/test_windows.py: # XXX Need a test that finds the spec via the registry.
  Failure with 3.4.0c1:
  [170/394/5] test_importlib
  test test_importlib failed -- Traceback (most recent call last):
  File "C:\Programs\Python34.32\lib\unittest\case.py", line 57, in testPartExecutor
  yield
  File "C:\Programs\Python34.32\lib\unittest\case.py", line 574, in run
  testMethod()
  File "C:\Programs\Python34.32\lib\unittest\loader.py", line 32, in testFailure
  raise exception
  ImportError: Failed to import test module: test.test_importlib.source.test_abc_loader
  Traceback (most recent call last):
  File "C:\Programs\Python34.32\lib\unittest\loader.py", line 312, in _find_tests
  module = self._get_module_from_name(name)
  File "C:\Programs\Python34.32\lib\unittest\loader.py", line 290, in _get_module_from_name
  __import__(name)
  File "C:\Programs\Python34.32\lib\test\test_importlib\source\test_abc_loader.py", line 73, in <module>
  class PyLoaderMock(abc.PyLoader):
  AttributeError: 'module' object has no attribute 'PyLoader'
  On the other hand, the test passes on a fresh debug build.
  Terry has admitted in other bugs he filed that he didn't use a fresh install location, so the test failure is most likely a red herring.
  I've gone ahead and moved all remaining work into separate issues. Once the 4 remaining sub-issues are resolved, this one can be closed. I've *finally* got a couple days free so I want to close this out. :)
  Given that PEP 451 is listed in the "Finished PEPs (done, implemented in code repository)" section of the PEP index can we close this and associated issues out?
  Dependencies 19711 and 21099 are still open.
  I've asked for patch reviewss on the three dependencies outstanding, #19711, #19714 and #21099.
  I've asked for updates on the two remaining dependencies #19711 and #21099.
  The last dependencies have now been closed!