Skipping over failed imports until they are needed (if ever) We have a program with many dependencies. Because the user may not want to use all of them, the program should skip over failed imports and only raise an error if absolutely necessary. e.g. if the user tries to call that class.

I have implemented a way of doing this but I'm not sure it is as good as it could be, it certainly seems somewhat inelegant. I presume there are also some overlooked bugs.

The important file structure is:

Program:
  - engines
    - __init__.py
    - base_engine.py
    - ...
    - psi4.py
    - rdkit.py
  - run.py


The base_engine.py contains the base class Engines which all the other engines (PSI4 in psi4.py, RDKit in rdkit.py ... ) inherit from, as well as a Default class. 

If any class fails to import, Default is imported instead, but with the name of the class that failed to import (see the __init__.py file). 

base_engine.py:

class Engines:
    """
    Engines base class containing core information that all other engines (PSI4, etc) will have.
    """

    def __init__(self, molecule):

        self.molecule = molecule


class Default:
    """
    If there is an import error, this class replaces the class which failed to be imported.
    Then, only if initialised, an import error will be raised notifying the user of a failed call.
    """

    def __init__(self, *args, **kwargs):
        # self.name is set when the failed-to-import class is set to Default.
        raise ImportError(
            f'The class {self.name} you tried to call is not importable; '
            f'this is likely due to it not being installed.')


__init__.py:

try:
    from .psi4 import PSI4
except ImportError:
    from .base_engine import Default as PSI4
    setattr(PSI4, 'name', 'PSI4')

try:
    from .rdkit import RDKit
except ImportError:
    from .base_engine import Default as RDKit
    setattr(RDKit, 'name', 'RDKit')


psi4.py:

from Program.engines.base_engine import Engines

# Example import that would fail
import abcdefg


class PSI4(Engines):
    """
    Writes and executes input files for psi4.
    """

    def __init__(self, molecule):

        super().__init__(molecule)

    def generate_input(self):

        ...


run.py:

from Program.engines import PSI4

PSI4('example_molecule').generate_input()


So now when classes are imported at the top of the run.py file, there is no problem even if there's an import error; if there's a failed import with PSI4 because abcdefg cannot be imported, Default is simply imported as PSI4 and given the name PSI4 via the setattr().

Then, only if that Default class is called the ImportError is raised and the user can see that the issue was with PSI4.

This seems to work quite well, even when there are multiple failed imports. We can also add extended error messages for each different failed import. Is this the best way of doing something like this, though? It gets quite messy since we have so many files in our engines package.

Please let me know if some relevant code has been omitted and I can add it back. Thanks for any help.



EDIT for @jpmc26:

I appreciate the time spent on your post but it's not practical for us. The program still fails fast (when necessary) because the imports are first initialised when configs are set. We have ~12 (large) stages of execution, each with multiple options which are handled by these configs. We handle this by reading the configs and terminal commands and calling whatever the option is e.g. EngineForStageFive = PSI4 is set from the configs, then we just call EngineForStageFive(args).do_something() where do_something() exists for all of the stage five classes available. All of this is nicely handled by our terminal command and config file interpreters.

The PSI4 class for example is called many times and repeatedly calling for its import with some additional logic is not what we want in our run file. We'd end up repeating a lot of code unnecessarily. e.g. every stage would need a long if/elif chain or dictionary to determine how it would be used.
There is one issue with your approach which is, in the case of multiple import failure, only the last one can be properly reported:

&gt;&gt;&gt; try:
...   from .foo import Foo
... except ImportError:
...   from .base_engine import Default as Foo
...   setattr(Foo, 'name', 'Foo')
... 
&gt;&gt;&gt; try:
...   from .bar import Bar
... except ImportError:
...   from .base_engine import Default as Bar
...   setattr(Bar, 'name', 'Bar')
... 
&gt;&gt;&gt; Bar()
Traceback (most recent call last):
…
ImportError: The class Bar you tried to call is not importable; this is likely due to it not being installed.
&gt;&gt;&gt; Foo()
Traceback (most recent call last):
…
ImportError: The class Bar you tried to call is not importable; this is likely due to it not being installed.


You instead want to generate a new class with the proper message each time.

Something along the line should do:

def mock_missing(name):
    def init(self, *args, **kwargs):
        raise ImportError(
            f'The class {name} you tried to call is not importable; '
            f'this is likely due to it not being installed.')
    return type(name, (), {'__init__': init})


Usage being:

try:
    from .foo import Foo
except ImportError:
    Foo = mock_missing('Foo')