AppEngine application using Django fails to load
Question: Django is constantly causing our application to crash. After deployment the
application is running fine, but once the initial instance is
restarted/shutdown it often fails to start with an error similar to the
following:

    
    
    Traceback (most recent call last):  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
        result = handler(dict(self._environ), self._StartResponse)
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 236, in **call**
        self.load_middleware()
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 53, in load_middleware
        raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
    ImproperlyConfigured: Error importing middleware myfolder.middleware: "No module named myfolder.middleware"
    

`

Our file structure is similar to this:

    
    
    |- app.yaml
    |- _ _ init _ _.py
    |- settings.py
    |- myfolder |
    |           |- _ _ init _ _.py
    |           |- middleware.py
    |           |- ...
    |-...
    |
    

`

Our app.yaml:

    
    
    application: XXXXX
    module: app
    version: master
    runtime: python27
    api_version: 1
    threadsafe: true
    
    
    
    
    
    handlers:
    - url: /api/(login|logout|passwd|master._|banners._)
      script: app.handler
      secure: always
    ...
    
    
    
    
    
    builtins:
    - django_wsgi: on
    
    
    
    
    
    libraries:
    - name: django
      version: 1.5
    
    
    
    
    
    env_variables:
        DJANGO_SETTINGS_MODULE: 'settings'
    

`

We have 2 modules in our application and they both exhibit this behaviour
(they have similar configurations). Sometimes the modules will stay up for a
whole day before crashing again. After they fail to load, all subsequent
requests fail with he same error. Deploying one more time always solves the
problem temporarily.

We are using plain django with CloudSql. The problem is not reproducible in
the development server. After deployment everything in both modules works
fine. All middleware, ndb, memcache, cloudsql, taskqueue, etc, including all
the modules inside the "myfolder" and every other library xcopied.

The following attempts at solving this problem haven't worked:

  * We have tried using the appengine_config.py to force django to reload the settings with from django.conf import settings\nsettings._target = None\n
  * Originally we had shared settings inside "myfolder" and were importing them with "from myfolder.shared_settings import *" inside the root settings.py but django could not load the module myfolder.shared_settings either (similar problem)
  * using a custom mysettings.py and defining the DJANGO_SETTINGS_MODULE in the app.yaml or in python

The system is not live yet but will be soon and we are running out of options.

Other traces of similarly failing configurations:

    
    
    Traceback (most recent call last):
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
        result = handler(dict(self._environ), self._StartResponse)
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 236, in __call__
        self.load_middleware()
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 45, in load_middleware
        for middleware_path in settings.MIDDLEWARE_CLASSES:
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 53, in __getattr__
        self._setup(name)
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 48, in _setup
        self._wrapped = Settings(settings_module)
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 134, in __init__
        raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
    ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named myfolder.settings
    
    
    
    Traceback (most recent call last):
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 239, in Handle
        handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 353, in __getattr__
        self._update_configs()
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 289, in _update_configs
        self._registry.initialize()
      File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 164, in initialize
        import_func(self._modname)
      File "/base/data/home/apps/s~blue-myapp/app:master.375531077560785947/appengine_config.py", line 17, in 
        settings._target = None
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/utils/functional.py", line 227, in __setattr__
        self._setup()
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 48, in _setup
        self._wrapped = Settings(settings_module)
      File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 134, in __init__
        raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
    ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named myfolder.settings
    

This is our current appengine_config.py:

    
    
    import sys
    import logging
    
    logging.debug(",\n".join(sys.path))
    
    # Since Google App Engine's webapp framework uses Django templates, Django will half-initialize when webapp is loaded.
    # This causes the initialization of the rest of Django's setting to be skipped. If you are getting this error, you need
    # to explicitly force Django to reload your settings:
    
    from django.conf import settings
    settings._target = None
    

Logging sys.path from appengine_config.py does not change between a successful
instance start and a failed instance start (apart from the XXXXXXXXXXX bit of
course):

    
    
    /base/data/home/apps/s~blue-persomi/app:master.3759720XXXXXXXXXXX,
    /base/data/home/runtimes/python27/python27_dist/lib/python27.zip,
    /base/data/home/runtimes/python27/python27_dist/lib/python2.7,
    /base/data/home/runtimes/python27/python27_dist/lib/python2.7/plat-linux2,
    /base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-tk,
    /base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-old,
    /base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-dynload,
    /base/data/home/runtimes/python27/python27_dist/lib/python2.7/site-packages,
    /base/data/home/runtimes/python27/python27_lib/versions/1,
    /base/data/home/runtimes/python27/python27_lib/versions/third_party/MySQLdb-1.2.4b4,
    /base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5,
    /base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0,
    /base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2,
    /base/data/home/runtimes/python27/python27_lib/versions/third_party/webob-1.1.1,
    /base/data/home/runtimes/python27/python27_lib/versions/third_party/yaml-3.10
    


Answer: It seems to be a path related issue as people have mentioned in your
question's comments

Possible short-sighted solution add everything to your path manually - look at
the top answer here: [How to import modules in Google App
Engine?](http://stackoverflow.com/questions/2710861/how-to-import-modules-in-
google-app-engine) At the very least, this will help narrow the problem to
path related.

what the docs say: <https://developers.google.com/appengine/docs/python/>

> The Python module include path includes your application's root directory
> (the directory containing the app.yaml file). Modules you create in your
> application's root directory are available using a path from the root. Don't
> forget to create **init**.py files in sub-directories, so Python will
> recognize the sub-directories as packages.

so from what I can tell, b/c everything is at or below the app.yaml file in
your question the path should already be correct.

  1. double check all your `__init__.py` file files are in place and spelled correctly. 
  2. try deleting all of your `*.pyc` files and letting them be regenerated.
  3. try importing from the container's folder name `FOLDER_CONTAINING_YAML.myfolder.middleware`

