How to import lib folder within Modules
Question: I had a GAE app which contains three Modules and a lib folder. When I tried to
import the 3rd party library from the lib folder. GAE pops a ImportError.

I could get it to work by symlinking ./lib to ./Module_1/lib and
./Module_2/lib and also creating a appengine_config.py in each of the modules.
But doing this seemed really dirty.  
Is there a cleaner way to import app_root/lib from module_1 and module_2?

This seemed to be
promising(<https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Includes>),
but don't know what to put inside include.yaml.

    
    
    -- App Root/
        -- Module_1/
           module_1.yaml 
           module_1.py
    
        -- Module_2/
           module_2.yaml
           module_2.py
    
        -- lib/ 
           -- cloudstorage/
              ..
           -- 3rd_library_1/
              ..
              ..
           -- 3rd_library_2/
              ..
              ..
    
        appengine_config.py
        main.py (default module)
        app.yaml(default module)
        queue.yaml
        dispatch.yaml
    

In module_1.py or module_2.py, when I do

    
    
        import cloudstorage as gcs
    

It complains

    
    
        ImportError: No module named cloudstorage
    

However, when it's being imported within main.py, it works fine.

In the appengine_config.py:

    
    
    import os
    import sys
    
    # Add ./lib to sys path
    sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))
    

Also tried to print sys.path from main.py:

    
    
    sys.path in main.py :
    [ 
      '/base/data/home/apps/s~my-app/2.381942946570489905', 
      '/base/data/home/apps/s~my-app/2.381942946570489905/lib', 
      ...
      ...
     ]  
    

sys.path in module_1.py:

    
    
     [
      '/base/data/home/apps/s~my-app/module_1:2.381942955973772449',      
      '/base/data/home/runtimes/python27/python27_dist/lib/python27.zip', 
       ...
       ...
      ]
    


Answer: Credit goes to Google Cloud Platform Tech Solutions Representative Adam:

> Modules documentations may not be explicitly stated, but the folder
> 'Module1', 'Module2' as well as the default module actually run inside
> separate Python virtual environments on separate instances and need to be
> self contained. They cannot 'see' any directories above them which exist on
> the local filesystem, and 'default.py' can't see anything in each of the
> module directories. The whole folder tree isn't copied to each module
> instance.

He suggested that instead of making symlinks, just copy ./lib to each of the
modules.

I do not like the idea very much.

Firstly, these modules share some base class, duplicating them is really an
anti-pattern.

Secondly, copying the lib folders everywhere corrupts unit tests as nose will
try to run all unit tests it can run, also because it's a pain to explicitly
exclude the directories.

At the end of the day, I wrote a makefile to help deployment / testing
easier...

    
    
    # Create simlinks before deployment.
    deploy: mksimlnks
        appcfg.py --oauth2 update $(CURDIR)/app.yaml
        appcfg.py --oauth2 update $(CURDIR)/MODULE_1/module_1.yaml
        appcfg.py --oauth2 update $(CURDIR)/MODULE_2/module_2.yaml
        appcfg.py --oauth2 update_queues $(CURDIR)
    
    mksimlnks:
        ln -s $(CURDIR)/lib $(CURDIR)/MODULE_1/lib
        ln -s $(CURDIR)/lib $(CURDIR)/MODULE_2/lib
    
    # Need to remove symlinks before unittest
    # or unit test will explode.
    test: rmsimlnks
        nosetests --exclude-dir=lib --with-gae -w $(CURDIR) --with-coverage --cover-html
    
    # Remove all symlinks
    rmsimlnks:
        rm -rf $(shell find * -type l)
    
    # remove symlinks and other stuff
    clean: rmsimlnks
        rm -f $(shell find * -name *.pyc)
        rm -f $(shell find * -name .DS_Store)
        rm -f .coverage
        rm -rf $(CURDIR)/cover
    

