Scrapy ImportError: No module named project.settings when using subprocess.Popen
Question: I have scrapy crawler scraping thru sites. On some occasions scrapy kills
itself due to RAM issues. I rewrote the spider such that it can be split and
run for a site.

After the initial run, I use subprocess.Popen to submit the scrapy crawler
again with new start item.

But I am getting error

`ImportError: No module named shop.settingsTraceback (most recent call last):
File "/home/kumar/envs/ishop/bin/scrapy", line 4, in <module> execute() File
"/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/cmdline.py", line
109, in execute settings = get_project_settings() File
"/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/utils/project.py",
line 60, in get_project_settings settings.setmodule(settings_module_path,
priority='project') File "/home/kumar/envs/ishop/lib/python2.7/site-
packages/scrapy/settings/__init__.py", line 109, in setmodule module =
import_module(module) File "/usr/lib64/python2.7/importlib/__init__.py", line
37, in import_module __import__(name)ImportError: No module named
shop.settings`

The subprocess cmd is

`newp = Popen(comm, stderr=filename, stdout=filename, cwd=fp, shell=True)`

  * comm - `source /home/kumar/envs/ishop/bin/activate && cd /home/kumar/projects/usg/shop/spiders/../.. && /home/kumar/envs/ishop/bin/scrapy crawl -a category=laptop -a site=newsite -a start=2 -a numpages=10 -a split=1 'allsitespider'`

  * cwd - **/home/kumar/projects/usg**

I checked sys.path and it is correct `['/home/kumar/envs/ishop/bin',
'/home/kumar/envs/ishop/lib64/python27.zip',
'/home/kumar/envs/ishop/lib64/python2.7',
'/home/kumar/envs/ishop/lib64/python2.7/plat-linux2',
'/home/kumar/envs/ishop/lib64/python2.7/lib-tk',
'/home/kumar/envs/ishop/lib64/python2.7/lib-old',
'/home/kumar/envs/ishop/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7',
'/usr/lib/python2.7', '/home/kumar/envs/ishop/lib/python2.7/site-packages']`

But looks like the import statement is using
`"/usr/lib64/python2.7/importlib/__init__.py"` instead of my virtual env.

Where am I wrong? Help please?


Answer: Looks like the settings in not being loaded properly. One solution would be to
build an egg and deploy it in the env before starting the crawler.

Official docs, [Eggify scrapy
project](http://doc.scrapy.org/en/0.7/topics/scrapyd.html#deploying-your-
project)

