Scrapy and Pycharm - Stange import error - No module named [insert name of scrapyproject here]
Question: Hi Stackoverflow Community

I encountered the following issue. I have a scrapy project which I added to my
project:

    
    
    -.idea
    -associate
    -core
    -scrapyproject
       -- scrapyproject_one
         --- spiders
           ---- __iniy.py__
           ---- dmoz_spider.py
       -- __init__.py
       -- items.py
       -- pipelines.py
       -- settings.py
    

My dmoz_spider.py looks like this:

    
    
    import scrapy
    
    from scrapyproject.scrapyproject_one import items
    
    class DmozSpider(scrapy.Spider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
           "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]
    
        def parse(self, response):
            for sel in response.xpath('//ul/li'):
                item = items.ScrapyprojectItem()
                item['title'] = sel.xpath('a/text()').extract()
                item['link'] = sel.xpath('a/@href').extract()
                item['desc'] = sel.xpath('text()').extract()
                yield item
    

But when I navigate into the scrapyproject> folder and execute

    
    
    scrapy dmoz crawl
    

I receive the following error:

    
    
        Traceback (most recent call last):
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\runpy.py", line 170, in _run_module_as_main
        "__main__", mod_spec)
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\Scripts\scrapy.exe\__main__.py", line 9, in <module>
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\cmdline.py", line 108, in execute
        settings = get_project_settings()
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\utils\project.py", line 60, in get_proj
    ect_settings
        settings.setmodule(settings_module_path, priority='project')
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\settings\__init__.py", line 282, in set
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\utils\project.py", line 60, in g
    et_project_settings
        settings.setmodule(settings_module_path, priority='project')
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\settings\__init__.py", line 282,
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\cmdline.py", line 108, in
    execute
        settings = get_project_settings()
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\utils\project.py", line 60
    , in get_project_settings
        settings.setmodule(settings_module_path, priority='project')
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\settings\__init__.py", lin
    e 282, in setmodule
    ne 60, in get_project_settings
        settings.setmodule(settings_module_path, priority='project')
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\settings\__init__.py"
    , line 282, in setmodule
    ct.py", line 60, in get_project_settings
        settings.setmodule(settings_module_path, priority='project')
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\site-packages\scrapy\settings\__
    init__.py", line 282, in setmodule
        module = import_module(module)
      File "c:\users\admin\appdata\local\programs\python\python35-32\lib\importlib\__init__.py", line 126
     126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 986, in _gcd_import
      File "<frozen importlib._bootstrap>", line 969, in _find_and_load
      File "<frozen importlib._bootstrap>", line 944, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 986, in _gcd_import
      File "<frozen importlib._bootstrap>", line 969, in _find_and_load
      File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
    ImportError: No module named 'scrapyproject'
    

Wondering whether anyone would know how I might be able to approach this. Any
advice would be highly appreciated!

M


Answer: All right, I figured it out.

What I needed to do was to declare my 'spiderproject' folder as a 'Sources
folder' in PyCharm.

You can do that by going to File>Settings>Project:[Project Name]>Project
Structure.

Select the level 1 project folder of your scrapy project (in this case
'spiderproject') and Mark as Sources by clicking the blue Folder at the top.

Then go to your spider and

    
    
     from spiderproject.items import [whatever you named your item class you defined in items.py ]
    

Hope this helps.

M

