How to run Scrapy/Portia on Azure Web App
Question: I am trying to run Scrapy or Portia on a Microsoft Azure Web App. I have
installed Scrapy by creating a virtual environment:

    
    
    D:\Python27\Scripts\virtualenv.exe D:\home\Python
    

And then installed Scrapy:

    
    
    D:\home\Python\Scripts\pip install Scrapy
    

The installation seemed to work. But executing a spider returns the following
output:

    
    
    D:\home\Python\Scripts\tutorial>d:\home\python\scripts\scrapy.exe crawl example 2015-09-13 23:09:31 [scrapy] INFO: Scrapy 1.0.3 started (bot: tutorial)
    
    2015-09-13 23:09:31 [scrapy] INFO: Optional features available: ssl, http11
    
    2015-09-13 23:09:31 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial'}
    
    2015-09-13 23:09:34 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
    
    Unhandled error in Deferred:
    
    2015-09-13 23:09:35 [twisted] CRITICAL: Unhandled error in Deferred:
    
    Traceback (most recent call last):
    
      File "D:\home\Python\lib\site-packages\scrapy\cmdline.py", line 150, in _run_command
    
        cmd.run(args, opts)
    
      File "D:\home\Python\lib\site-packages\scrapy\commands\crawl.py", line 57, in run
    
        self.crawler_process.crawl(spname, **opts.spargs)
    
      File "D:\home\Python\lib\site-packages\scrapy\crawler.py", line 153, in crawl
    
        d = crawler.crawl(*args, **kwargs)
    
      File "D:\home\Python\lib\site-packages\twisted\internet\defer.py", line 1274, in unwindGenerator
    
        return _inlineCallbacks(None, gen, Deferred())
    
    --- <exception caught here> ---
    
      File "D:\home\Python\lib\site-packages\twisted\internet\defer.py", line 1128, in _inlineCallbacks
    
        result = g.send(result)
    
      File "D:\home\Python\lib\site-packages\scrapy\crawler.py", line 71, in crawl
    
        self.engine = self._create_engine()
    
      File "D:\home\Python\lib\site-packages\scrapy\crawler.py", line 83, in _create_engine
    
        return ExecutionEngine(self, lambda _: self.stop())
    
      File "D:\home\Python\lib\site-packages\scrapy\core\engine.py", line 66, in __init__
    
        self.downloader = downloader_cls(crawler)
    
      File "D:\home\Python\lib\site-packages\scrapy\core\downloader\__init__.py", line 65, in __init__
    
        self.handlers = DownloadHandlers(crawler)
    
      File "D:\home\Python\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 23, in __init__
    
        cls = load_object(clspath)
    
      File "D:\home\Python\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
    
        mod = import_module(module)
    
      File "D:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    
        __import__(name)
    
      File "D:\home\Python\lib\site-packages\scrapy\core\downloader\handlers\s3.py", line 6, in <module>
    
        from .http import HTTPDownloadHandler
    
      File "D:\home\Python\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 5, in <module>
    
        from .http11 import HTTP11DownloadHandler as HTTPDownloadHandler
    
      File "D:\home\Python\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 15, in <module>
    
        from scrapy.xlib.tx import Agent, ProxyAgent, ResponseDone, \
    
      File "D:\home\Python\lib\site-packages\scrapy\xlib\tx\__init__.py", line 3, in <module>
    
        from twisted.web import client
    
      File "D:\home\Python\lib\site-packages\twisted\web\client.py", line 42, in <module>
    
        from twisted.internet.endpoints import TCP4ClientEndpoint, SSL4ClientEndpoint
    
      File "D:\home\Python\lib\site-packages\twisted\internet\endpoints.py", line 34, in <module>
    
        from twisted.internet.stdio import StandardIO, PipeAddress
    
      File "D:\home\Python\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
    
        from twisted.internet import _win32stdio
    
      File "D:\home\Python\lib\site-packages\twisted\internet\_win32stdio.py", line 7, in <module>
    
        import win32api
    
    exceptions.ImportError: No module named win32api
    
    
    
    2015-09-13 23:09:35 [twisted] CRITICAL:
    

The documentation <http://doc.scrapy.org/en/latest/intro/install.html> says
that I have to install **pywin32**. I don't know how I can download/install it
via command line since I am in the web app environment.

Is it even possible to run Scrapy or Portia on an Azure Web App or do I have
to use a fully fledged Virtual Machine on Azure?

Thank you!


Answer: You can't run general purpose Windows applications "on" an Azure Web App.
Things that run on Azure as web apps have to be built specifically to do so.
So, you have to use a full-fledged Virtual Machine on Azure.

It seems Azure Webapps can run some Python apps, if they are built on certain
frameworks: <https://azure.microsoft.com/en-us/documentation/articles/web-
sites-python-configure/>

