Scrapy: How do I set an HTTP proxy to connect to HTTPS websites (HTTP works)?
Question: I'm using a middleware to enable a proxy like this:

I have this in settings.py

    
    
    HTTP_PROXY='127.0.0.1:8080'
    

This is my middleware

    
    
    from mybot.settings import HTTP_PROXY
    
    class ProxyMiddleware(object):
          def process_request(self, request, spider):
              request.meta['proxy'] = 'http://%s' % HTTP_PROXY
    

It works fine for HTTP sites but not for HTTPS. What am I doing wrong? I have
tested the proxy using curl and it can connect to HTTPS without a problem.

This is the error:

    
    
    [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
    


Answer: You need to enable proxy for **http** and **https** both. For example in
Windows, you can do it directly from shell like

    
    
    set http_proxy = 127.0.0.1:8080
    set https_proxy = 127.0.0.1:8080
    

