title: Disable SSL certificate verification in Scrapy
body: <p>I am currently struggling with an issue I am having with Scrapy. Whenever I used Scrapy to scrape an HTTPS site where the certificate's CN value matches the server's domain name, Scrapy works great! On the other hand, though, whenever I try scraping a site where the certificate's CN value does NOT match the server's domain name, I get the following:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 415, in dataReceived
      self._write(bytes)
    File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 554, in _write
      sent = self._tlsConnection.send(toSend)
    File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1270, in send
      result = _lib.SSL_write(self._ssl, buf, len(buf))
    File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 926, in wrapper
      callback(Connection._reverse_mapping[ssl], where, return_code)
  --- &lt;exception caught here&gt; ---
    File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_sslverify.py", line 1055, in infoCallback
      return wrapped(connection, where, ret)
    File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_sslverify.py", line 1154, in _identityVerifyingInfoCallback
      verifyHostname(connection, self._hostnameASCII)
    File "/usr/local/lib/python2.7/dist-packages/service_identity/pyopenssl.py", line 30, in verify_hostname
      obligatory_ids=[DNS_ID(hostname)],
    File "/usr/local/lib/python2.7/dist-packages/service_identity/_common.py", line 235, in __init__
      raise ValueError("Invalid DNS-ID.")
  exceptions.ValueError: Invalid DNS-ID.
  </code></pre>
  
  <p>I have looked through as much documentation as I can, and as far as I can tell Scrapy does not have a way to disable SSL certificate verification. Even the documentation for the Scrapy Request object (which I would assume is where this functionality would lie) has no reference:</p>
  
  <p><a href="http://doc.scrapy.org/en/1.0/topics/request-response.html#scrapy.http.Request" rel="noreferrer">http://doc.scrapy.org/en/1.0/topics/request-response.html#scrapy.http.Request</a>
  <a href="https://github.com/scrapy/scrapy/blob/master/scrapy/http/request/__init__.py" rel="noreferrer">https://github.com/scrapy/scrapy/blob/master/scrapy/http/request/<strong>init</strong>.py</a></p>
  
  <p>There are also no Scrapy settings which address the issue:</p>
  
  <p><a href="http://doc.scrapy.org/en/1.0/topics/settings.html" rel="noreferrer">http://doc.scrapy.org/en/1.0/topics/settings.html</a></p>
  
  <p>Short of using Scrapy by source and modifying the source as needed, does anyone have any ideas for how I can disable the SSL certificate verification?</p>
  
  <p>Thank you!</p>