prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  fetch errors on some https sites
  $ scrapy fetch 'https://flixbus.de'
  
  ```
  2014-12-12 10:01:50+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
  2014-12-12 10:01:50+0100 [scrapy] INFO: Optional features available: ssl, http11
  2014-12-12 10:01:50+0100 [scrapy] INFO: Overridden settings: {}
  2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole,  CloseSpider, WebService, CoreStats, SpiderState
  2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
  2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
  2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled item pipelines: 
  2014-12-12 10:01:50+0100 [default] INFO: Spider opened
  2014-12-12 10:01:50+0100 [default] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2014-12-12 10:01:50+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
  2014-12-12 10:01:50+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
  2014-12-12 10:01:50+0100 [default] DEBUG: Retrying <GET https://flixbus.de> (failed 1 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
  2014-12-12 10:01:50+0100 [default] DEBUG: Retrying <GET https://flixbus.de> (failed 2 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
  2014-12-12 10:01:50+0100 [default] DEBUG: Gave up retrying <GET https://flixbus.de> (failed 3 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
  2014-12-12 10:01:50+0100 [default] ERROR: Error downloading <GET https://flixbus.de>: [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
  2014-12-12 10:01:50+0100 [default] INFO: Closing spider (finished)
  2014-12-12 10:01:50+0100 [default] INFO: Dumping Scrapy stats:
      {'downloader/exception_count': 3,
       'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 3,
       'downloader/request_bytes': 627,
       'downloader/request_count': 3,
       'downloader/request_method_count/GET': 3,
       'finish_reason': 'finished',
       'finish_time': datetime.datetime(2014, 12, 12, 9, 1, 50, 463722),
       'log_count/DEBUG': 5,
       'log_count/ERROR': 1,
       'log_count/INFO': 7,
       'scheduler/dequeued': 3,
       'scheduler/dequeued/memory': 3,
       'scheduler/enqueued': 3,
       'scheduler/enqueued/memory': 3,
       'start_time': datetime.datetime(2014, 12, 12, 9, 1, 50, 288873)}
  2014-12-12 10:01:50+0100 [default] INFO: Spider closed (finished)
  ```
  
  $ pip freeze
  
  ```
  Scrapy==0.24.4
  Twisted==14.0.2
  argparse==1.2.1
  cffi==0.8.6
  characteristic==14.2.0
  cryptography==0.6.1
  cssselect==0.9.1
  distribute==0.7.3
  lxml==3.4.1
  pyOpenSSL==0.14
  pyasn1==0.1.7
  pyasn1-modules==0.0.5
  pycparser==2.10
  queuelib==1.2.2
  service-identity==14.0.0
  six==1.8.0
  w3lib==1.10.0
  wsgiref==0.1.2
  zope.interface==4.1.1
  ```
  
  Same error on scrapinghub ;)
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `scrapy/core/downloader/contextfactory.py`
  Content:
  ```
  1 from OpenSSL import SSL
  2 from twisted.internet.ssl import ClientContextFactory
  3 
  4 
  5 class ScrapyClientContextFactory(ClientContextFactory):
  6     "A SSL context factory which is more permissive against SSL bugs."
  7     # see https://github.com/scrapy/scrapy/issues/82
  8     # and https://github.com/scrapy/scrapy/issues/26
  9 
  10     def __init__(self):
  11         # see this issue on why we use TLSv1_METHOD by default
  12         # https://github.com/scrapy/scrapy/issues/194
  13         self.method = SSL.TLSv1_METHOD
  14 
  15     def getContext(self, hostname=None, port=None):
  16         ctx = ClientContextFactory.getContext(self)
  17         # Enable all workarounds to SSL bugs as documented by
  18         # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html
  19         ctx.set_options(SSL.OP_ALL)
  20         return ctx
  21
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py
  --- a/scrapy/core/downloader/contextfactory.py
  +++ b/scrapy/core/downloader/contextfactory.py
  @@ -1,11 +1,17 @@
   from OpenSSL import SSL
   from twisted.internet.ssl import ClientContextFactory
  +try:
  +    # available since twisted 14.0
  +    from twisted.internet._sslverify import ClientTLSOptions
  +except ImportError:
  +    ClientTLSOptions = None
   
   
   class ScrapyClientContextFactory(ClientContextFactory):
       "A SSL context factory which is more permissive against SSL bugs."
       # see https://github.com/scrapy/scrapy/issues/82
       # and https://github.com/scrapy/scrapy/issues/26
  +    # and https://github.com/scrapy/scrapy/issues/981
   
       def __init__(self):
           # see this issue on why we use TLSv1_METHOD by default
  @@ -17,4 +23,6 @@
           # Enable all workarounds to SSL bugs as documented by
           # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html
           ctx.set_options(SSL.OP_ALL)
  +        if hostname and ClientTLSOptions is not None: # workaround for TLS SNI
  +            ClientTLSOptions(hostname, ctx)
           return ctx
  
verification_info: {"golden_diff": "diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py\n--- a/scrapy/core/downloader/contextfactory.py\n+++ b/scrapy/core/downloader/contextfactory.py\n@@ -1,11 +1,17 @@\n from OpenSSL import SSL\n from twisted.internet.ssl import ClientContextFactory\n+try:\n+    # available since twisted 14.0\n+    from twisted.internet._sslverify import ClientTLSOptions\n+except ImportError:\n+    ClientTLSOptions = None\n \n \n class ScrapyClientContextFactory(ClientContextFactory):\n     \"A SSL context factory which is more permissive against SSL bugs.\"\n     # see https://github.com/scrapy/scrapy/issues/82\n     # and https://github.com/scrapy/scrapy/issues/26\n+    # and https://github.com/scrapy/scrapy/issues/981\n \n     def __init__(self):\n         # see this issue on why we use TLSv1_METHOD by default\n@@ -17,4 +23,6 @@\n         # Enable all workarounds to SSL bugs as documented by\n         # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html\n         ctx.set_options(SSL.OP_ALL)\n+        if hostname and ClientTLSOptions is not None: # workaround for TLS SNI\n+            ClientTLSOptions(hostname, ctx)\n         return ctx\n", "issue": "fetch errors on some https sites\n$ scrapy fetch 'https://flixbus.de'\n\n```\n2014-12-12 10:01:50+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)\n2014-12-12 10:01:50+0100 [scrapy] INFO: Optional features available: ssl, http11\n2014-12-12 10:01:50+0100 [scrapy] INFO: Overridden settings: {}\n2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole,  CloseSpider, WebService, CoreStats, SpiderState\n2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats\n2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware\n2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled item pipelines: \n2014-12-12 10:01:50+0100 [default] INFO: Spider opened\n2014-12-12 10:01:50+0100 [default] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\n2014-12-12 10:01:50+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023\n2014-12-12 10:01:50+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080\n2014-12-12 10:01:50+0100 [default] DEBUG: Retrying <GET https://flixbus.de> (failed 1 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]\n2014-12-12 10:01:50+0100 [default] DEBUG: Retrying <GET https://flixbus.de> (failed 2 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]\n2014-12-12 10:01:50+0100 [default] DEBUG: Gave up retrying <GET https://flixbus.de> (failed 3 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]\n2014-12-12 10:01:50+0100 [default] ERROR: Error downloading <GET https://flixbus.de>: [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]\n2014-12-12 10:01:50+0100 [default] INFO: Closing spider (finished)\n2014-12-12 10:01:50+0100 [default] INFO: Dumping Scrapy stats:\n    {'downloader/exception_count': 3,\n     'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 3,\n     'downloader/request_bytes': 627,\n     'downloader/request_count': 3,\n     'downloader/request_method_count/GET': 3,\n     'finish_reason': 'finished',\n     'finish_time': datetime.datetime(2014, 12, 12, 9, 1, 50, 463722),\n     'log_count/DEBUG': 5,\n     'log_count/ERROR': 1,\n     'log_count/INFO': 7,\n     'scheduler/dequeued': 3,\n     'scheduler/dequeued/memory': 3,\n     'scheduler/enqueued': 3,\n     'scheduler/enqueued/memory': 3,\n     'start_time': datetime.datetime(2014, 12, 12, 9, 1, 50, 288873)}\n2014-12-12 10:01:50+0100 [default] INFO: Spider closed (finished)\n```\n\n$ pip freeze\n\n```\nScrapy==0.24.4\nTwisted==14.0.2\nargparse==1.2.1\ncffi==0.8.6\ncharacteristic==14.2.0\ncryptography==0.6.1\ncssselect==0.9.1\ndistribute==0.7.3\nlxml==3.4.1\npyOpenSSL==0.14\npyasn1==0.1.7\npyasn1-modules==0.0.5\npycparser==2.10\nqueuelib==1.2.2\nservice-identity==14.0.0\nsix==1.8.0\nw3lib==1.10.0\nwsgiref==0.1.2\nzope.interface==4.1.1\n```\n\nSame error on scrapinghub ;)\n\n", "before_files": [{"content": "from OpenSSL import SSL\nfrom twisted.internet.ssl import ClientContextFactory\n\n\nclass ScrapyClientContextFactory(ClientContextFactory):\n    \"A SSL context factory which is more permissive against SSL bugs.\"\n    # see https://github.com/scrapy/scrapy/issues/82\n    # and https://github.com/scrapy/scrapy/issues/26\n\n    def __init__(self):\n        # see this issue on why we use TLSv1_METHOD by default\n        # https://github.com/scrapy/scrapy/issues/194\n        self.method = SSL.TLSv1_METHOD\n\n    def getContext(self, hostname=None, port=None):\n        ctx = ClientContextFactory.getContext(self)\n        # Enable all workarounds to SSL bugs as documented by\n        # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html\n        ctx.set_options(SSL.OP_ALL)\n        return ctx\n", "path": "scrapy/core/downloader/contextfactory.py"}], "after_files": [{"content": "from OpenSSL import SSL\nfrom twisted.internet.ssl import ClientContextFactory\ntry:\n    # available since twisted 14.0\n    from twisted.internet._sslverify import ClientTLSOptions\nexcept ImportError:\n    ClientTLSOptions = None\n\n\nclass ScrapyClientContextFactory(ClientContextFactory):\n    \"A SSL context factory which is more permissive against SSL bugs.\"\n    # see https://github.com/scrapy/scrapy/issues/82\n    # and https://github.com/scrapy/scrapy/issues/26\n    # and https://github.com/scrapy/scrapy/issues/981\n\n    def __init__(self):\n        # see this issue on why we use TLSv1_METHOD by default\n        # https://github.com/scrapy/scrapy/issues/194\n        self.method = SSL.TLSv1_METHOD\n\n    def getContext(self, hostname=None, port=None):\n        ctx = ClientContextFactory.getContext(self)\n        # Enable all workarounds to SSL bugs as documented by\n        # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html\n        ctx.set_options(SSL.OP_ALL)\n        if hostname and ClientTLSOptions is not None: # workaround for TLS SNI\n            ClientTLSOptions(hostname, ctx)\n        return ctx\n", "path": "scrapy/core/downloader/contextfactory.py"}]}