<p dir="auto">Hi</p>
<p dir="auto">I am having problem using proxy with Scrapy, I have private proxy with HTTPS supported.</p>
<p dir="auto">I tried with simplest spider without proxy with the following extensions:<br>
EXTENSIONS = {<br>
'scrapy.extensions.telnet.TelnetConsole': None,<br>
}<br>
SPIDER_MIDDLEWARES = {<br>
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,<br>
}<br>
DOWNLOADER_MIDDLEWARES = {<br>
'scrapy_splash.SplashCookiesMiddleware': 723,<br>
'scrapy_splash.SplashMiddleware': 725,<br>
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,<br>
}</p>
<p dir="auto">It worked fine 200 ok response</p>
<p dir="auto">Then I configured the proxy as suggested in the documenation:<br>
<a href="https://docs.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=proxy#module-scrapy.downloadermiddlewares.httpproxy" rel="nofollow">https://docs.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=proxy#module-scrapy.downloadermiddlewares.httpproxy</a></p>
<ol dir="auto">
<li></li>
</ol>
<p dir="auto">added this to the middle ware:<br>
DOWNLOADER_MIDDLEWARES  = {<br>
...<br>
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 1<br>
...<br>
}</p>
<ol start="2" dir="auto">
<li>I added the meta key proxy per each request as follows:<br>
request = SplashRequest(url='<a href="https://www.example.com" rel="nofollow">https://www.example.com</a>', callback=self.parse, endpoint='execute',<br>
args={'wait': 1,<br>
'lua_source': login_script,<br>
'timeout':90,<br>
}<br>
)<br>
request.meta['proxy'] = proxy<br>
yield request</li>
</ol>
<p dir="auto">where proxy = '<a href="https://username:password@proxyip:proxyport" rel="nofollow">https://username:password@proxyip:proxyport</a>'</p>
<p dir="auto">I am getting the error below when running scrapy:<br>
2019-07-19 16:11:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET <a href="https://www.example.com" rel="nofollow">https://www.example.com</a> via <a href="http://127.0.0.1:8050/execute%3E" rel="nofollow">http://127.0.0.1:8050/execute&gt;</a> (failed 1 times): [&lt;twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL rout<br>
ines', 'ssl3_get_record', 'wrong version number')]&gt;]</p>
<p dir="auto">I tried to fix by using the values of DOWNLOADER_CLIENT_TLS_METHOD such as:<br>
'TLSv1.0' 'TLSv1.1' 'TLSv1.2' 'SSLv3'<br>
but it didn't change anything same error.</p>
<p dir="auto">Thanks</p>
<p dir="auto">I'm unable to scrape https sites through https supported proxies. I've tried with proxymesh as well as other proxy services. I can scrape most of this sites without proxies or using Tor.</p>
<p dir="auto">Curl seems to work fine too:<br>
<code class="notranslate">curl -x https://xx.xx.xx.xx:xx --proxy-user user:pass -L https://www.base.net:443</code><br>
Retrieves the site's html.</p>
<p dir="auto"><strong>Setup</strong>:</p>
<ul dir="auto">
<li>OS:<br>
OS X El Capitan v10.11.3</li>
</ul>
<p dir="auto"><strong>Scrapy</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="scrapy version -v
Scrapy    : 1.0.5
lxml      : 3.5.0.0
libxml2   : 2.9.2
Twisted   : 15.5.0
Python    : 2.7.11 (default, Dec  7 2015, 23:36:10) - [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
pyOpenSSL : 0.15.1 (OpenSSL 1.0.2g  1 Mar 2016)
Platform  : Darwin-15.3.0-x86_64-i386-64bit"><pre class="notranslate"><code class="notranslate">scrapy version -v
Scrapy    : 1.0.5
lxml      : 3.5.0.0
libxml2   : 2.9.2
Twisted   : 15.5.0
Python    : 2.7.11 (default, Dec  7 2015, 23:36:10) - [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
pyOpenSSL : 0.15.1 (OpenSSL 1.0.2g  1 Mar 2016)
Platform  : Darwin-15.3.0-x86_64-i386-64bit
</code></pre></div>
<p dir="auto"><strong>Solutions tried</strong>:<br>
1 - Installing Scrapy-1.1.0rc3<br>
<code class="notranslate">2016-03-09 12:44:59 [scrapy] ERROR: Error downloading &lt;GET https://www.base.net/&gt;: [&lt;twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]&gt;]</code><br>
Other website:<br>
<code class="notranslate">2016-03-09 12:56:45 [scrapy] DEBUG: Retrying &lt;GET https://es.alojadogatopreto.com/es-es/&gt; (failed 1 times): [&lt;twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]&gt;]</code></p>
<p dir="auto">2 - <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="131659833" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/1764" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/1764/hovercard?comment_id=181950638&amp;comment_type=issue_comment" href="https://github.com/scrapy/scrapy/issues/1764#issuecomment-181950638">#1764 (comment)</a><br>
Using SSLv23_METHOD<br>
<code class="notranslate">2016-03-09 12:22:40 [scrapy] ERROR: Error downloading &lt;GET https://www.base.net/&gt;: [&lt;twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]&gt;]</code><br>
Using other SSL methods<br>
<code class="notranslate">2016-03-09 12:24:11 [scrapy] ERROR: Error downloading &lt;GET https://www.base.net/&gt;: [&lt;twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_RECORD', 'wrong version number')]&gt;]</code></p>
<p dir="auto">3 - <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="75785040" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/1227" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/1227/hovercard?comment_id=154890557&amp;comment_type=issue_comment" href="https://github.com/scrapy/scrapy/issues/1227#issuecomment-154890557">#1227 (comment)</a>  | Get same errors as in 1 &amp; 2.<br>
4 -  <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="100553508" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/1429" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/1429/hovercard?comment_id=131187012&amp;comment_type=issue_comment" href="https://github.com/scrapy/scrapy/issues/1429#issuecomment-131187012">#1429 (comment)</a> | Get same errors as in 1 &amp; 2.</p>