<p dir="auto">Scrapy       : 1.5.0<br>
lxml         : 4.1.1.0<br>
libxml2      : 2.9.5<br>
cssselect    : 1.0.3<br>
parsel       : 1.4.0<br>
w3lib        : 1.19.0<br>
Twisted      : 17.9.0<br>
Python       : 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]<br>
pyOpenSSL    : 17.6.0.dev0 (OpenSSL 1.1.0g  2 Nov 2017)<br>
cryptography : 2.1.4<br>
Platform     : Windows-7-6.1.7601-SP1</p>
<p dir="auto">2018-03-12 21:20:46 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET <a href="http://www.farmerscompress.com/%3E" rel="nofollow">http://www.farmerscompress.com/&gt;</a> (referer: None)<br>
2018-03-12 21:20:46 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;POST <a href="https://www.farmerscompress.com/ProcessUser.aspx%3E" rel="nofollow">https://www.farmerscompress.com/ProcessUser.aspx&gt;</a> (failed 1 times): []<br>
2018-03-12 21:20:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;POST <a href="https://www.farmerscompress.com/ProcessUser.aspx%3E" rel="nofollow">https://www.farmerscompress.com/ProcessUser.aspx&gt;</a> (failed 2 times): []<br>
2018-03-12 21:20:50 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying &lt;POST <a href="https://www.farmerscompress.com/ProcessUser.aspx%3E" rel="nofollow">https://www.farmerscompress.com/ProcessUser.aspx&gt;</a> (failed 3 tim<br>
es): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection<br>
lost.&gt;]<br>
2018-03-12 21:20:50 [scrapy.core.scraper] ERROR: Error downloading &lt;POST <a href="https://www.farmerscompress.com/ProcessUser.aspx%3E" rel="nofollow">https://www.farmerscompress.com/ProcessUser.aspx&gt;</a>: [&lt;twisted.python.failure.F<br>
ailure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]<br>
2018-03-12 21:20:51 [scrapy.core.engine] INFO: Closing spider (finished)<br>
2018-03-12 21:20:51 [scrapy.statscollectors] INFO: Dumping Scrapy stats:</p>
<p dir="auto">Hi,</p>
<p dir="auto">I'm attempting to run scrapy on <code class="notranslate">https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences</code>.</p>
<p dir="auto">Despite upgrading all my brew packages, scrapy installation via <code class="notranslate">pip install -U scrapy</code>, the website disconnects uncleanly. Specifying <code class="notranslate">-s DOWNLOADER_CLIENT_TLS_METHOD=TLSv1.0</code> makes no difference to the following run. Neither does a user-agent clear things up with <code class="notranslate">-s USER_AGENT="Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"</code></p>
<p dir="auto">I am uncertain as how to debug this further.</p>
<p dir="auto"><code class="notranslate">scrapy shell https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2018-01-09 18:03:33 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-01-09 18:03:33 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.3.1, w3lib 1.18.0, Twisted 17.9.0, Python 3.6.2 (default, Jul 17 2017, 16:44:45) - [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), cryptography 2.1.4, Platform Darwin-16.7.0-x86_64-i386-64bit
2018-01-09 18:03:33 [scrapy.crawler] INFO: Overridden settings: {'DOWNLOADER_CLIENT_TLS_METHOD': 'TLSv1.0', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-01-09 18:03:33 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-01-09 18:03:33 [scrapy.core.engine] INFO: Spider opened
2018-01-09 18:03:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences&gt; (failed 1 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
2018-01-09 18:03:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences&gt; (failed 2 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
2018-01-09 18:03:35 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying &lt;GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences&gt; (failed 3 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
Traceback (most recent call last):
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/bin/scrapy&quot;, line 11, in &lt;module&gt;
    sys.exit(execute())
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py&quot;, line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py&quot;, line 90, in _run_print_help
    func(*a, **kw)
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py&quot;, line 157, in _run_command
    cmd.run(args, opts)
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/commands/shell.py&quot;, line 73, in run
    shell.start(url=url, redirect=not opts.no_redirect)
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/shell.py&quot;, line 48, in start
    self.fetch(url, spider, redirect=redirect)
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/shell.py&quot;, line 115, in fetch
    reactor, self._schedule, request, spider)
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/twisted/internet/threads.py&quot;, line 122, in blockingCallFromThread
    result.raiseException()
  File &quot;/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/twisted/python/failure.py&quot;, line 385, in raiseException
    raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseNeverReceived: [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]"><pre class="notranslate"><code class="notranslate">2018-01-09 18:03:33 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-01-09 18:03:33 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.3.1, w3lib 1.18.0, Twisted 17.9.0, Python 3.6.2 (default, Jul 17 2017, 16:44:45) - [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), cryptography 2.1.4, Platform Darwin-16.7.0-x86_64-i386-64bit
2018-01-09 18:03:33 [scrapy.crawler] INFO: Overridden settings: {'DOWNLOADER_CLIENT_TLS_METHOD': 'TLSv1.0', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-01-09 18:03:33 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-01-09 18:03:33 [scrapy.core.engine] INFO: Spider opened
2018-01-09 18:03:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences&gt; (failed 1 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
2018-01-09 18:03:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying &lt;GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences&gt; (failed 2 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
2018-01-09 18:03:35 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying &lt;GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences&gt; (failed 3 times): [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
Traceback (most recent call last):
  File "/Users/BenJolitz/.virtualenvs/cpython36/bin/scrapy", line 11, in &lt;module&gt;
    sys.exit(execute())
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py", line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
    func(*a, **kw)
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py", line 157, in _run_command
    cmd.run(args, opts)
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/commands/shell.py", line 73, in run
    shell.start(url=url, redirect=not opts.no_redirect)
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/shell.py", line 48, in start
    self.fetch(url, spider, redirect=redirect)
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/shell.py", line 115, in fetch
    reactor, self._schedule, request, spider)
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/twisted/python/failure.py", line 385, in raiseException
    raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseNeverReceived: [&lt;twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.&gt;]
</code></pre></div>
<p dir="auto"><code class="notranslate">scrapy version -v</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Scrapy       : 1.5.0
lxml         : 4.1.1.0
libxml2      : 2.9.7
cssselect    : 1.0.3
parsel       : 1.3.1
w3lib        : 1.18.0
Twisted      : 17.9.0
Python       : 3.6.2 (default, Jul 17 2017, 16:44:45) - [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
pyOpenSSL    : 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017)
cryptography : 2.1.4
Platform     : Darwin-16.7.0-x86_64-i386-64bit"><pre class="notranslate"><code class="notranslate">Scrapy       : 1.5.0
lxml         : 4.1.1.0
libxml2      : 2.9.7
cssselect    : 1.0.3
parsel       : 1.3.1
w3lib        : 1.18.0
Twisted      : 17.9.0
Python       : 3.6.2 (default, Jul 17 2017, 16:44:45) - [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
pyOpenSSL    : 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017)
cryptography : 2.1.4
Platform     : Darwin-16.7.0-x86_64-i386-64bit
</code></pre></div>
<p dir="auto"><code class="notranslate">curl -sLv https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences 2&gt;&amp;1 | head -20</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="*   Trying 160.92.134.3...
* TCP_NODELAY set
* Connected to agences.creditfoncier.fr (160.92.134.3) port 443 (#0)
* TLS 1.0 connection using TLS_RSA_WITH_3DES_EDE_CBC_SHA
* Server certificate: agences.creditfoncier.fr
* Server certificate: RapidSSL RSA CA 2018
* Server certificate: DigiCert Global Root CA
&gt; GET /credit-immobilier/toutes-nos-agences HTTP/1.1
&gt; Host: agences.creditfoncier.fr
&gt; User-Agent: curl/7.54.0
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 200 OK
&lt; Date: Wed, 10 Jan 2018 02:06:54 GMT
&lt; Server: Microsoft-IIS/6.0
&lt; X-Powered-By: ASP.NET
&lt; Content-Length: 35884
&lt; Content-Type: text/html
&lt; Set-Cookie: ASPSESSIONIDQQBACDTS=JHOJNNCCJNIFABOPLNCEEAFH; path=/
&lt; Cache-control: private"><pre class="notranslate"><code class="notranslate">*   Trying 160.92.134.3...
* TCP_NODELAY set
* Connected to agences.creditfoncier.fr (160.92.134.3) port 443 (#0)
* TLS 1.0 connection using TLS_RSA_WITH_3DES_EDE_CBC_SHA
* Server certificate: agences.creditfoncier.fr
* Server certificate: RapidSSL RSA CA 2018
* Server certificate: DigiCert Global Root CA
&gt; GET /credit-immobilier/toutes-nos-agences HTTP/1.1
&gt; Host: agences.creditfoncier.fr
&gt; User-Agent: curl/7.54.0
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 200 OK
&lt; Date: Wed, 10 Jan 2018 02:06:54 GMT
&lt; Server: Microsoft-IIS/6.0
&lt; X-Powered-By: ASP.NET
&lt; Content-Length: 35884
&lt; Content-Type: text/html
&lt; Set-Cookie: ASPSESSIONIDQQBACDTS=JHOJNNCCJNIFABOPLNCEEAFH; path=/
&lt; Cache-control: private
</code></pre></div>