<p dir="auto">twisted.internet.base.ThreadedResolver currently (18.9.1.dev0) uses socket.gethostbyname to do DNS resolution, which does not support IPv6.</p>
<p dir="auto">Meanwhile, scrapy currently (1.5.0) uses ThreadedResolver to do DNS resolution, thus unable to crawl host name with IPv6 address only or pure IPv6 ip address.</p>
<p dir="auto"><a href="https://github.com/scrapy/scrapy/blob/1fd1702a11a56ecbe9851ba4f9d3c10797e262dd/scrapy/resolver.py">https://github.com/scrapy/scrapy/blob/1fd1702a11a56ecbe9851ba4f9d3c10797e262dd/scrapy/resolver.py</a></p>
<p dir="auto">Script to reproduce the scrapy exception:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if [ ! -d testvenv ]; then virtualenv -p python testvenv ; fi
. testvenv/bin/activate
pip install -q scrapy  # or install twisted and scrapy from source repository

python -m scrapy shell http://ipv6.google.com
# raises:
# twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: ipv6.google.com.

python -m scrapy shell http://[::1]  # no need for existing web server in [::1]:80
# raises:
# twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: ::1."><pre class="notranslate"><span class="pl-k">if</span> [ <span class="pl-k">!</span> <span class="pl-k">-d</span> testvenv ]<span class="pl-k">;</span> <span class="pl-k">then</span> virtualenv -p python testvenv <span class="pl-k">;</span> <span class="pl-k">fi</span>
<span class="pl-c1">.</span> testvenv/bin/activate
pip install -q scrapy  <span class="pl-c"><span class="pl-c">#</span> or install twisted and scrapy from source repository</span>

python -m scrapy shell http://ipv6.google.com
<span class="pl-c"><span class="pl-c">#</span> raises:</span>
<span class="pl-c"><span class="pl-c">#</span> twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: ipv6.google.com.</span>

python -m scrapy shell http://[::1]  <span class="pl-c"><span class="pl-c">#</span> no need for existing web server in [::1]:80</span>
<span class="pl-c"><span class="pl-c">#</span> raises:</span>
<span class="pl-c"><span class="pl-c">#</span> twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: ::1.</span></pre></div>
<p dir="auto">But the default resolver in recent twisted supports IPv6 well.</p>
<p dir="auto">Delete <code class="notranslate">reactor.installResolver(self._get_dns_resolver())</code> in<br>
<a href="https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L289">https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L289</a><br>
and test again:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -m scrapy shell http://ipv6.google.com
python -m scrapy shell http://[::1]"><pre class="notranslate"><code class="notranslate">python -m scrapy shell http://ipv6.google.com
python -m scrapy shell http://[::1]
</code></pre></div>
<p dir="auto">it works and no DNSLookupError exception raised.</p>
<p dir="auto">Although CachingThreadedResolver is used to cache DNS query, I think that compatibility with IPv6 outweigh the requirement of in process DNS cache, or we can reimplement DNS cache using middleware. At least, how about supply a option for user to explicitly enable IPv6 support in case the <code class="notranslate">DNSCACHE_ENABLED = False</code>?<br>
Regarding the timeout option in getHostByName, I think there should be other approaches to set DNS query timeout (I'm not familiar with twisted).</p>
<p dir="auto">In deed, I tried creating PR to twisted to make <code class="notranslate">ThreadedResolver.getHostByName</code> use <code class="notranslate">getaddrinfo</code> to support IPv6, but after discussing with cdunklau in irc://freenode/twisted-dev , I realised it's a backward incompatible change and is unlikely to be accepted, then concluded that the best option is to use the the default IPv6 compatible resolver in twisted instead of antiquated <code class="notranslate">ThreadedResolver.getHostByName</code>.</p>
<p dir="auto">(note: to use the new default resolver which is IPv6 compatible, the required Twisted version should also be updated)</p>
<p dir="auto">I'm running into problems while trying to run the example on the scrapy.org home page from the FOSDEM IPv6-only Wi-Fi network. (The scraper works fine from an IPv4 network.)</p>
<p dir="auto">If both IPv4 and IPv6 are enabled on my computer (OS X Yosemite), and the IPv4 is configured with DHCP, and thus gets a self-assigned address (169.254.x.x), then I get timeout errors:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ scrapy runspider myspider.py
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'.  Please install it from &lt;https://pypi.python.org/pypi/service_identity&gt; and make sure all of its dependencies are satisfied.  Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.
2015-01-31 12:13:32+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
2015-01-31 12:13:32+0100 [scrapy] INFO: Optional features available: ssl, http11
2015-01-31 12:13:32+0100 [scrapy] INFO: Overridden settings: {}
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled item pipelines:
2015-01-31 12:13:32+0100 [blogspider] INFO: Spider opened
2015-01-31 12:13:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:13:32+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-01-31 12:13:32+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2015-01-31 12:14:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:14:48+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 1 times): TCP connection timed out: 60: Operation timed out.
2015-01-31 12:15:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:16:03+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 2 times): TCP connection timed out: 60: Operation timed out.
2015-01-31 12:16:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:17:18+0100 [blogspider] DEBUG: Gave up retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 3 times): TCP connection timed out: 60: Operation timed out.
2015-01-31 12:17:18+0100 [blogspider] ERROR: Error downloading &lt;GET http://blog.scrapinghub.com&gt;: TCP connection timed out: 60: Operation timed out.
2015-01-31 12:17:18+0100 [blogspider] INFO: Closing spider (finished)
2015-01-31 12:17:18+0100 [blogspider] INFO: Dumping Scrapy stats:
    {'downloader/exception_count': 3,
     'downloader/exception_type_count/twisted.internet.error.TCPTimedOutError': 3,
     'downloader/request_bytes': 657,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 3,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2015, 1, 31, 11, 17, 18, 494774),
     'log_count/DEBUG': 5,
     'log_count/ERROR': 1,
     'log_count/INFO': 10,
     'scheduler/dequeued': 3,
     'scheduler/dequeued/memory': 3,
     'scheduler/enqueued': 3,
     'scheduler/enqueued/memory': 3,
     'start_time': datetime.datetime(2015, 1, 31, 11, 13, 32, 955024)}
2015-01-31 12:17:18+0100 [blogspider] INFO: Spider closed (finished)"><pre class="notranslate"><code class="notranslate">$ scrapy runspider myspider.py
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'.  Please install it from &lt;https://pypi.python.org/pypi/service_identity&gt; and make sure all of its dependencies are satisfied.  Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.
2015-01-31 12:13:32+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
2015-01-31 12:13:32+0100 [scrapy] INFO: Optional features available: ssl, http11
2015-01-31 12:13:32+0100 [scrapy] INFO: Overridden settings: {}
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-01-31 12:13:32+0100 [scrapy] INFO: Enabled item pipelines:
2015-01-31 12:13:32+0100 [blogspider] INFO: Spider opened
2015-01-31 12:13:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:13:32+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-01-31 12:13:32+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2015-01-31 12:14:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:14:48+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 1 times): TCP connection timed out: 60: Operation timed out.
2015-01-31 12:15:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:16:03+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 2 times): TCP connection timed out: 60: Operation timed out.
2015-01-31 12:16:32+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:17:18+0100 [blogspider] DEBUG: Gave up retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 3 times): TCP connection timed out: 60: Operation timed out.
2015-01-31 12:17:18+0100 [blogspider] ERROR: Error downloading &lt;GET http://blog.scrapinghub.com&gt;: TCP connection timed out: 60: Operation timed out.
2015-01-31 12:17:18+0100 [blogspider] INFO: Closing spider (finished)
2015-01-31 12:17:18+0100 [blogspider] INFO: Dumping Scrapy stats:
    {'downloader/exception_count': 3,
     'downloader/exception_type_count/twisted.internet.error.TCPTimedOutError': 3,
     'downloader/request_bytes': 657,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 3,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2015, 1, 31, 11, 17, 18, 494774),
     'log_count/DEBUG': 5,
     'log_count/ERROR': 1,
     'log_count/INFO': 10,
     'scheduler/dequeued': 3,
     'scheduler/dequeued/memory': 3,
     'scheduler/enqueued': 3,
     'scheduler/enqueued/memory': 3,
     'start_time': datetime.datetime(2015, 1, 31, 11, 13, 32, 955024)}
2015-01-31 12:17:18+0100 [blogspider] INFO: Spider closed (finished)
</code></pre></div>
<p dir="auto">If I turn off IPv4 completely, then scrapy fails with "No route to host" errors:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ scrapy runspider myspider.py
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'.  Please install it from &lt;https://pypi.python.org/pypi/service_identity&gt; and make sure all of its dependencies are satisfied.  Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.
2015-01-31 12:10:06+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
2015-01-31 12:10:06+0100 [scrapy] INFO: Optional features available: ssl, http11
2015-01-31 12:10:06+0100 [scrapy] INFO: Overridden settings: {}
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled item pipelines:
2015-01-31 12:10:06+0100 [blogspider] INFO: Spider opened
2015-01-31 12:10:06+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:10:06+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-01-31 12:10:06+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2015-01-31 12:10:06+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 1 times): No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 2 times): No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] DEBUG: Gave up retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 3 times): No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] ERROR: Error downloading &lt;GET http://blog.scrapinghub.com&gt;: No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] INFO: Closing spider (finished)
2015-01-31 12:10:06+0100 [blogspider] INFO: Dumping Scrapy stats:
    {'downloader/exception_count': 3,
     'downloader/exception_type_count/twisted.internet.error.NoRouteError': 3,
     'downloader/request_bytes': 657,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 3,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2015, 1, 31, 11, 10, 6, 482287),
     'log_count/DEBUG': 5,
     'log_count/ERROR': 1,
     'log_count/INFO': 7,
     'scheduler/dequeued': 3,
     'scheduler/dequeued/memory': 3,
     'scheduler/enqueued': 3,
     'scheduler/enqueued/memory': 3,
     'start_time': datetime.datetime(2015, 1, 31, 11, 10, 6, 463382)}
2015-01-31 12:10:06+0100 [blogspider] INFO: Spider closed (finished)"><pre class="notranslate"><code class="notranslate">$ scrapy runspider myspider.py
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'.  Please install it from &lt;https://pypi.python.org/pypi/service_identity&gt; and make sure all of its dependencies are satisfied.  Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.
2015-01-31 12:10:06+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
2015-01-31 12:10:06+0100 [scrapy] INFO: Optional features available: ssl, http11
2015-01-31 12:10:06+0100 [scrapy] INFO: Overridden settings: {}
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-01-31 12:10:06+0100 [scrapy] INFO: Enabled item pipelines:
2015-01-31 12:10:06+0100 [blogspider] INFO: Spider opened
2015-01-31 12:10:06+0100 [blogspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-31 12:10:06+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-01-31 12:10:06+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2015-01-31 12:10:06+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 1 times): No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] DEBUG: Retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 2 times): No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] DEBUG: Gave up retrying &lt;GET http://blog.scrapinghub.com&gt; (failed 3 times): No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] ERROR: Error downloading &lt;GET http://blog.scrapinghub.com&gt;: No route to host: 51: Network is unreachable.
2015-01-31 12:10:06+0100 [blogspider] INFO: Closing spider (finished)
2015-01-31 12:10:06+0100 [blogspider] INFO: Dumping Scrapy stats:
    {'downloader/exception_count': 3,
     'downloader/exception_type_count/twisted.internet.error.NoRouteError': 3,
     'downloader/request_bytes': 657,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 3,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2015, 1, 31, 11, 10, 6, 482287),
     'log_count/DEBUG': 5,
     'log_count/ERROR': 1,
     'log_count/INFO': 7,
     'scheduler/dequeued': 3,
     'scheduler/dequeued/memory': 3,
     'scheduler/enqueued': 3,
     'scheduler/enqueued/memory': 3,
     'start_time': datetime.datetime(2015, 1, 31, 11, 10, 6, 463382)}
2015-01-31 12:10:06+0100 [blogspider] INFO: Spider closed (finished)
</code></pre></div>
<p dir="auto">Note that I can open the blog.scrapinghub.com site in Safari, so the target web site does support IPv6 and the problem seems to be on scrapy's side.</p>