<p dir="auto">when I request <a href="http://tjw_161130192022480.company.qihuiwang.com/" rel="nofollow">http://tjw_161130192022480.company.qihuiwang.com/</a><br>
return  this:    invalid hostname: tjw_161130192022480.company.qihuiwang.com,<br>
I don't know how to solve this question !<br>
hlep me !</p>
<p dir="auto">I have url with invalid hostname - it does not match IDNA standards. Scrapy fails with that.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="scrapy fetch &quot;https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=[%22747190%22]&quot;

2018-07-06 11:53:09 [scrapy.core.scraper] ERROR: Error downloading &lt;GET https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=[%22747190%22]&gt;
Traceback (most recent call last):
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py&quot;, line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py&quot;, line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File &quot;/home/pawel/scrapy/scrapy/core/downloader/middleware.py&quot;, line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File &quot;/home/pawel//scrapy/scrapy/utils/defer.py&quot;, line 45, in mustbe_deferred
    result = f(*args, **kw)
  File &quot;/home/pawel/scrapy/scrapy/core/downloader/handlers/__init__.py&quot;, line 65, in download_request
    return handler.download_request(request, spider)
  File &quot;/home/pawel/scrapy/scrapy/core/downloader/handlers/http11.py&quot;, line 67, in download_request
    return agent.download_request(request)
  File &quot;/home/pawelscrapy/scrapy/core/downloader/handlers/http11.py&quot;, line 331, in download_request
    method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py&quot;, line 1649, in request
    endpoint = self._getEndpoint(parsedURI)
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py&quot;, line 1633, in _getEndpoint
    return self._endpointFactory.endpointForURI(uri)
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py&quot;, line 1510, in endpointForURI
    uri.port)
  File &quot;/home/pawel/scrapy/scrapy/core/downloader/contextfactory.py&quot;, line 59, in creatorForNetloc
    return ScrapyClientTLSOptions(hostname.decode(&quot;ascii&quot;), self.getContext())
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py&quot;, line 1152, in __init__
    self._hostnameBytes = _idnaBytes(hostname)
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/_idna.py&quot;, line 30, in _idnaBytes
    return idna.encode(text)
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.py&quot;, line 355, in encode
    result.append(alabel(label))
  File &quot;/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.py&quot;, line 265, in alabel
    raise IDNAError('The label {0} is not a valid A-label'.format(label))
IDNAError: The label mediaworld_it_api2 is not a valid A-label"><pre class="notranslate"><code class="notranslate">scrapy fetch "https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=[%22747190%22]"

2018-07-06 11:53:09 [scrapy.core.scraper] ERROR: Error downloading &lt;GET https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=[%22747190%22]&gt;
Traceback (most recent call last):
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/pawel/scrapy/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/home/pawel//scrapy/scrapy/utils/defer.py", line 45, in mustbe_deferred
    result = f(*args, **kw)
  File "/home/pawel/scrapy/scrapy/core/downloader/handlers/__init__.py", line 65, in download_request
    return handler.download_request(request, spider)
  File "/home/pawel/scrapy/scrapy/core/downloader/handlers/http11.py", line 67, in download_request
    return agent.download_request(request)
  File "/home/pawelscrapy/scrapy/core/downloader/handlers/http11.py", line 331, in download_request
    method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py", line 1649, in request
    endpoint = self._getEndpoint(parsedURI)
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py", line 1633, in _getEndpoint
    return self._endpointFactory.endpointForURI(uri)
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py", line 1510, in endpointForURI
    uri.port)
  File "/home/pawel/scrapy/scrapy/core/downloader/contextfactory.py", line 59, in creatorForNetloc
    return ScrapyClientTLSOptions(hostname.decode("ascii"), self.getContext())
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1152, in __init__
    self._hostnameBytes = _idnaBytes(hostname)
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/_idna.py", line 30, in _idnaBytes
    return idna.encode(text)
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.py", line 355, in encode
    result.append(alabel(label))
  File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.py", line 265, in alabel
    raise IDNAError('The label {0} is not a valid A-label'.format(label))
IDNAError: The label mediaworld_it_api2 is not a valid A-label
</code></pre></div>
<p dir="auto">IDNA error is legitmate. This url <a href="https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=%5B%22747190%22%5D" rel="nofollow">https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=[%22747190%22]</a> is not valid according to IDNA standard.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [1]: x = &quot;https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=[%22747190%22]&quot;

In [2]: import idna

In [3]: idna.encode(x)
---------------------------------------------------------------------------
IDNAError                                 Traceback (most recent call last)
&lt;ipython-input-3-c97070e17b57&gt; in &lt;module&gt;()
----&gt; 1 idna.encode(x)

/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.pyc in encode(s, strict, uts46, std3_rules, transitional)
    353         trailing_dot = True
    354     for label in labels:
--&gt; 355         result.append(alabel(label))
    356     if trailing_dot:
    357         result.append(b'')

/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.pyc in alabel(label)
    263             ulabel(label)
    264         except IDNAError:
--&gt; 265             raise IDNAError('The label {0} is not a valid A-label'.format(label))
    266         if not valid_label_length(label):
    267             raise IDNAError('Label too long')

IDNAError: The label https://mediaworld_it_api2 is not a valid A-label

"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">1</span>]: <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s">"https://mediaworld_it_api2.frosmo.com/?method=products&amp;products=[%22747190%22]"</span>

<span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">import</span> <span class="pl-s1">idna</span>

<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-s1">idna</span>.<span class="pl-en">encode</span>(<span class="pl-s1">x</span>)
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span>
<span class="pl-v">IDNAError</span>                                 <span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>)
<span class="pl-c1">&lt;</span><span class="pl-s1">ipython</span><span class="pl-c1">-</span><span class="pl-s1">input</span><span class="pl-c1">-</span><span class="pl-c1">3</span><span class="pl-c1">-</span><span class="pl-s1">c97070e17b57</span><span class="pl-c1">&gt;</span> <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>()
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">1</span> <span class="pl-s1">idna</span>.<span class="pl-en">encode</span>(<span class="pl-s1">x</span>)

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pawel</span><span class="pl-c1">/</span>.<span class="pl-s1">virtualenvs</span><span class="pl-c1">/</span><span class="pl-s1">scrapy</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">idna</span><span class="pl-c1">/</span><span class="pl-s1">core</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">encode</span>(<span class="pl-s1">s</span>, <span class="pl-s1">strict</span>, <span class="pl-s1">uts46</span>, <span class="pl-s1">std3_rules</span>, <span class="pl-s1">transitional</span>)
    <span class="pl-c1">353</span>         <span class="pl-s1">trailing_dot</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
    <span class="pl-c1">354</span>     <span class="pl-k">for</span> <span class="pl-s1">label</span> <span class="pl-c1">in</span> <span class="pl-s1">labels</span>:
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">355</span>         <span class="pl-s1">result</span>.<span class="pl-en">append</span>(<span class="pl-en">alabel</span>(<span class="pl-s1">label</span>))
    <span class="pl-c1">356</span>     <span class="pl-k">if</span> <span class="pl-s1">trailing_dot</span>:
    <span class="pl-c1">357</span>         <span class="pl-s1">result</span>.<span class="pl-en">append</span>(<span class="pl-s">b''</span>)

<span class="pl-c1">/</span><span class="pl-s1">home</span><span class="pl-c1">/</span><span class="pl-s1">pawel</span><span class="pl-c1">/</span>.<span class="pl-s1">virtualenvs</span><span class="pl-c1">/</span><span class="pl-s1">scrapy</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">idna</span><span class="pl-c1">/</span><span class="pl-s1">core</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">alabel</span>(<span class="pl-s1">label</span>)
    <span class="pl-c1">263</span>             <span class="pl-s1">ulabel</span>(<span class="pl-s1">label</span>)
    <span class="pl-c1">264</span>         <span class="pl-k">except</span> <span class="pl-v">IDNAError</span>:
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">265</span>             <span class="pl-s1">raise</span> <span class="pl-v">IDNAError</span>(<span class="pl-s">'The label {0} is not a valid A-label'</span>.<span class="pl-en">format</span>(<span class="pl-s1">label</span>))
    <span class="pl-c1">266</span>         <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">valid_label_length</span>(<span class="pl-s1">label</span>):
    <span class="pl-c1">267</span>             <span class="pl-s1">raise</span> <span class="pl-v">IDNAError</span>(<span class="pl-s">'Label too long'</span>)

<span class="pl-v">IDNAError</span>: <span class="pl-v">The</span> <span class="pl-s1">label</span> <span class="pl-s1">https</span>:<span class="pl-c1">//</span><span class="pl-s1">mediaworld_it_api2</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-s1">a</span> <span class="pl-s1">valid</span> <span class="pl-v">A</span><span class="pl-c1">-</span><span class="pl-s1">label</span>
</pre></div>
<p dir="auto">How should scrapy handle this url? Should we download it regardless of validity?</p>