<h2 dir="auto">Summary</h2>
<p dir="auto">When configuring <code class="notranslate">ROBOTSTXT_OBEY = True</code> "delay requirement" <code class="notranslate">Crawl-delay:</code> is not considered.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">We want to fully comply with robots.txt requirements.</p>
<h2 dir="auto">Workaround</h2>
<p dir="auto">Review <code class="notranslate">robots.txt</code> and checking delay config by hand.</p>
<p dir="auto"><a href="http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive" rel="nofollow">Crawl-Delay</a> directive in robots.txt looks useful. If it is present the delay suggested there looks like a good way to adjust crawling rate. When crawling unknown domains it can be better than autothrottle.</p>
<p dir="auto">A draft implementation: <a href="https://gist.github.com/kmike/76aca46cad18915b8695">https://gist.github.com/kmike/76aca46cad18915b8695</a></p>
<p dir="auto">Some notes:</p>
<ul dir="auto">
<li>it may be useful to respect only Crawl-Delay but not User-Agent rules, so this middleware is not inherited from an existing middleware;</li>
<li>Crawl-Delay maps nicely to Scrapy downloader slots; it is possible to implement this feature without per-request download delays (see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="37790671" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/802" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/802/hovercard" href="https://github.com/scrapy/scrapy/issues/802">#802</a>) for which semantics are unclear;</li>
<li>currently RobotsCrawlDelayMiddleware doesn't use User-Agent because UA set by Scrapy is quite arbitrary, but I'm not sure this is a right thing to do;</li>
<li>instead of stdlib robotparser <a href="https://github.com/seomoz/reppy">https://github.com/seomoz/reppy</a> is used; robotparser can't parse robots.txt with Crawl-Delay properly.</li>
</ul>