title: Scrapy item processing blocking new requests in broad crawl
body: <p>I'm currently building a Scrapy script that performs a broad crawl with the following settings:</p>
  <pre><code>'CONCURRENT_REQUESTS': 600,
  'CONCURRENT_REQUESTS_PER_DOMAIN': 25,
  'CONCURRENT_ITEMS': 250,
  'REACTOR_THREADPOOL_MAXSIZE': 600,
  </code></pre>
  <p>To perform the broad crawl I loop through a list of domains here in my spider like this:</p>
  <pre><code>    def start_requests(self):
          for domain in self.domains:
              yield scrapy.Request(&quot;https://www.&quot; + domain['domain'], self.parse, meta={'domain':domain})
  </code></pre>
  <p>The loop works well, making more than 10 calls per second.</p>
  <p>I noticed however that my entire <strong>scraping</strong> slows down as soon as I send my first items to be processed in my pipeline (which performs simple loops and conditional checks).</p>
  <p>The issue seem to be that the loop in start_requests() waits for currently processed items to finish, before continuing my broad crawl for new domains, and this despite being way below my 'CONCURRENT_REQUESTS' limit.</p>
  <p>Result is a script that runs 2 to 3x slower.</p>
  <p>Isn't the item processing supposed to not block the requests?</p>