title: How to speed up scrapy
body: <p>I need to collect a lot(really a lot) of data for statistics, all the necessary information is in <code>&lt;script type=&quot;application/ld+json&quot;&gt;&lt;/script&gt;</code>
  and I wrote scrapy parser (script inside html) under it, but parsing is very slow (about 3 pages per second). Is there any way to speed up the process? Ideally I would like to see 10+ pages per second</p>
  <p>#spider.py:</p>
  <pre><code>import scrapy
  import json
  
  class Spider(scrapy.Spider):
      name = 'scrape'
      start_urls = [
                   about 10000 urls
                   ]
  
      def parse(self, response):
          data = json.loads(response.css('script[type=&quot;application/ld+json&quot;]::text').extract_first())
          name = data['name']
          image = data['image']
          path = response.css('span[itemprop=&quot;name&quot;]::text').extract()
          yield {
              'name': name,
              'image': image,
              'path': path
          }
          return
  </code></pre>
  <p>#settings.py:</p>
  <pre><code>USER_AGENT = &quot;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0&quot;
  ROBOTSTXT_OBEY = False
  
  CONCURRENT_REQUESTS = 32
  
  DOWNLOAD_DELAY = 0.33
  DEFAULT_REQUEST_HEADERS = {
     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
     'Accept-Language': 'en',
  }
  EXTENSIONS = {
    'scrapy.extensions.telnet.TelnetConsole': None,
  }
  AUTOTHROTTLE_DEBUG = False
  LOG_ENABLED = False
  </code></pre>
  <p>My PC specs:</p>
  <p>16GB ram, i5 2400, ssd, 1gb ethernet</p>
  <p>#Edited</p>