title: Scrapy - Scrape multiple URLs using results from the first URL
body: <ol>
  <li>I use Scrapy to scrape data from the first URL.</li>
  <li>The first URL returns a response contains a list of URLs.</li>
  </ol>
  
  <p>So far is ok for me. My question is how can I further scrape this list of URLs? After searching, I know I can return a request in the parse but it seems only can process one URL.</p>
  
  <p>This is my parse:</p>
  
  <pre><code>def parse(self, response):
      # Get the list of URLs, for example:
      list = ["http://a.com", "http://b.com", "http://c.com"]
      return scrapy.Request(list[0])
      # It works, but how can I continue b.com and c.com?
  </code></pre>
  
  <p>May I do something like that?</p>
  
  <pre><code>def parse(self, response):
      # Get the list of URLs, for example:
      list = ["http://a.com", "http://b.com", "http://c.com"]
  
      for link in list:
          scrapy.Request(link)
          # This is wrong, though I need something like this
  </code></pre>
  
  <p>Full version:</p>
  
  <pre><code>import scrapy
  
  class MySpider(scrapy.Spider):
      name = "mySpider"
      allowed_domains = ["x.com"]
      start_urls = ["http://x.com"]
  
      def parse(self, response):
          # Get the list of URLs, for example:
          list = ["http://a.com", "http://b.com", "http://c.com"]
  
          for link in list:
              scrapy.Request(link)
              # This is wrong, though I need something like this
  </code></pre>