title: Scrapy multiple domains and start urls
body: <p>I have a simple <code>Sracpy</code> spider that exports all pages in a domain to an individual csv file. Most people recommend writing a different spider for each site, but given how simple the information is that I am requesting I thought it made sense to figure out how to iterate through a list of domains. There will eventually be thousands of domains I want to get links from, all with very different structures so I would like the spider to scale. </p>
  
  <p>Here are a couple lines from the csv the spider is pulling domains from:</p>
  
  <p><a href="https://i.stack.imgur.com/fw1No.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fw1No.png" alt="enter image description here"></a></p>
  
  <p>Here is my most recent attempt:</p>
  
  <pre><code>import scrapy
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  from NONPROF.items import NonprofItem
  from scrapy.http import Request
  import pandas as pd
  
  file_path = 'C:/listofdomains.csv'
  open_list = pd.read_csv(file_path)
  urlorgs = open_list.urls.tolist()
  
  open_list2 = pd.read_csv(file_path)
  domainorgs = open_list2.domain.tolist()
  
  class Nonprof(CrawlSpider):
      name = "responselist"
      allowed_domains = domainorgs
      start_urls = urlorgs
  
      rules = [
          Rule(LinkExtractor(
              allow=['.*']),
               callback='parse_item',
               follow=True)
          ]
  
      def parse_item (self, response):
          item = NonprofItem()
          item['responseurl'] = response.url
          yield item
  </code></pre>
  
  <p>I don't get an apparent errors when running the spider but it doesn't appear to be yielding anything: <code>[scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)</code></p>
  
  <p>So I have a few questions:
  1. Is scrapy capable of processing a request like this? 
  2. Is there a better way to get the spider to iterate through a list of domains and matching start_url?</p>