title: Stuck Scraping Multiple Domains sequentially - Python Scrapy
body: <p>I am fairly new to python as well as web scraping. My first project is web scraping random Craiglist cities (5 cities total) under the transportation sub-domain (i.e. <a href="https://dallas.craigslist.org" rel="nofollow noreferrer">https://dallas.craigslist.org</a>), though I am stuck on having to manually run the script per city after manually updating each cities respective domain under the constants >>>> (start_urls = and   absolute_next_url = ) in the script . Is there anyway that I can adjust the script to sequentially run through the cities I have defined (i.e. miami, new york, houston, chicago, etc), and auto-populate the constants for its respective city (start_urls = and  absolute_next_url = ) ?</p>
  
  <p>Also, is there a way to adjust the script to output each city into its own .csv >> (i.e. miami.csv, houston.csv, chicago.csv, etc) ?   </p>
  
  <p>Thank you in advance</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  from scrapy import Request
  
  class JobsSpider(scrapy.Spider):
      name = "jobs"
      allowed_domains = ["craigslist.org"]
      start_urls = ['https://dallas.craigslist.org/d/transportation/search/trp']
  
      def parse(self, response):
          jobs = response.xpath('//p[@class="result-info"]')
  
          for job in jobs:
              listing_title = job.xpath('a/text()').extract_first()
              city = job.xpath('span[@class="result-meta"]/span[@class="result-hood"]/text()').extract_first("")[2:-1]
              job_posting_date = job.xpath('time/@datetime').extract_first()
              job_posting_url = job.xpath('a/@href').extract_first()
              data_id = job.xpath('a/@data-id').extract_first()
  
  
              yield Request(job_posting_url, callback=self.parse_page, meta={'job_posting_url': job_posting_url, 'listing_title': listing_title, 'city':city, 'job_posting_date':job_posting_date, 'data_id':data_id})
  
          relative_next_url = response.xpath('//a[@class="button next"]/@href').extract_first()
          absolute_next_url = "https://dallas.craigslist.org" + relative_next_url
  
          yield Request(absolute_next_url, callback=self.parse)
  
      def parse_page(self, response):
          job_posting_url = response.meta.get('job_posting_url')
          listing_title = response.meta.get('listing_title')
          city = response.meta.get('city')
          job_posting_date = response.meta.get('job_posting_date')
          data_id = response.meta.get('data_id')
  
          description = "".join(line for line in response.xpath('//*[@id="postingbody"]/text()').extract()).strip()
  
          compensation = response.xpath('//p[@class="attrgroup"]/span[1]/b/text()').extract_first()
          employment_type = response.xpath('//p[@class="attrgroup"]/span[2]/b/text()').extract_first()
          latitude = response.xpath('//div/@data-latitude').extract_first()
          longitude = response.xpath('//div/@data-longitude').extract_first()
          posting_id = response.xpath('//p[@class="postinginfo"]/text()').extract()
  
  
          #yield{'job_posting_url': job_posting_url, 'listing_title': listing_title, 'city':city, 'job_posting_date':job_posting_date, 'description':description, #'compensation':compensation, 'employment_type':employment_type, 'posting_id':posting_id, 'longitude':longitude, 'latitude':latitude }
  
          yield{'job_posting_url':job_posting_url,
                        'data_id':data_id,
                  'listing_title':listing_title,
                           'city':city,
                    'description':description,
                   'compensation':compensation,
                'employment_type':employment_type,
                       'latitude':latitude,
                      'longitude':longitude,
               'job_posting_date':job_posting_date,
                     'posting_id':posting_id,
                        'data_id':data_id
                }
  
  </code></pre>