title: Scrapy - remove duplicates and output data as a single list?
body: <p>I am using the below code to crawl through multiple links on a page and grab a list of data from each corresponding link:</p>
  
  <pre><code>import scrapy
  
  class testSpider(scrapy.Spider):
      name = "quotes"
      start_urls = ['http://www.website.com']
  
      def parse(self, response):
          urls = response.css('div.subject_wrapper &gt; a::attr(href)').extract()
          for url in urls:
              url = response.urljoin(url)
              yield scrapy.Request(url=url, callback=self.getData)
  
      def getData(self, response):
          data = {'data': response.css('strong.data::text').extract()}
          yield data
  </code></pre>
  
  <p>It works fine, but as it's returning a list of data for each link, when I output to CSV it looks like the following:</p>
  
  <pre><code>"dalegribel,Chad,Ninoovcov,dalegribel,Gotenks,sillydog22"
  
  "kaylachic,jmargerum,kaylachic"
  
  "Kempodancer,doctordbrew,Gotenks,dalegribel"
  
  "Gotenks,dalegribel,jmargerum"
  
  ...
  </code></pre>
  
  <p>Is there any simple/efficient way of outputting the data as a single list of rows without any duplicates (the same data can appear on multiple pages), similar to the following?</p>
  
  <pre><code>dalegribel
  Chad
  Ninoovcov
  Gotenks
  ...
  </code></pre>
  
  <p>I have tried using an array then looping over each element to get an output, but get an error saying yield only supports 'Request, BaseItem, dict or None'. Also, as I would be running this over approx 10k entries, i'm not sure if storing the data in an array would slow the scrape down too much. Thanks.</p>