title: Scrapy returns correct data without going "deeper", but returns duplicates when going "deeper"
body: <p>I am trying to scrape product URL from starting page and after that scrape each product details. For some weird reason, on smaller set of code which basically tells Scrapy to only scrape first "layer", it returns correct data without duplicates, but when I run it on a bigger set of code, it returns duplicates for 1st layer and returns correct data for 2nd layer of web page.</p>
  
  <p>Here is the code for the simplified program:</p>
  
  <pre><code>import scrapy
  from scrapy.linkextractors import LinkExtractor
  
  
  class EglupiV2Spider(scrapy.Spider):
      name = "eGlupi_vTest"
      start_urls = ["https://www.ekupi.hr/hr/Ku%C4%87anski-aparati/Bijela-tehnika/Perilice---su%C5%A1ilice-rublja/c/10128"]
  
      def parse(self, response):
          main_category_selector = response.xpath("//ol[@class='breadcrumb']/li[2]/a/text()").extract_first().strip()
          sub_category_selector = response.xpath("//ol[@class='breadcrumb']/li[3]/a/text()").extract_first().strip()
          sub_sub_category_selector = response.xpath("//ol[@class='breadcrumb']/li[@class='active']/text()").extract_first().strip()
  
          main_product_selector = "//div[@class='product-item']"
  
          product_link_selector = ".//a[@class='thumb']/@href"
  
          productData = {}
  
          for product in response.xpath(main_product_selector):
              productData["mainCategoryName"] = main_category_selector
              productData["subCategoryName"] = sub_category_selector
              productData["subSubCategoryName"] = sub_sub_category_selector
              productData["productLink"] = "https://www.ekupi.hr" + product.xpath(product_link_selector).extract_first()
  
              yield productData
  
              le = LinkExtractor(restrict_xpaths=[
                  "//div[@class='pagination-bar top']/div[@class='pagination-toolbar']/div[@class='sort-refine-bar']/div[@class='row']/div[@class='col-xs-12 col-sm-6 col-md-5 pagination-wrap']/ul[@class='pagination']/li[@class='pagination-next']/a[@class='glyphicon glyphicon-chevron-right']"])
              for link in le.extract_links(response):
                  yield scrapy.Request(link.url, callback=self.parse)
  </code></pre>
  
  <p>Here is the more complex code that goes to 2nd layer and correctly scrapes product details, but returns duplicates for productLink <strong>unlike</strong> in the first program:</p>
  
  <pre><code>import scrapy
  from scrapy.linkextractors import LinkExtractor
  
  
  class EglupiV2Spider(scrapy.Spider):
      name = "eGlupi_v2"
      start_urls = ["https://www.ekupi.hr/hr/Ku%C4%87anski-aparati/Bijela-tehnika/Perilice---su%C5%A1ilice-rublja/c/10128"]
  
      def parse(self, response):
          productData = {}
  
          main_category_selector = response.xpath("//ol[@class='breadcrumb']/li[2]/a/text()").extract_first().strip()
          sub_category_selector = response.xpath("//ol[@class='breadcrumb']/li[3]/a/text()").extract_first().strip()
          sub_sub_category_selector = response.xpath("//ol[@class='breadcrumb']/li[@class='active']/text()").extract_first().strip()
  
          main_product_selector = "//div[@class='product-item']"
  
          product_link_selector = ".//a[@class='thumb']/@href"
  
          for product in response.xpath(main_product_selector):
              productData["mainCategoryName"] = main_category_selector
              productData["subCategoryName"] = sub_category_selector
              productData["subSubCategoryName"] = sub_sub_category_selector
              productData["productLink"] = "https://www.ekupi.hr" + product.xpath(product_link_selector).extract_first()
  
              product_url = product.xpath(product_link_selector).extract_first()
              yield response.follow(product_url, self.parse_product_detail, meta={"productData": productData}, dont_filter=True)
  
          le = LinkExtractor(restrict_xpaths=["//div[@class='pagination-bar bottom']/div[@class='pagination-toolbar']/div[@class='sort-refine-bar']/div[@class='row']/div[@class='col-xs-12 col-sm-6 col-md-5 pagination-wrap']/ul[@class='pagination']/li[@class='pagination-next']/a[@class='glyphicon glyphicon-chevron-right']"])
          for link in le.extract_links(response):
              yield scrapy.Request(link.url, callback=self.parse, dont_filter=True)
  
      def parse_product_detail(self, response):
          product_detail_selector = "//div[@class='product-main-info']"
          table_rows = "//table/tbody"
  
          productData = response.meta["productData"]
  
          for productDetail in response.xpath(product_detail_selector):
              productData["productName"] = productDetail.xpath(".//div[@class='product-details page-title hidden-xs hidden-sm']/div[@class='name']/text()").extract_first().strip()
              productData["productPrice"] = productDetail.xpath(".//dd[@class='final-price']/text()").extract_first().strip()
              productData["productPriceOld"] = productDetail.xpath(".//dd[@class='old-price']/text()").extract_first(default="").strip()
              productData["productWarranty"] = productDetail.xpath(".//div[@class='info-hld']/p/b[contains(text(), 'Jamstvo:')]/text()").extract_first(default="").strip()
              productData["productPaymentWay"] = productDetail.xpath(".//div[@class='info-hld']/p[contains(text(), 'Platite ')]/text()").extract_first(default="").strip()
              productData["productReturn"] = productDetail.xpath(".//div[@class='info-hld']/p[contains(text(), 'Povrat ')]/text()").extract_first(default="").strip()
              productData["productDeliveryDate"] = productDetail.xpath(".//span[@class='ddate']/text()").extract_first(default="").strip()
  
              for table_row in response.xpath(table_rows):
                  first_column = table_row.xpath("./tr/td[1]/text()").extract_first().strip()
                  second_column = table_row.xpath("./tr/td[2]/text()").extract_first().strip()
                  productData[first_column] = second_column
  
          yield productData
  </code></pre>
  
  <p>Here is the example which data is returned when I execute 2nd program. Sorry that I shared the link on Pastebin as SO allows only so much characters. Basically, it returns duplicate values for "<strong>productLink</strong>" Pastebin: <a href="https://pastebin.com/e2wrpfRu" rel="nofollow noreferrer">https://pastebin.com/e2wrpfRu</a></p>