title: How to Retrieve Dynamic Link from Economic-Times using Scrapy
body: <p>The parse_new_items function doesn't retrieve the link, I think the link is being generated dynamically hence the problem. I have looked at other posts but I am not able to solve this problem. Any help will be much appreciated.</p>
  <pre><code>import scrapy
  import json
  from scrapy.crawler import CrawlerProcess
  
  
  class EtSpider(scrapy.Spider):
      name = 'et'
      start_urls = [&quot;https://economictimes.indiatimes.com/archive.cms&quot;]
  
      def parse(self, response):
          months = response.xpath('//table//tr//a/@href').re(r'/archive/year-\d+,month-\d+.cms')
          for month in months:
              month = 'https://economictimes.indiatimes.com' + month  
              yield scrapy.Request(month, self.parse_news_item)
              
      def parse_news_item(self, response):
          days = response.xpath('//table//tr//td//tbody//tr//td//a/@href').re(r'/archivelist/year-\d+,month-\d+,starttime-\d+.cms')
          for day in days:
              self.logger.info(day)
          
  process = CrawlerProcess()
  process.crawl(EtSpider)
  process.start()
  </code></pre>