title: Scraping Infinite Scrolling Pages with "load more" button using Scrapy
body: <p>How do you scrap a web page with infinite scrolling where the response is html/text instead of json.</p>
  
  <p>My first try was using Rule and LinkExtractor which gets me around 80% of the jobs url</p>
  
  <pre><code>class JobsetSpider(CrawlSpider):
      name = 'test'
      allowed_domains = ['jobs.et']
      start_urls = ['https://jobs.et/jobs/']
  
      rules = (
          Rule(LinkExtractor(allow='https://jobs.et/job/\d+/'), callback='parse_link'),
          Rule(LinkExtractor(), follow=True),
      )
  
      def parse_link(self, response):
          yield {
              'url': response.url
          }
  </code></pre>
  
  <p>My second attempt was to use the example from <a href="https://blog.scrapinghub.com/2016/06/22/scrapy-tips-from-the-pros-june-2016/" rel="noreferrer">SCRAPING INFINITE SCROLLING PAGES</a> but the response is in text/html not json.</p>
  
  <p>When "load more" button clicked, i can see from Network on Chrome Developer tool the request url </p>
  
  <pre><code>https://jobs.et/jobs/?searchId=1509738711.5142&amp;action=search&amp;page=2
  </code></pre>
  
  <p>while the "page" number increase. </p>
  
  <p>My question is</p>
  
  <ol>
  <li>How do i extract the above url from the response header with scrapy when
  the "load more" button is clicked</li>
  <li>Is there a better way to approach this problem? </li>
  </ol>