title: scrapy+selenium how to crawl a different page list once i'm done with one?
body: <p>i'm trying to scrape data from an &quot;action/user trading&quot; website, it is in italian so i'll try to be as clear as possible.
  I'm also really new to Python and Scrapy, it is my first project.
  The website has not an easy way to follow links, so i had to come up with a few things.
  First i go to the general list, where all the pages are listed, this is pretty easy as the first page is &quot;https://www.subito.it/annunci-italia/vendita/usato/?o=1&quot; and goes onto &quot;/?o=218776&quot;, i pick the first link of the page and open it with selenium, once here i get the data i need and the click the &quot;next page&quot; button, but here's the tricky part.
  if i go to the same page using the same URL there isn't a &quot;next page&quot; button, it works only if you are first in the list page, and then click on the page link, from here you can now follow the other links.
  i thought it would be done, but i was wrong. the general list is divided in pages (.../?o=1, .../?o=2, etc), each page has an X number of Links (i haven't counted them), when you are on one of the auction pages (coming from the list page so you can use the &quot;next page&quot; button) and you click the &quot;next page&quot; you follow the order of the links in the general list.
  to be clearer if the general list has 200k pages, and each page has 50 links, when you click on the first link of the page you can then click &quot;next page&quot; for 49 times, after that the &quot;next page&quot; button is inactive and you can't go to older link, you must go back to the list and go to the next page, and repeat the process.</p>
  <pre><code>import scrapy
  from scrapy.http import HtmlResponse
  
  from scrapy.selector import Selector
  
  from selenium import webdriver
  
  class NumeriSpider(scrapy.Spider):
  name = &quot;quotes&quot;
  start_urls = [
      'https://www.subito.it/annunci-italia/vendita/usato/?o=41',
  ]
      
  def __init__(self):
      self.driver = webdriver.Firefox()
  
  def parse(self, response):
      self.driver.get(response.url)
      self.driver.find_element_by_xpath('/html/body/div[5]/div/main/div[2]/div[5]/div[2]/div[1]/div[3]/div[1]/a').click()
      while True:
          sel = Selector(text=self.driver.page_source)
          item = {
              'titolo': sel.xpath('//h1[@class= &quot;classes_sbt-text-atom__2GBat classes_token-h4__3_Swu size-normal classes_weight-semibold__1RkLc ad-info__title&quot;]/text()').get(),
              'nome': sel.xpath(&quot;//p[@class='classes_sbt-text-atom__2GBat classes_token-subheading__3yij_ size-normal classes_weight-semibold__1RkLc user-name jsx-1261801758']/text()&quot;).get(),
              'luogo': sel.xpath(&quot;//span[@class='classes_sbt-text-atom__2GBat classes_token-overline__2P5H8 size-normal ad-info__location__text']/text()&quot;).get()
              }
          yield item
          next = self.driver.find_element_by_xpath('/html/body/div[5]/div/main/div[2]/div[1]/section[1]/nav/div/div/button[2]')
          try:
              next.click()
          except:
              driver.quit()
  </code></pre>
  <p>this is the code i wrote with the help of the scrapy docs and many websites/stackoverflow pages.
  i give it the page of the general list to scrape, in this case <code>https://www.subito.it/annunci-italia/vendita/usato/?o=41</code>, it finds the first link of the page (<code>self.driver.find_element_by_xpath('/html/body/div[5]/div/main/div[2]/div[5]/div[2]/div[1]/div[3]/div[1]/a').click()</code>)and then starts getting the data i want. Once it is done it clicks on the &quot;next page&quot; button (<code>next = self.driver.find_element_by_xpath('/html/body/div[5]/div/main/div[2]/div[1]/section[1]/nav/div/div/button[2]')</code>) and repeats the &quot;get data-click next page&quot; process.
  the last item page will have an inactive &quot;next page&quot; button, so at the moment the crawler is stuck, i manually close the browser, edit with notepad++ the &quot;start_urls&quot; link to be the page after the one i've just scraped, and run the crawler again to scrape this page.</p>
  <p>I'd like it to be fully automatic, so i can leave it do its thing for hours (i'm saving the data in a json file atm).</p>
  <p>the &quot;inactive&quot; next-page button is different by the active one only by a <code>disabled=&quot;&quot;</code> attribute, how do i detect that? and once detected, how do i tell the crawler to go back to list page plus 1 and do again the data scraping process?
  My issue is only detecting that inactive button and make a loop that adds 1 to the list page i gave(if i start with the link &quot;<code>https://www.subito.it/annunci-italia/vendita/usato/?o=1</code>&quot; it should then go to &quot;<code>https://www.subito.it/annunci-italia/vendita/usato/?o=2</code>&quot; and do the same thing)</p>