title: Scrapy, could not get the next page
body: <p>I'm working on Google search results crawling through this link.</p>
  <p><a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+&quot;model3&quot;+intext:model3&amp;hl=en&amp;rlz</a></p>
  <p>I have disabled Javascript in the Chrome Browser and obtained the xpath value for Next.
  <a href="https://i.stack.imgur.com/WvnHz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WvnHz.png" alt="enter image description here" /></a></p>
  <p>xpath -&gt; //*[@id=&quot;main&quot;]/footer/div<a href="https://i.stack.imgur.com/WvnHz.png" rel="nofollow noreferrer">1</a>/div/div/a</p>
  <p>Here is my code</p>
  <pre><code>import scrapy
  from ..items import GooglescrapyItem
  from urllib.parse import urlparse, parse_qs
  
  
   class GoogleBotsSpider(scrapy.Spider):
      name = 'GoogleScrapyBot'
      allowed_domains = ['google.com']
  
      start_urls = [f'https://www.google.com/search?q=telsa+&quot;model3&quot;+intext:model3&amp;hl=en&amp;rlz']
  
      def parse(self, response):
  
         titles = response.xpath('//*[@id=&quot;main&quot;]/div/div/div/a/h3/div//text()').extract()
         links = response.xpath('//*[@id=&quot;main&quot;]/div/div/div/a/@href').extract()
         next_page = response.xpath('//*[@id=&quot;main&quot;]/footer/div/div/div/a/@href').extract()
         items = []
  
         for idx in range(len(titles)):
            item = GooglescrapyItem()
            item['title'] = titles[idx]
            parsed_url = urlparse(links[idx])
            query_params = parse_qs(parsed_url.query)
            item['link'] = query_params[&quot;q&quot;][0]
            items.append(item)
  
         if next_page:
            next_href = next_page[0]
            next_page_url = 'https://www.google.com/search?q=telsa+&quot;model3&quot;+intext:model3&amp;hl=en&amp;rlz' + next_href
            request = scrapy.Request(url=next_page_url)
            yield request
  
      return items
  </code></pre>
  <p>output</p>
  <p>DEBUG: Crawled (200) &lt;GET <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz%3E" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz&gt;</a> (referer: None)</p>
  <p>DEBUG: Crawled (200) &lt;GET <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LMrhYP3IOY6v0PEPmKGNoAg&amp;start=10&amp;sa=N%3E" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LMrhYP3IOY6v0PEPmKGNoAg&amp;start=10&amp;sa=N&gt;</a> (referer: <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz</a>)</p>
  <p>DEBUG: Crawled (200) &lt;GET <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LsrhYIf-AdSTr7wPtt-LyA4&amp;start=0&amp;sa=N%3E" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LsrhYIf-AdSTr7wPtt-LyA4&amp;start=0&amp;sa=N&gt;</a> (referer: <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LMrhYP3IOY6v0PEPmKGNoAg&amp;start=10&amp;sa=N" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LMrhYP3IOY6v0PEPmKGNoAg&amp;start=10&amp;sa=N</a>)</p>
  <p>DEBUG: Crawled (200) &lt;GET <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=L8rhYJCNCI7_0gSA5qKAAg&amp;start=10&amp;sa=N%3E" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=L8rhYJCNCI7_0gSA5qKAAg&amp;start=10&amp;sa=N&gt;</a> (referer: <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LsrhYIf-AdSTr7wPtt-LyA4&amp;start=0&amp;sa=N" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=LsrhYIf-AdSTr7wPtt-LyA4&amp;start=0&amp;sa=N</a>)</p>
  <p>DEBUG: Crawled (200) &lt;GET <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=MMrhYOLRHeLFmAX2w4ioBA&amp;start=0&amp;sa=N%3E" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=MMrhYOLRHeLFmAX2w4ioBA&amp;start=0&amp;sa=N&gt;</a> (referer: <a href="https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=L8rhYJCNCI7_0gSA5qKAAg&amp;start=10&amp;sa=N" rel="nofollow noreferrer">https://www.google.com/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;rlz/search?q=telsa+%22model3%22+intext:model3&amp;hl=en&amp;ie=UTF-8&amp;ei=L8rhYJCNCI7_0gSA5qKAAg&amp;start=10&amp;sa=N</a>)</p>
  <p>...Skip</p>
  <p>Question</p>
  <p>Google Search Pages will increase by 10 steps, such as &amp;start=0 &amp;start=10 &amp;start=20 &amp;start=30.
  However, my code repeats &amp;start=0 and &amp;start=10 and does not move to &amp;start=20</p>
  <p>Could you please go over my code?</p>