title: Scrape "next page" in Python
body: <p>I am trying to scrape a web page's next page. They are total 20 pages of this. I want to use first page's url to scrape the next page.</p>
  
  <p>Code:</p>
  
  <pre><code>b=[]
  url="https://abcde.com/cate6-%E7%BE%8E%E5%A6%9D%E4%BF%9D%E9%A4%8A/"
  res=requests.get(url)
  soup = BeautifulSoup(res.text,"lxml")
  b.append(url)
  while True:   
      try:
          dct = {"data-icon":"k"}
          url=soup.find('',dct)
          url=(url['href'])
          print(url)
      except TypeError:   
          break
      if url:
          url=("https://abcde.com"+url)
          print(url)  
          b.append(url) 
  print(b)
  </code></pre>
  
  <p>next page's HTML:</p>
  
  <pre><code>&lt;li class="next"&gt;&lt;a href="https://abcde.com/cate6-%E7%BE%8E%E5%A6%9D%E4%BF%9D%E9%A4%8A/?p=2" data-icon="k"&gt;next page&lt;/a&gt;&lt;/li&gt;
  </code></pre>
  
  <p>last page's HTML:</p>
  
  <pre><code>&lt;li class="next disabled"&gt;&lt;a href="" data-icon="k"&gt;next page&lt;/a&gt;&lt;/li&gt;
  </code></pre>
  
  <p>It only prints out the first page's url.</p>