title: How to get beautiful soup function to get URLs from next page
body: <p>I have this craigslist aggregator application and I want to get the url link of each listing on the page. </p>
  
  <p>Right now i can get listing title, price, date posted, and distance from zip code from each page in the <code>load_craigslist_url()</code>. The <code>extract_post_urls()</code> returns me a list of urls, but i noticed I was only getting one page's listings URLs not the subsequent pages listings urls.</p>
  
  <p>I put extract_post_urls() function in the function that controls going to next page. I made a list links[] and had <code>links.append(self.extract_post_urls())</code> that gave me the list of URLs, but when I went to the next page, the list was appended but with the urls from the first page not the current page that is being scraped. </p>
  
  <p>I tried <code>.append</code> and <code>.extend</code> and both gave me just lists of urls from the first page not the current page being scraped</p>
  
  <p>Controls going to next page</p>
  
  <pre><code>def load_craigslist_url(self):
  
      links = []
      data = []  # List that will hold all of the information scraped
      self.driver.get(self.url)
      while True:
          try:
              wait = WebDriverWait(self.driver, self.delay)
              wait.until(EC.presence_of_element_located((By.ID,
                                                         "searchform")))  # Once the driver find the web elemet 'searchform' it knows the full page is loaded
              print("Page is loaded")
  
              data.append(self.extract_post_information())  # Append the data found from the extract post method into the data list
              links.append(self.extract_post_urls())
              WebDriverWait(self.driver, 2).until(
                  EC.element_to_be_clickable((By.XPATH,
                                              '//*[@id="searchform"]/div[3]/div[3]/span[2]/a[3]'))).click()  # Wait after clicking 'next page' if there is a next page else break
          except:
              print("Last page")
              break
  
      return data, links
  </code></pre>
  
  <p>Scrapes listings data from current page</p>
  
  <p>Gets url links of each listing on the current page</p>
  
  <pre><code>def extract_post_urls(self):
      url_list = []
      html_page = urllib.request.urlopen(self.url)
      soup = BeautifulSoup(html_page, "lxml")
      for link in soup.findAll("a", {"class": "result-title hdrlnk"}):
          # print(link)
          url_list.append(link["href"])
      return url_list
  </code></pre>
  
  <p>My list looks like </p>
  
  <pre><code>links
                     0: [https://newyork.....]
                     1: [https://newyork.....]
  </code></pre>
  
  <p>just that for as many pages a certain item has on craigslist. the index is equal to the page that was scraped. So page 1 listings URLs are in index 0 and page 2 is in index 1 and so on</p>