title: Limiting number of Job Posts under <Li> tags Collected through Selenium Web Scraping
body: <p>I am currently working on a web scraping script to pull job posting data from LinkedIn. I have managed to get nearly everything to work except for a way to limit the number of job posts to be included in my final data frame. I am trying to find a way to include something like a while or if statement to set a specific number of job posts to be included (Say like 200, 300, 400 etc.). My script will pull all the jobs from the search and put them into a dataframe, but when I try to add something like a while loop/if statement, it only returns an empty dataframe. I know this has something to do with the way I am gathering the job list (I just included everything in the job results by using driver.find_element_by_class_name('jobs-search__results-list') and pulled all the job postings from that by using find element by tag name (li) ) Below is my current script that pulls all the job postings along with my attempted while statement that I commented out. My knowledge of these kinds of statements is limited, so what I am trying may not even be possible, so any kind of suggestion on what I could try to incorporate to improve my script is greatly appreciated. There are a few excess imports at the top as I am troubleshooting only this part of my script and those imports were for later on in the script</p>
  <pre><code>from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  import time
  import pandas as pd
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.by import By
  from selenium.common.exceptions import NoSuchElementException
  
  options = Options()
  options.add_argument(&quot;window-size=1400,1400&quot;)
  
  pd.set_option('display.max_columns', None)
  pd.set_option('display.width', None)
  pd.set_option('display.max_colwidth', None)
  
  PATH = &quot;C://Users//dcfitzsi//Downloads//chromedriver_win32/chromedriver.exe&quot;
  driver = webdriver.Chrome(PATH)
  
  url = 'https://www.linkedin.com/jobs/search? 
  keywords=Chemical%20Engineer&amp;location=Arizona&amp;geoId=&amp;trk=public_jobs_jobs-search-bar_search- 
  submit&amp;position=1&amp;pageNum=0'
  
  driver.maximize_window()
  
  driver.get(url)
  
  no_of_jobs = int(driver.find_element_by_xpath('/html/body/div[1]/div/main/div/h1/span[1]').get_attribute('innerText'))
  
  job_id = []
  job_title = []
  company_name = []
  location = []
  date = []
  job_link = []
  
  
  i = 2
  while i &lt;= int(no_of_jobs/25)+1:
      driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')
      i = i + 1
      try:
          driver.find_element_by_xpath('/html/body/div[1]/div/main/section[2]/button').click()
          time.sleep(5)
      except:
          pass
          time.sleep(5)
  
  job_lists = driver.find_element_by_class_name('jobs-search__results-list')
  jobs = job_lists.find_elements_by_tag_name('li')
  print(len(jobs))
  
  #while int(len(jobs)) &lt;= 26:
  for job in jobs:
          job_id0 = job.get_attribute('data-id')
          job_id.append(job_id0)
  
          job_title0 = job.find_element_by_css_selector('h3').get_attribute('innerText')
          job_title.append(job_title0)
  
          company_name0 = job.find_element_by_css_selector('h4').get_attribute('innerText')
          company_name.append(company_name0)
  
          location0 = job.find_element_by_css_selector('[class=&quot;job-search-card__location&quot;]').get_attribute('innerText')
          location.append(location0)
  
          date0 = job.find_element_by_css_selector('div&gt;div&gt;time').get_attribute('datetime')
          date.append(date0)
  
          job_link0 = job.find_element_by_css_selector('a').get_attribute('href')
          job_link.append(job_link0)
  
  job_data = pd.DataFrame({'ID': job_id,
                           'Job Title': job_title,
                           'Company Name': company_name,
                           'Location': location,
                           'Date': date,
                           'Job Link': job_link,
                           })
  print(job_data)
  job_data.to_csv('test_scrape.csv', index=False, encoding='utf-8')
  </code></pre>