title: How to web scrape data with Selenium and BeautifulSoup and output to CSV file?
body: <p>I am relatively new to Python and the Stack Overflow community as well. I am using selenium to web scrape <a href="https://freightliner.com/dealer-search/" rel="nofollow noreferrer">https://freightliner.com/dealer-search/</a> for dealership names and addresses in North/South America and have been able to print it as a single string with no problems, but I cannot figure out how to export it to a csv file. The difference between the way that I am printing it in my code and how I want to export it to csv is that I am printing the name and address as a single string delimited by a semicolon whereas I want to export it to a csv as separate columns (name, address). The following is what I have tried:</p>
  <pre><code>'''
  #! python3
  # fl_dealers.py - Scrapes freightliner website for north american locations.
  
  # import selenium
  from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.common.action_chains import ActionChains
  import time, os, csv
  from bs4 import BeautifulSoup
  
  # set Chrome options to automatically download file
  options = webdriver.ChromeOptions()
  prefs = {'download.default_directory': r'C:\Users\username\Downloads\\'}
  options.add_experimental_option('prefs',prefs)
  chromedriver = 'C:/Users/username/chromedriver.exe'
  
  # change directory to Downloads folder
  os.chdir(&quot;C:\\Users\\username\\Downloads&quot;)
  
  # create webdriver object and call Chrome options
  browser = webdriver.Chrome(executable_path=chromedriver, options=options)
  
  # maximize the browser window
  browser.maximize_window()
  
  # set wait time to allow browser to open
  browser.implicitly_wait(10) # seconds
  
  # open freightliner website
  browser.get('https://freightliner.com/dealer-search/')
  
  # maximize the browser window
  browser.maximize_window()
  
  time.sleep(5)
  
  # find all locations in north america
  search = browser.find_element_by_xpath('//*[@id=&quot;by-location&quot;]/div/div/input')
  ActionChains(browser).move_to_element(search).click().key_down(Keys.CONTROL).send_keys('a').key_up(Keys.CONTROL).send_keys(&quot;USA&quot;).perform()
  
  #search.send_keys('USA')
  search_button = browser.find_element_by_xpath('//*[@id=&quot;by-location&quot;]/button').click()
  
  time.sleep(10)
  
  # create variable for webpage AFTER searching for results
  page_source = browser.page_source
  
  # create bs4 object
  soup = BeautifulSoup(page_source, 'html.parser')
  
  # create variables for dealer name and address
  names = soup.find_all('h2')[1:]
  addresses = soup.find_all(class_='address')
  
  # print the names and addresses
  for name, address in zip(names, addresses):
      print(name.get_text(separator=&quot;   &quot;).strip(), &quot;;&quot;, address.get_text(separator=&quot;, &quot;).strip())
      
  with open('fl_dealers.csv', mode='w', newline='') as outputFile:
      dealershipsCSV = csv.writer(outputFile)
      dealershipsCSV.writerow(['name', 'address'])
      
  for name in names:
      dealer_name = name.get_text
      for address in addresses:
          dealer_address = address.get_text
          
  dealershipsCSV.writerow([dealer_name, dealer_address])
  '''
  </code></pre>
  <p>The code does create a CSV file, but it only creates the column headers and does not export any of the actual names and addresses. I have searched numerous stack overflow, github and youtube posts related to the issue, but have not been able to find a solution. I have reached the limit of my knowledge thus far. There is a high likelihood that I am missing something very simply. Alas, I am still new to Python.</p>
  <p>One thing to note - The reasoning for entering &quot;USA&quot; in the search bar is to override the website's default of using my location to search for nearby dealers. Even though the query is for &quot;USA&quot;, it returns all North/South American dealers which is what I want.</p>
  <p>Any and all help is greatly appreciated! Thank you.</p>