Scraper to deal with some complicated site with ads I've written a script in Python in combination with Selenium to scrape data from a slightly complicated web page. When the web page is made to open, it comes up with an ad sitting on the data making the browser break unless it is kicked out or the scraper waits a certain time for the ad to disappear. 

I have made the scraper remove it successfully and reach the data as i wanted to. Another problem was to click on the next page button which I've been able to do effectively as well. Now, it is working perfectly along with writing the scraped data in a CSV file. There are around 100 pages the website has traversed. Although it's a very small script to get reviewed, I suppose any clue to make it better might be useful for future reference.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
import csv

driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)

driver.get("https://www.inc.com/inc5000/list/2017")

try:
    wait.until(EC.element_to_be_clickable((By.XPATH, "//span[.='SKIP']"))).click()
except TimeoutException: pass

outfile = open("inc.csv","w",newline='')
writer = csv.writer(outfile)
writer.writerow(["Rank","Company","Growth","Revenue","Industry"])

while True:
    for item in driver.find_elements_by_css_selector("#data-container .row"):
        rank = item.find_elements_by_css_selector(".rank")[0].text
        company = item.find_elements_by_css_selector(".company a")[0].text
        growth =item.find_elements_by_css_selector(".growth")[0].text
        revenue = item.find_elements_by_css_selector(".revenue")[0].text
        industry = item.find_elements_by_css_selector(".industry")[0].text
        writer.writerow([rank,company,growth,revenue,industry])
        print(rank,company,growth,revenue,industry)
    try:
        wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#arrow-r .fa-caret-right"))).click()
    except:break

driver.quit()
There is a room for improvement - like there always is:


avoid bare exception handling. Instead of having an except clause without specifying exception classes to handle, you can handle TimeoutException (which you even already have imported) 
use with context manager when dealing with files
instead of using the find_elements_by_css_selector() method and then getting the first result - just use find_element_by_css_selector() directly
use try/finally to make sure the browser is closed in case of an unexpected exception during the web-scraping
respect PEP8 rules - specifically: spaces between arguments, order of imports
adding couple clarifying comments might help to improve readability
I think you can switch to "by id" locator for the "next" button


Improved version:

import csv

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

driver.get("https://www.inc.com/inc5000/list/2017")

# skip ad popup if present
try:
    wait.until(EC.element_to_be_clickable((By.XPATH, "//span[.='SKIP']"))).click()
except TimeoutException:
    pass

try:
    with open("inc.csv", "w", newline='') as outfile:
        writer = csv.writer(outfile)
        writer.writerow(["Rank", "Company", "Growth", "Revenue", "Industry"])

        while True:
            for row in driver.find_elements_by_css_selector("#data-container .row"):
                rank = row.find_element_by_css_selector(".rank").text
                company = row.find_element_by_css_selector(".company a").text
                growth = row.find_element_by_css_selector(".growth").text
                revenue = row.find_element_by_css_selector(".revenue").text
                industry = row.find_element_by_css_selector(".industry").text

                writer.writerow([rank, company, growth, revenue, industry])
                print(rank, company, growth, revenue, industry)

            # next page
            try:
                wait.until(EC.element_to_be_clickable((By.ID, "arrow-r"))).click()
            except TimeoutException:
                break  # reached the end of the page
finally:
    driver.quit()