I've written a script in python to parse the name, tweets, following and follower of those people available in view all section in my profile page of twitter. My scraper is able to parse those aforesaid fields flawlessly. Any input on the improvement of my parser will be highly appreciated. Here is what I've written:from selenium import webdriverimport timedef browsing_pages():    driver = webdriver.Chrome()    driver.get('https://twitter.com/?lang=en')    driver.find_element_by_xpath('//input[@id=signin-email]').send_keys('username')    driver.find_element_by_xpath('//input[@id=signin-password]').send_keys('password')    driver.find_element_by_xpath('//button[@type=submit]').click()    time.sleep(5)    #Clicking the viewall link    driver.find_element_by_xpath(//small[@class='view-all']//a[contains(@class,'js-view-all-link')]).click()    time.sleep(5)    for links in driver.find_elements_by_xpath(//div[@class='stream-item-header']//a[contains(@class,'js-user-profile-link')]):        scraping_docs(links.get_attribute(href))        #tracking down each profile links under viewall sectiondef scraping_docs(item_link):    driver = webdriver.Chrome()    driver.get(item_link)    # gleaning information of each profile holder    for prof in driver.find_elements_by_xpath(//div[@class='route-profile']):        name = prof.find_elements_by_xpath(.//h1[@class='ProfileHeaderCard-name']//a[contains(@class,'ProfileHeaderCard-nameLink')])[0]        tweet = prof.find_elements_by_xpath(.//span[@class='ProfileNav-value'])[0]        following = prof.find_elements_by_xpath(.//span[@class='ProfileNav-value'])[1]        follower = prof.find_elements_by_xpath(.//span[@class='ProfileNav-value'])[2]        print(name.text, tweet.text, following.text, follower.text)        driver.quit()browsing_pages()
Parser for gleaning data from twitter
python;python 3.x;web scraping;twitter;selenium
I'd focus on 3 main things:don't use time.sleep() to wait for elements on a page. With hardcoded time delays there is a tendency to wait more than actually needed most of the time and less than needed sometimes - not reliable at all. Instead, use Explicit Waits with WebDriverWait class and a set of Expected Conditionsremove the overhead of firing up a separate Chrome instance - collect the links into a list a reuse the same WebDriver instance - you should expect improvements in page load times as well improve your locators - XPath locators are generally the slowest - use by id locators whenever possible; handling class attributes with CSS selectors is more reliable (raw contains() in XPath may generate false positives - it can be a bit better with concat())In the end, you should have something like this:from selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.wait import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECdef browsing_pages(driver):    driver.get('https://twitter.com/login')    wait = WebDriverWait(driver, 10)    driver.find_element_by_css_selector('form.signin input.email-input').send_keys('username')    driver.find_element_by_css_selector('form.signin input.js-password-field').send_keys('password')    driver.find_element_by_css_selector('form.signin button[type=submit]').click()    view_all = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, .view-all a.js-view-all-link)))    view_all.click()    # wait for a profile link to become visible    wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, .stream-item-header a.js-user-profile-link)))    links = [link.get_attribute(href) for link in driver.find_elements_by_css_selector(.stream-item-header a.js-user-profile-link)]    for link in links:        scraping_docs(driver, link)def scraping_docs(driver, item_link):    driver.get(item_link)    # gleaning information of each profile holder    for prof in driver.find_elements_by_css_selector(.route-profile):        name = prof.find_element_by_css_selector(h1.ProfileHeaderCard-name a.ProfileHeaderCard-nameLink)        tweet = prof.find_element_by_css_selector(.ProfileNav-value)        _, following, follower = prof.find_elements_by_css_selector(.ProfileNav-value)[:3]        print(name.text, tweet.text, following.text, follower.text)if __name__ == '__main__':    driver = webdriver.Chrome()    try:        browsing_pages(driver)    finally:        driver.quit()You may also go for a class with the driver instance kept at self.driver, but I think it's pretty much okay to do it this functional way considering that you have only two functions. Good read on the subject:Stop Writing ClassesStart Writing More Classes