Elegant Method for Sleeping with Selenium Webscraper My goal is to scrape the no. 1 suggestion of the suggestion list on https://finance.yahoo.com for any entry. If there is no suggestion the code should write f"nan_{what_ever_the_entry_was}".

As basis I used the solution from https://stackoverflow.com/questions/50355120/how-to-scrape-the-yahoo-finance-search-auto-suggestion-result-with-selenium-pyth/60496531#60496531.

Since I want to scrape the first suggestion for various values in a list, I created a for-loop. If I had kept the third to last line of the original solution

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='yfin-usr-qry']"))).send_keys("goog")


the function won't work properly, because the elements are already loaded for the second loop. That's why I added two time.sleep() blocks instead. Now the function works well.

Question

The time.sleep() part does not seem very elegant to me. Is this the best way or are there more elegant/pythonic ways to resolve the problem?

Input

from typing import List
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument('headless')


def get_ticker_symbols(isin_list: List) -&gt; List:
    driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Program Files (x86)\Google\Chrome\Chromedriver\chromedriver.exe')

    driver.get('https://finance.yahoo.com/')    
    driver.find_element_by_xpath("//button[@type='submit' and @value='agree']").click()

    ticker_list=[]
    for isin in isin_list:
        try:
            WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='yfin-usr-qry']"))).send_keys(f"{isin}")
            time.sleep(1)
            ticker = driver.find_element_by_xpath('(//div[@class="_0ea0377c _4343c2a0 _50f34a35"])[1]').text

        except:
            ticker = f'nan_{isin}'
            time.sleep(0.1)

        finally:
            ticker_list.append(ticker)
            driver.find_element_by_xpath("//input[@name='yfin-usr-qry']").clear()

    return ticker_list

get_ticker_symbols(['GB00BD2ZT390','bmw','dasdas', 'nike', 'GB00BK8FL363'])


Output

['GPH.L', 'BMW.DE', 'nan_dasdas', 'NKE', 'HZD.L']
WebDriverWait

Your intuition about time.sleep() not being ideal here is correct. A better alternative is to use the Selenium WebDriverWait API to wait for elements of interest to appear/disappear. More specifically,


after entering a query in the query field, we want to wait for the list elements to show up in the DOM so we can retrieve the first element
after clearing the query field, we want to wait for any of the aforementioned list elements to disappear from the DOM, as a sanity check to ensure we've reset the page to a state that is ready for our next query


For waiting for the elements to show up in the DOM, we can use presence_of_all_elements_located:

first_element = WebDriverWait(driver, 5).until(
    EC.presence_of_all_elements_located(
        (By.XPATH, "//div[@class='_0ea0377c _4343c2a0 _50f34a35']")
    )
)[0]
ticker = first_element.text


For waiting for any of the above elements to disappear from the DOM, we can use staleness_of or invisibility_of_element_located:

WebDriverWait(driver, 5).until(EC.staleness_of(first_element))




# also works, but we're repeating ourselves a bit here
WebDriverWait(driver, 5).until(
    EC.invisibility_of_element_located(
        (By.XPATH, "//div[@class='_0ea0377c _4343c2a0 _50f34a35']")
    )
)


Query field: retrieve once then reuse


  for isin in isin_list:
    try:
        WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='yfin-usr-qry']"))).send_keys(f"{isin}")



We only need to wait for the search query field to become clickable once. So we can move the WebDriverWait line outside of the loop and save its return value (reference to the query field element) in query_field:

query_field = WebDriverWait(driver, 5).until(
    EC.element_to_be_clickable((By.XPATH, "//input[@name='yfin-usr-qry']"))
)
for query in queries:
    try:
        query_field.send_keys(query)
    # ...


Incorrect assumption about modal always appearing


  driver.find_element_by_xpath("//button[@type='submit' and @value='agree']").click()



This appears to handle a pop-up modal by clicking past it, but I don't think the modal shows up all the time. When I ran the original script the above line threw a NoSuchElementException.

To handle the case when the modal doesn't appear, ignore NoSuchElementException when it is raised from that line:

try:
    driver.find_element_by_xpath(
        "//button[@type='submit' and @value='agree']"
    ).click()
except NoSuchElementException:
    pass


Use None to represent the absence of a value


  If there is no suggestion the code should write f"nan_{what_ever_the_entry_was}".


I would highly recommend against doing this. Instead, use None to represent the absence of a value (see typing.Optional). 

get_ticker_symbols would then return a List[Optional[str]], which is a clear way of expressing in the function signature that it is possible for a search query to return an "empty" result.

As for the current function signature,


  def get_ticker_symbols(isin_list: List) -&gt; List:



this can be improved by specifying the types of elements in each list:

def get_ticker_symbols(queries: List[str]) -&gt; List[Optional[str]]:


Also, isin_list is not a very descriptive name and it almost sounds like a boolean, which is misleading, so I'd go with something like queries instead.

Specify the exceptions you are handling

Avoid using bare except clauses, which catch all exceptions and can obscure/hide real issues or bugs. Instead, specify the exception you are handling:

try:
    # some code that could throw a `TimeoutException`,
    # e.g. `presence_of_all_elements_located`
except TimeoutException:
    # ...


Close resources after use

It's a good idea to call close() or quit() on the WebDriver to exit the browser once we're done using it. That said, if we invoke the WebDriver as a context manager, we don't have to worry about this:

with webdriver.Chrome(options=options) as driver:
    driver.get("https://finance.yahoo.com/")
    # ...


Selenium allows us to invoke the WebDriver as a context manager as shown above, which means we don't need to call driver.close() or driver.quit() explicitly because it will be done for us when the flow of execution leaves the indented block.

Refactored version

#!/usr/bin/env python3

from typing import List, Optional

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait


options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("headless")


def get_ticker_symbols(queries: List[str]) -&gt; List[Optional[str]]:
    with webdriver.Chrome(options=options) as driver:
        driver.get("https://finance.yahoo.com/")

        try:
            driver.find_element_by_xpath(
                "//button[@type='submit' and @value='agree']"
            ).click()
        except NoSuchElementException:
            pass

        query_field = WebDriverWait(driver, 5).until(
            EC.element_to_be_clickable(
                (By.XPATH, "//input[@name='yfin-usr-qry']")
            )
        )
        ticker_list = []
        for query in queries:
            try:
                query_field.send_keys(query)
                first_element = WebDriverWait(driver, 5).until(
                    EC.presence_of_all_elements_located(
                        (
                            By.XPATH,
                            "//div[@class='_0ea0377c _4343c2a0 _50f34a35']",
                        )
                    )
                )[0]
                ticker = first_element.text
            except TimeoutException:
                ticker = None

            ticker_list.append(ticker)
            # web_element.clear() doesn't fire any keyboard events, and
            # as a result the change to the text field isn't registered
            # as we would expect. So as a workaround, we delete the query
            # via Keys.BACKSPACE
            query_field.send_keys(Keys.BACKSPACE * len(query))
            WebDriverWait(driver, 5).until(EC.staleness_of(first_element))

    return ticker_list


if __name__ == "__main__":
    queries = ["GB00BD2ZT390", "bmw", "dasdas", "nike", "GB00BK8FL363"]
    print(get_ticker_symbols(queries))




Alternative solution: Use undocumented API

This is a bit out of scope for this code review, but I do think it deserves a mention. Simply put, there is an even better way of doing all of this without using Selenium or other web scraping libraries.

Using something like Chrome Developer Tools, we can track the XHRs made when typing in the search field and figure out which API call is providing the data we care about.

In this case it's an undocumented API https://query1.finance.yahoo.com/v1/finance/search which we can call directly with requests:

#!/usr/bin/env python3

import requests

from typing import Any, Dict, Optional, List


def get_ticker_symbols(queries: List[str]) -&gt; List[Optional[str]]:
    def get_params(query: str) -&gt; Dict[str, Any]:
        return {
            "q": query,
            "lang": "en-US",
            "region": "US",
            "quotesCount": 1,
            "newsCount": 0,
            "enableFuzzyQuery": False,
        }

    ticker_list = []
    with requests.Session() as session:
        for query in queries:
            response = session.get(
                "https://query1.finance.yahoo.com/v1/finance/search",
                params=get_params(query),
            )
            response.raise_for_status()
            quotes = response.json()["quotes"]
            ticker_symbol = quotes[0]["symbol"] if quotes else None
            ticker_list.append(ticker_symbol)

    return ticker_list


if __name__ == "__main__":
    queries = ["GB00BD2ZT390", "bmw", "dasdas", "nike", "GB00BK8FL363"]
    print(get_ticker_symbols(queries))