I'm getting a recursive error [RuntimeError: maximum recursion depth exceeded while calling a Python object] - but my code is iterative -- or is it?
Question: I'm getting a recursive error:

> RuntimeError: maximum recursion depth exceeded while calling a Python object

But my code is iterative...or is it? I thought it was, based on documentation
(here, for example: <http://www.pythonlearn.com/html-008/cfbook006.html>).
I've been reading on how to change an algorithm/code from recursive to
iterative (e.g., <http://blog.moertel.com/posts/2013-05-11-recursive-to-
iterative.html>) but I just don't see how it is recursive in the first place.

This code goes to a website, does a search and returns about 122 pages of
results. It then clicks through each results page and gathers the links. Then
it is meant to click on each link and scrape the text/html from each one.

The code runs beautifully until it gets to the final for loop: `for url in
article_urls:`. It will capture and store (on dropbox) just over 200 shtml
pages before it returns the error.

The puzzle I'm trying to solve is: how do I avoid getting this error?

Here is the code:

    
    
    
    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    def isReady(browser):
        return browser.execute_script("return document.readyState") == "complete"
    
    def waitUntilReady(browser):
        if not isReady(browser):
            waitUntilReady(browser)
    
    browser = webdriver.Firefox()
    browser.get('http://www.usprwire.com/cgi-bin/news/search.cgi')
    
    # make a search
    query = WebDriverWait(browser, 60).until(EC.presence_of_element_located((By.NAME, "query")))
    query.send_keys('"test"')
    submit = browser.find_element_by_xpath("//input[@value='Search']")
    submit.click()
    numarticles = 0
    
    # grab article urls
    npages = 1
    article_urls = []
    for page in range(1, npages + 1):
        article_urls += [elm.get_attribute("href") for elm in browser.find_elements_by_class_name('category_links')]
        if page <= 121: #click to the next page
            browser.find_element_by_link_text('[>>]').click()
        if page == 122: #last page in search results, so no '[>>]'' to click on. Move on to next steps.
            continue
    
    
    
    # iterate over urls and save the HTML source
    for url in article_urls:
        browser.get(url)
        waitUntilReady(browser)
        numarticles = numarticles+1
        title = browser.current_url.split("/")[-1]
        with open('/Users/My/Dropbox/File/Place/'+str(numarticles)+str(title), 'w') as fw:
            fw.write(browser.page_source.encode('utf-8'))
    

Many thanks in advance for any input.


Answer: Obviously, your `waitUntilReady` goes into infinite recursion, calling itself.

You should change it to something like this:

    
    
    while not isReady(browser):
        time.sleep(1)
    

Waiting for page to completely load in Selenium is not as obvious as it seems,
you can read more in [Harry J.W. Percival's
article](http://www.obeythetestinggoat.com/how-to-get-selenium-to-wait-for-
page-load-after-a-click.html)

