python won't accept the value of a variable
Question: This runs and scrapes links exactly how I want it except python doesn't
recognize the value of "scraped_pages" when I run it in the terminal scraped
pages will increment by 1 every loop but it just carries on when the integer
is higher than "page_nums". when I set "page_nums" to an integer below 5 it
will run and stop at 5 but any more and it will crash. I apologize if I've not
worded the question the best I've been up all night on this. All code above
this is working this is the problem code. All modules are imported correctly
also. It uses selenium, I'm not sure the explicit wait is working since it
crashes before it will reach the "page_nums" value.

    
    
    page_nums = raw_input("how many pages to scrape?: ")
    urls_list = []
    scraped_pages = 0
    scraped_links = 0
    
    while scraped_pages <= page_nums:
        for li in list_items:
            for a in li.find_all('a', href=True):
                url = a['href']
                if slicer(url,'http'):
                    url1 = slicer(url,'http')
                    urls_list.append(url1)
                    scraped_links += 1
                elif slicer(url,'www'):
                    url1 = slicer(url,'www')
                    urls_list.append(url1)
                    scraped_links += 1
                else:
                    pass
        scraped_pages += 1
        WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, "/html/body/div[5]/div[4]/div[9]/div[1]/div[3]/div/div[5]/div/span[1]/div/table/tbody/tr/td[12]")))
        driver.find_element_by_xpath("/html/body/div[5]/div[4]/div[9]/div[1]/div[3]/div/div[5]/div/span[1]/div/table/tbody/tr/td[12]").click()
        print scraped_links
    print urls_list
    

Here is part of the error returned.

    
    
    1
    2
    Traceback (most recent call last):
      File "google page click 2.py", line 51, in <module>
       driver.find_element_by_xpath("/html/body/div[5]/div[4]/div[9]/div[1]/div[3]/div/div[5]/div/span[1]/div/table/tbody/tr/td[12]").click()
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 75, in click
        self._execute(Command.CLICK_ELEMENT)
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 454, in _execute
        return self._parent.execute(command, params)
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 201, in execute
        self.error_handler.check_response(response)
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 181, in check_response
        raise exception_class(message, screen, stacktrace)
    selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
    Stacktrace:
        at fxdriver.preconditions.visible (file:///tmp/tmpzSHEeb/extensions/fxdriver@googlecode.com/components/command-processor.js:9981)
        at DelayedCommand.prototype.checkPreconditions_ (file:///tmp/tmpzSHEeb/extensions/fxdriver@googlecode.com/components/command-processor.js:12517)
        at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpzSHEeb/extensions/fxdriver@googlecode.com/components/command-processor.js:12534)
        at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpzSHEeb/extensions/fxdriver@googlecode.com/components/command-processor.js:12539)
        at DelayedCommand.prototype.execute/< (file:///tmp/tmpzSHEeb/extensions/fxdriver@googlecode.com/components/command-processor.js:12481)
    


Answer: This problem is because Selenium has successfully loaded a webpage, but the
button you are trying to click is 'not in view'. There are two likely
scenarios:

**1.** If, for some reason, the element is hidden (ie hidden via CSS), then
you will not be able to click it.

**2.** The element might not be visible because it is in part of the webpage
that you have to scroll to view. To circumvent this issue you can use:

    
    
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html/body/div[5]/div[4]/div[9]/div[1]/div[3]/div/div[5]/div/span[1]/div/table/tbody/tr/td[12]")))
    element_to_click = driver.find_element_by_xpath("/html/body/div[5]/div[4]/div[9]/div[1]/div[3]/div/div[5]/div/span[1]/div/table/tbody/tr/td[12]")
    driver.execute_script('return arguments[0].scrollIntoView();', element_to_click)
    element_to_click.click()
    

The third line executes some JavaScript which scrolls the element into view
before you ask Selenium to click it. If you are using this script on
hundreds/thousands of webpages then you may want to put use `time.sleep(1)`
before `element_to_click.click()`, thus giving the browser time to scroll the
element into view before attempting to click it.

As an aside, you should do a little reading on XPaths - it is a lot easier to
try and select elements by `id` or `class` name eg:

    
    
    //div[@id="some-id"]
    

Selects the element in the document with `id="some-id"`.

