Python Selenium Scrape Hidden Data
Question: I'm trying to scrape the following page (just page 1 for the purpose of this
question):

<https://www.sportstats.ca/display-results.xhtml?raceid=4886>

I can use Selinium to grab the source then parse it, but not all of the data
that I'm looking for is in the source. Some of it needs to be found by
clicking on elements.

For example, for the first person I can get all the visible fields from the
source. But if you click the +, there is more data I'd like to scrape. For
example, the "Chip Time" (01:15:29.9), and also the City (Oakville) that pops
up on the right after clicking the + for a person.

I don't know how to identify the element that needs to be clicked to expand
the +, then even after clicking it, I don't know how to find the values I'm
looking for.

Any tips would be great.


Answer: Here is a sample code for your requirement. This code is base on python ,
selenium with crome exe file.

    
    
                from selenium import webdriver
                from lxml.html import tostring,fromstring
                import time
                import csv
    
                myfile = open('demo_detail.csv', 'wb')
                wr = csv.writer(myfile, quoting=csv.QUOTE_ALL)
                driver=webdriver.Chrome('./chromedriver.exe')
                csv_heading=["","","BIB","NAME","CATEGORY","RANK","GENDER PLACE","CAT. PLACE","GUN TIME","SPLIT NAME","SPLIT DISTANCE","SPLIT TIME","PACE","DISTANCE","RACE TIME","OVERALL (/814)","GENDER (/431)","CATEGORY (/38)","TIME OF DAY"]
                wr.writerow(csv_heading)
                count=0
                try:
                    url="https://www.sportstats.ca/display-results.xhtml?raceid=4886"
                    driver.get(url)
                    table_tr=driver.find_elements_by_xpath("//table[@class='results overview-result']/tbody/tr[@role='row']")
                    for tr in table_tr:
                        lst=[]
                        count=count+1
                        table_td=tr.find_elements_by_tag_name("td")
                        for td in table_td:
                            lst.append(td.text)
    
                        table_td[1].find_element_by_tag_name("div").click()
                        time.sleep(5)
                        table=driver.find_elements_by_xpath("//div[@class='ui-datatable ui-widget']")
                        for demo_tr in driver.find_elements_by_xpath("//tr[@class='ui-expanded-row-content ui-widget-content view-details']/td/div/div/table/tbody/tr"):
                            for demo_td in demo_tr.find_elements_by_tag_name("td"):
                                lst.append(demo_td.text)
                        wr.writerow(lst)
                        table_td[1].find_element_by_tag_name("div").click()
                        time.sleep(5)
                        print count
                    time.sleep(5)
                    driver.quit()
                except Exception as e:
                    print e
                    driver.quit()
    

