Get all td content inside tbody of tr in python using lxml
Question: I am getting values of header of html table below using lxml but when I am
trying to parse the contents of the td's inside tr which is in tbody using
xpath its giving me empty value because the data is generated dynamically.
Below is my python code with its output value I am getting. How can I get the
values?

    
    
    <table id="datatabl" class="display compact cell-border dataTable no-footer" role="grid" aria-describedby="datatabl_info">
         <thead>
            <tr role="row">
                  <th class="dweek sorting_desc" tabindex="0" aria-controls="datatabl" rowspan="1" colspan="1" style="width: 106px;" aria-label="Week: activate to sort column ascending" aria-sort="descending">Week</th>
       <th class="dnone sorting" tabindex="0" aria-controls="datatabl" rowspan="1" colspan="1" style="width: 100px;" aria-label="None: activate to sort column ascending">None</th>
            </tr>
         </thead>
    
    
         <tbody>
            <tr class="odd" role="row">
                <td class="sorting_1">2016-05-03</td>
                <td>4.27</td>
                <td>21.04</td>
            </tr>
           <tr class="even" role="row">
                <td class="sorting_1">2016-04-26</td>
                <td>4.24</td>
                <td>95.76</td>
               <td>21.04</td>
           </tr>
           </tbody>
    

My Python code

    
    
       from lxml import etree
       import urllib
    
       web = urllib.urlopen("http://droughtmonitor.unl.edu/MapsAndData/DataTables.aspx")
       s = web.read()
    
       html = etree.HTML(s)
    
       ## Get all 'tr'
       tr_nodes = html.xpath('//table[@id="datatabl"]/thead')
       print tr_nodes
    
       ## 'th' is inside first 'tr'
       header = [i[0].text for i in tr_nodes[0].xpath("tr")]
       print header
    
       ## tbody
       tr_nodes_content = html.xpath('//table[@id="datatabl"]/tbody')
       print tr_nodes_content
    
       td_content = [[td[0].text for td in tr.xpath('td')] for tr in tr_nodes_content[0]]
       print td_content
    

output in terminal:

    
    
        [<Element thead at 0xb6b250ac>]
        ['Week']
       [<Element tbody at 0xb6ad20cc>]
        []
    


Answer: The data is dynamically loaded from the
`http://droughtmonitor.unl.edu/Ajax.aspx/ReturnTabularDM` endpoint. One option
would be to try to mimic that request and get the data from the JSON response.

Or, you can stay on a high-level and solve it via
[`selenium`](http://selenium-python.readthedocs.io/):

    
    
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    
    driver = webdriver.Firefox()
    driver.maximize_window()
    
    wait = WebDriverWait(driver, 10)
    
    url = 'http://droughtmonitor.unl.edu/MapsAndData/DataTables.aspx'
    driver.get(url)
    
    # wait for the table to load
    wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table#datatabl tr[role=row]")))
    rows = driver.find_elements_by_css_selector("table#datatabl tr[role=row]")[1:]
    
    for row in rows:
        cells = row.find_elements_by_tag_name("td")
        print(cells[2].text)
    
    driver.close()
    

Prints the contents of the D0-D4 column:

    
    
    33.89
    39.64
    39.28
    39.20
    ...
    36.74
    38.45
    43.61
    

