Nested Tags/Table in BeautifulSoup Python scraping
Question: I've pored over Google for half a day looking for the right answer to this.
The closest thing I've come to is this StackOverflow post: [Nested tags in
BeautifulSoup - Python](http://stackoverflow.com/questions/15749354/nested-
tags-in-beautifulsoup-python)

Effectively I'm scraping wait time data from a complex page with nested
elements using BeautifulSoup in Python. Some of the HTML elements have
classes/ids, but most do not. Looking at the DOM I can see the path to the
elements I want. I've written a preliminary script that points to the right
path (...I think) but the console keeps printing out an empty array. Even
changing this code to print out something simple (like soup.select('body h2'))
doesn't print anything. Here is my code

    
    
    from BeautifulSoup import BeautifulSoup
    import requests
    
    url = 'http://www.alexianbrothershealth.org/wait-times'
    r = requests.get(url)
    
    soup = BeautifulSoup(r.text)
    wait_times = soup.select('body div div div div div div div table tbody tr td')
    
    print wait_times
    

Any ideas what I need to change to make this work? I have many more sites to
get to, so figuring out the right syntax for the .select() pointer would
really be helpful. I've tried using lxml with XPath and that also prints out
an empty array. The page source tells me that it's in the HTML and not loading
via javascript on the client so I should be okay there.

PS I'm a newb, so any complex answer will be entirely lost on me ;)


Answer: I don't think select is the BeautifulSoup method you're looking for. Select
uses css selectors, but you're just looking for the right set of tags.

If all the times you're looking for are in 's, I'd use

    
    
    tds = soup.find_all("td")
    for cell in tds:
        children = cell.findChildren()
        ... do actual work ...
    

Alternatively, if you want to use select (which you definitely can), try
removing the whole first set of tags:

    
    
    soup.select("table tbody tr td")
    

works fine.

