How to fetch some data conditionally with Python and Beautiful Soup
Question: Sorry if you feel like this has been asked but I have read the related
questions and being quite new to Python I could not find how to write this
request in a clean manner.

For now I have this minimal Python code:

    
    
    from mechanize import Browser
    
    from BeautifulSoup import BeautifulSoup
    import re
    import urllib2
    
    
    
    br = Browser()
    br.open("http://www.atpworldtour.com/Rankings/Singles.aspx")
    
    filename = "rankings.html"
    FILE = open(filename,"w")
    
    html = br.response().read(); 
    
    soup = BeautifulSoup(html);
    links = soup.findAll('a', href=re.compile("Players"));
    for link in links:
        print link['href'];
    
    FILE.writelines(html);
    

It retrieves all the link where the href contains the word player.

Now the HTML I need to parse looks something like this:

    
    
    <tr>
      <td>1</td>
      <td><a href="/Tennis/Players/Top-Players/Roger-Federer.aspx">Federer,&nbsp;Roger</a>&nbsp;(SUI)</td>
      <td><a href="/Tennis/Players/Top-Players/Roger-Federer.aspx?t=rb">10,550</a></td>
      <td>0</td>
      <td><a href="/Tennis/Players/Top-Players/Roger-Federer.aspx?t=pa&m=s">19</a></td>
    </tr>
    

The 1 contains the rank of the player. I would like to be able to retrieve
this data in a dictionary:

  * rank
  * name of the player
  * link to the detailed page (here /Tennis/Players/Top-Players/Roger-Federer.aspx)

Could you give me some pointers or if this is easy enough help me to build the
piece of code ? I am not sure about how to formulate the request in Beautiful
Soup.

Anthony


Answer: Searching for the players using your method will work, but will return 3
results per player. Easier to search for the table itself, and then iterate
over the rows (except the header):

    
    
    table=soup.find('table', 'bioTableAlt')
    for row in table.findAll('tr')[1:]:
        cells = row.findAll('td')
        #retreieve data from cells...
    

To get the data you need:

    
    
        rank = cells[0].string
        player = cells[1].a.string
        link = cells[1].a['href']
    

