Parsing out data using BeautifulSoup in Python
Question: I am attempting to use BeautifulSoup to parse through a DOM tree and extract
the names of authors. Below is a snippet of HTML to show the structure of the
code I'm going to scrape.

    
    
    <html>
    <body>
    <div class="list-authors">
    <span class="descriptor">Authors:</span> 
    <a href="/find/astro-ph/1/au:+Lin_D/0/1/0/all/0/1">Dacheng Lin</a>, 
    <a href="/find/astro-ph/1/au:+Remillard_R/0/1/0/all/0/1">Ronald A. Remillard</a>, 
    <a href="/find/astro-ph/1/au:+Homan_J/0/1/0/all/0/1">Jeroen Homan</a> 
    </div>
    <div class="list-authors">
    <span class="descriptor">Authors:</span> 
    <a href="/find/astro-ph/1/au:+Kosovichev_A/0/1/0/all/0/1">A.G. Kosovichev</a>
    </div>
    
    <!--There are many other div tags with this structure-->
    </body>
    </html>
    

My point of confusion is that when I do soup.find, it finds the first
occurrence of the div tag that I'm searching for. After that, I search for all
'a' link tags. At this stage, how do I extract the authors names from each of
the link tags and print them out? Is there a way to do it using BeautifulSoup
or do I need to use Regex? How do I continue iterating over every other other
div tag and extract the authors names?

    
    
    import re
    import urllib2,sys
    from BeautifulSoup import BeautifulSoup, NavigableString
    html = urllib2.urlopen(address).read()
        soup = BeautifulSoup(html)
    
        try:
    
            authordiv = soup.find('div', attrs={'class': 'list-authors'})
            links=tds.findAll('a')
    
    
            for link in links:
                print ''.join(link[0].contents)
    
            #Iterate through entire page and print authors
    
    
        except IOError: 
            print 'IO error'
    


Answer: just use findAll for the divs link you do for the links

for authordiv in soup.findAll('div', attrs={'class': 'list-authors'}):

