Scraping Multiple Tables with Beautiful Soup
Question: I'm a python newbie trying to scrape tables with Beautiful Soup.

I want to scrape tables similar to the one below with Ubuntu CVE information
and output the table to a csv document.

    
    
    <div class="pkg">
    <div class="field">Package</div><div class="value">Source: <a href="http://people.canonical.com/~ubuntu-security/cve/pkg/mysql-cluster-7.0.html">mysql-cluster-7.0</a></div>
    <table>
    <tr><td>(Lucid Lynx):</td><td>ignored
    </td></tr>
    <tr><td>(Precise Pangolin):</td><td>DNE
    </td></tr>
    <tr><td>(Quantal Quetzal):</td><td>DNE
    </td></tr>
    <tr><td>(Raring Ringtail):</td><td>DNE
    </td></tr>
    <tr><td>(Saucy Salamander):</td><td>DNE
    </td></tr>
    </table>
    </div>
    <div class="pkg">
    <div class="field">Package</div><div class="value"><a href="http://people.canonical.com/~ubuntu-security/cve/pkg/mysql-5.5.html">mysql-5.5</a></div>
    <table>
    <tr><td>(Lucid Lynx):</td><td>DNE
    </td></tr>
    <tr><td>(Precise Pangolin):</td><td>released
    (5.5.32-0ubuntu0.12.04.1)
    </td></tr>
    <tr><td>(Quantal Quetzal):</td><td>released
    (5.5.32-0ubuntu0.12.10.1)
    </td></tr>
    <tr><td>(Raring Ringtail):</td><td>released
    (5.5.32-0ubuntu0.13.04.1)
    </td></tr>
    <tr><td>(Saucy Salamander):</td><td>released
    (5.5.32-0ubuntu1)
    </td></tr>
    </table>
    </div>
    

I want the csv output to be in this format:

  1. release, status
  2. (Lucid Lynx),ignored 
  3. (Precise Pangolin),DNE
  4. etc....

I can pull all the tables with `table = soup.findAll("table")`But I'm unsure
of how to split the contents of all the table into two different cells onto a
spreadsheet.

Any advice would be much appreciated.


Answer: First, your file appears to be valid XML, so you should just use the normal
XML parsing in python. Beautiful Soup is not needed at all.

That said, you will really want to find_all rows, loop over them, and join the
columns.

    
    
    import csv
    with open('some.csv', 'rb') as f:
        writer = csv.writer(f)
        writer.writerow(['release', 'status'])
        for row in table.find_all('tr'):
            writer.writerow(col.string for col in row.find_all('td'))
    

