How to only print certain text using BeautifulSoup
Question: I am trying to pull some financial data for city governments using
BeautifulSoup (had to convert the files from pdf). I just want to get the data
as a csv file and then I'll analyze it in Excel or SAS. My problem is that I
do not want to print the "& nbsp;" that is in the original HTML, just the
numbers and the row heading. Any suggestions on how I can do this without
using regex?

Below is a sample of the html I am looking at. Next is my code (currently just
in proof of concept mode, need to prove I can get clean data before moving
on). New to Python and programming so any help is appreciated.

    
    
    <TD class="td1629">Investments (Note 2)</TD>
    
    <TD class="td1605">&nbsp;</TD>
    
    <TD class="td479">&nbsp;</TD>
    
    <TD class="td1639">-</TD>
    
    <TD class="td386">&nbsp;</TD>
    
    <TD class="td116">&nbsp;</TD>
    
    <TD class="td1634">2,207,592</TD>
    
    <TD class="td479">&nbsp;</TD>
    
    <TD class="td1605">&nbsp;</TD>
    
    <TD class="td1580">2,207,592</TD>
    
    <TD class="td301">&nbsp;</TD>
    
    <TD class="td388">&nbsp;</TD>
    
    <TD class="td1637">2,882,018</TD>
    

CODE

    
    
    import htmllib
    import urllib
    import urllib2
    import re
    from BeautifulSoup import BeautifulSoup
    
    CAFR = open("C:/Users/snown/Documents/CAFR2004 BFS Statement of Net Assets.html", "r")
    
    soup = BeautifulSoup(CAFR)
    
    assets_table = soup.find(True, id="page_27").find(True, id="id_1").find('table') 
    
    rows = assets_table.findAll('tr')    
    for tr in rows:    
      cols = tr.findAll('td')    
      for td in cols:    
        text = ''.join(td.find(text=True))
        print text+"|",    
      print
    


Answer: 
    soup = BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)
    

It converts `&nbsp;` and other html entities to appropriate characters.

To write it to a csv file:

    
    
    >>> import csv
    >>> import sys
    >>> csv_file = sys.stdout
    >>> writer = csv.writer(csv_file, delimiter="|")
    >>> soup = BeautifulSoup("<tr><td>1<td>&nbsp;<td>3",
    ...                      convertEntities=BeautifulSoup.HTML_ENTITIES)
    >>> writer.writerows([''.join(t.encode('utf-8') for t in td(text=True))
    ...                   for td in tr('td')] for tr in soup('tr'))
    1| |3
    

I've used `t.encode('utf-8')` due to `&nbsp;` is translated to non-ascii
`U+00A0` (no-break space) character.

