Python findAll not working on beautifulsoup 3
Question: I am trying to parse a html file and write the results to a csv file. The html
file is:

    
    
    <table BORDER='1' CELLSPACING='0' CELLPADDING='0'>
    <tr>
        <td><small>15</small></td >
        <td><small><small>Cat</small></small></td>
    </tr>
    <tr>
        <td><small><small>16</small></small></td>       
        <td><small><small>&nbsp;</small></small></td>       
    </tr>
    <tr>
        <td><small>17</small></td >
        <td><small><small>Dog</small></small></td>
    </tr>
    </table>
    

and the code I have atm is:

    
    
    import csv
    from BeautifulSoup import BeautifulSoup as bs
    
    soup = bs (open("Animals.html"))
    for i in soup.findAll('small'):
    if "&nbsp;" in i.text:
        i.string = '-'
    
    print soup
    
    f = csv.writer(open("Animals.csv", "a"))   # Open the output file for writing before the loop
    
    trs = soup.findAll('tr')
    
    for tr in trs:
    
        tds = tr.findAll("td")
    
        try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
        id = str(tds[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
        animal = str(tds[1].get_text())
    
    except:
        print "Bad tr string"
        continue #This tells the computer to move on to the next item after it encounters an error
    
    f.writerow([id, animal])
    

When I print out the contents of soup after replacing the `%nbsp;` I get:

    
    
    <table BORDER='1' CELLSPACING='0' CELLPADDING='0'>
    <tr>
        <td><small>15</small></td >
        <td><small></small><small>Cat</small></td > 
    </tr>
    <tr>
        <td><small><small>16</small></small></td >      
        <td><small></small><small>-</small></td >       
    </tr>
    <tr>
        <td><small>17</small></td >
        <td><small></small><small>Dog</small></td >
    </tr>
    </table>
    

But when I look at the .csv file it is empty. However if I change the code to
use BeautifulSoup 4, then I can't replace the `&nbsp;` but the results will be
saved to the .csv file. The other code that I use is:

    
    
    import csv
    from bs4 import BeautifulSoup as bs
    
    soup = bs (open("Animals.html"))
    
    f = csv.writer(open("Animals.csv", "w"))   # Open the output file for writing before the loop
    
    trs = soup.find_all('tr')
    
    for tr in trs:
    
    tds = tr.find_all("td")
    
    try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
        id = str(tds[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
        animal = str(tds[1].get_text())
    
    except:
        print "Bad tr string"
        continue #This tells the computer to move on to the next item after it encounters an error
    
    f.writerow([id, animal])
    

The reason why that one won't do me is because I want the `&nbsp;` to be
replaced with `-` and I haven't been able to get that (the find_all()) to work
with beautifulsoup 4. What is causing the information to be saved to the csv
file and how can I fix it (and/or get it working with beautifulsoup 4)?


Answer: BeautifulSoup 4 will convert that 'nbsp;' into a Unicode character \xa0 when
the 'soup' is constructed. If you search-and-replace on that unicode character
it will work:

    
    
    soup = BeautifulSoup(html)
    
    for i in soup.find_all('small'):
        i.string.replace_with(i.string.replace(u'\xa0', '-'))
    

The syntax there is a little verbose. This is because `i.string` is not a
string, but a `bs4.element.NavigableString`. You can't edit these in place
with a straightforward `i.string.replace(...)`; instead you must call
beautifulsoup's own `replace_with` method.

`replace_with` accepts just one argument, so we have to generate the new
version of the string and pass it in. For this we can use Python's built in
`replace` method for strings, to strip out the u'\xa0' characters and replace
them with '-'

Alternatively, you could just use regular expressions on the original HTML. If
all you need is to replace all `&nbsp;` instances with a `-`:

    
    
    import re
    
    newhtml = re.sub(r'&nbsp;', '-', html)
    

Though you could customise this further so it only affects `<small>` tags -
let me know if you'd like this added to the answer.

