utf8' codec can't decode byte 0x96 in python
<p>I am trying to check if a certain word is on a page for many sites. The script runs fine for say 15 sites and then it stops. </p>

<p><strong>UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 15344: invalid start byte</strong></p>

<p>I did a search on stackoverflow and found many issues on it but I can't seem to understand what went wrong in my case. </p>

<p>I would like to either solve it or if there is an error skip that site. Pls advice how I can do this as I am new and the below code itself has taken me a day to write. By the way the site which the script halted on was <a href="http://www.homestead.com" rel="nofollow">http://www.homestead.com</a></p>

<pre><code>filetocheck = open("bloglistforcommenting","r")
resultfile = open("finalfile","w")

for countofsites in filetocheck.readlines():
        sitename = countofsites.strip()
        htmlfile = urllib.urlopen(sitename)
        page = htmlfile.read().decode('utf8')
        match = re.search("Enter your name", page)
        if match:
            print "match found  : " + sitename
            resultfile.write(sitename+"\n")

        else:
            print "sorry did not find the pattern " +sitename

print "Finished Operations"
</code></pre>

<p>As per Mark's comments I changed the code to implement beautifulsoup</p>

<pre><code>htmlfile = urllib.urlopen("http://www.homestead.com")
page = BeautifulSoup((''.join(htmlfile)))
print page.prettify() 
</code></pre>

<p>now I am getting this error</p>

<pre><code>page = BeautifulSoup((''.join(htmlfile)))
TypeError: 'module' object is not callable
</code></pre>

<p>I am trying their quick start example from <a href="http://www.crummy.com/software/BeautifulSoup/documentation.html#Quick%20Start" rel="nofollow">http://www.crummy.com/software/BeautifulSoup/documentation.html#Quick%20Start</a>. If I copy paste it then the code works fine. </p>

<p><strong>I FINALLY got it to work. Thank you all for your help. Here is the final code.</strong></p>

<pre><code>import urllib
import re
from BeautifulSoup import BeautifulSoup

filetocheck = open("listfile","r")

resultfile = open("finalfile","w")
error ="for errors"

for countofsites in filetocheck.readlines():
        sitename = countofsites.strip()
        htmlfile = urllib.urlopen(sitename)
        page = BeautifulSoup((''.join(htmlfile)))  
        pagetwo =str(page) 
        match = re.search("Enter YourName", pagetwo)
        if match:
            print "match found  : " + sitename
            resultfile.write(sitename+"\n")

        else:
            print "sorry did not find the pattern " +sitename

print "Finished Operations"
</code></pre>
