python utf-8 problem
Question: this is my script

    
    
    # -*- coding: utf-8 -*-
    from BeautifulSoup import BeautifulSoup
    import urllib2
    
    res = urllib2.urlopen('http://tazeh.net')
    html = res.read()
    
    soup = BeautifulSoup(''.join(html))
    
    title = soup.findAll('title')
    print title
    

when im am running this script in terminal i get wronged text like that

    
    
    $ python test.py
    
    [<title>ŮžŘ§Ű&OElig;ÚŻŘ§Ů&Dagger; ŘŽŘ¨ŘąŰ&OElig; ŘŞŘ­Ů&bdquo;Ű&OElig;Ů&bdquo;Ű&OElig; ŘŞŘ§Ř˛Ů&Dagger;</title>]
    

this title in utf-8 encoding and persian language

i'm newer in python what's wrong?


Answer: if i add (like one of the comments suggested doing in less useful place):

    
    
    html = html[:10000].decode("utf-8")
    

(the slice is because decode failed at an offset further into the page)

before:

    
    
    soup = BeautifulSoup(html)
    

it prints:

    
    
    [<title>پایگاه خبری تحلیلی تازه</title>]
    

