Python: difficulty converting ascii to unicode
Question: My goal: get the page source from a url and count all instances of a keyword
within that page source

How I am doing it: getting the pagesource via urllib2, looping through each
char of the page source and comparing it to the keyword

My problem: my keyword is encoded in utf-8 while the page source is in
ascii... I am running into errors whenever I try conversions.

getting the page source:

    
    
    import urllib2
    response = urllib2.urlopen(myUrl)
    return response.read()
    

comparing page source and keyword:

    
    
    pageSource[i] == keyWord[j]
    

I need to convert one of these strings to the other's encoding. Intuitively I
felt that ascii (the page source) to utf-8 (the key word) would be the best
and easiest, so:

    
    
        pageSource = unicode(pageSource)
    UnicodeDecodeError: 'ascii' codec can't decode byte __ in position __: ordinal not in range(128)
    


Answer: When trying to work with text, don't leave your data as byte strings. Decode
to Unicode early, encode back to bytes as late as possible.

Decode your downloaded network data:

    
    
    import urllib2
    
    response = urllib2.urlopen(myUrl)
    # Latin-1 is the default for HTTP text/ responses, adjust as needed
    codec = response.info().getparam('charset', 'latin1')
    return response.read().decode(codec)
    

and do the same for your `keyWord` data. If it is encoded as UTF-8, decode it
as such, or use Unicode string literals.

You may want to read up on Python and Unicode:

  * [The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)](http://joelonsoftware.com/articles/Unicode.html) by Joel Spolsky

  * [Pragmatic Unicode](http://nedbatchelder.com/text/unipain.html) by Ned Batchelder

  * The [Python Unicode HOWTO](http://docs.python.org/2/howto/unicode.html)

