python urllib2 can't get correct webpage
Question: I'm writting a spider using urllib2 and beautifulsoup. But I come across some
problems.

  1. I can't download the webpage correctly. I tried 

`GET 'http://thesite.html'`,

`wget 'http://thesite.html'`,

`curl -O 'http://thesite.html'`,

on linux terminal, but got a lot mass, seems like wrong codec.

  2. I then tried using `file_get_contents('http://thesite.html')` and also can't get the correct webpage.

  3. Then I tried `urllib2.urlopen('http://thesite.html')`, can't work. 

  4. can't detect the encoding. ` s = urllib2.urlopen('http://thesite.html') print chardet.detect(s) ` and outputs `{'confidence':0.0, 'encoding':None}`
  5. I also tried python3 with urllib.request, I can get a byte string, but when I try to decode this bytes to utf-8, I got a error message.

Anyone can help me about this? How to get the correct webpage like a web
browser.


Answer: What are you downloading? Is it text, or something binary, e.g. an image?

A binary file might explain why standard tools such as wget and curl return a
lot of "mass" (mess?), and `chardet.detect()` will return `{'confidence': 0.0,
'encoding': None}` in that case.

    
    
    >>> import urllib2
    >>> import chardet
    
    >>> s = urllib2.urlopen('http://i.stack.imgur.com/uIM9Q.png?s=32&g=1').read()    # your avatar
    >>> chardet.detect(s)
    {'confidence': 0.0, 'encoding': None}
    

What does the byte string that you got in Python 3 look like? Otherwise, if
you are too coy to post the URL, traceback, or other details, perhaps you can
post a little of that string?

