title: python urllib2 can't get correct webpage
body: <p>I'm writting a spider using urllib2 and beautifulsoup. But I come across some problems.</p>
  
  <ol>
  <li><p>I can't download the webpage correctly. I tried </p>
  
  <p><code>GET 'http://thesite.html'</code>, </p>
  
  <p><code>wget 'http://thesite.html'</code>, </p>
  
  <p><code>curl -O 'http://thesite.html'</code>,</p></li>
  </ol>
  
  <p>on linux terminal, but got a lot mass, seems like wrong codec.</p>
  
  <ol start="2">
  <li><p>I then tried using <code>file_get_contents('http://thesite.html')</code> and also can't get the correct webpage.</p></li>
  <li><p>Then I tried <code>urllib2.urlopen('http://thesite.html')</code>, can't work.    </p></li>
  <li>can't detect the encoding.
  <code>
  s = urllib2.urlopen('http://thesite.html')
  print chardet.detect(s)
  </code>
  and outputs <code>{'confidence':0.0, 'encoding':None}</code></li>
  <li>I also tried python3 with urllib.request, I can get a byte string, but when I try to decode this bytes to utf-8, I got a error message.</li>
  </ol>
  
  <p>Anyone can help me about this? How to get the correct webpage like a web browser.</p>