Error in printing scraped webpage through bs4
Question: **Code:**

    
    
    import requests
    import urllib
    from bs4 import BeautifulSoup
    
    page1 = urllib.request.urlopen("http://en.wikipedia.org/wiki/List_of_human_stampedes")
    soup = BeautifulSoup(page1)
    print(soup.get_text())
    print(soup.prettify())
    

**Error:**

    
    
     Traceback (most recent call last):
      File "C:\Users\sony\Desktop\Trash\Crawler Try\try2.py", line 9, in <module>
        print(soup.get_text())
      File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode
        return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\u014d' in position 10487: character maps to <undefined>
    

I think the problem lies mainly with urlib package. Here I am using urllib3
package. They changed the urlopen syntax from 2 to 3 version, which maybe the
cause of error. But that being said I have included the latest syntax only.
Python version 3.4


Answer: since you are importing `requests` you can use it instead of urllib like this:

    
    
    import requests
    from bs4 import BeautifulSoup
    
    page1 = requests.get("http://en.wikipedia.org/wiki/List_of_human_stampedes")
    soup = BeautifulSoup(page1.text)
    print(soup.get_text())
    print(soup.prettify())
    

Your problem is that python cannot encode the characters from the page that
you are scraping. For some more information see here:
<http://stackoverflow.com/a/16347188/2638310>

Since the wikipedia page is in UTF-8, it seems that BeautifulSoup is guessing
the encoding incorrectly. Try passing the `from_encoding` argument in your
code like this:

    
    
    soup = BeautifulSoup(page1.text, from_encoding="UTF-8")
    

For more on encodings in BeautifulSoup have a look here:
<http://www.crummy.com/software/BeautifulSoup/bs4/doc/#encodings>

