BeautifulSoup not showing Korean letters properly
Question: I'm a newbie in Python who is learning parsing with BeautifulSoup.

This is my code,

    
    
    #-*- coding: utf-8 -*-
    
    import urllib
    from bs4 import BeautifulSoup
    
    soup = BeautifulSoup(urllib.urlopen('https://news.google.com/news/section?cf=all&pz=1&q=IoT').read())
    editData = soup.find_all('span',{'class','titletext'})
    
    print editData
    

and the result is as follows: (Korean letters are shown
"\uc720\ud50c\ub7ec\uc2a4" etc)

> [LG\uc720\ud50c\ub7ec\uc2a4, **IoT** \uc720\ub9dd\uae30\uc5c5
> \ubc1c\uad74\u2026 \uc0c1\uc0dd \ud611\ub825 \ucd94\uad6c,
> LG\uc720\ud50c\ub7ec\uc2a4, \uc720\ub9dd **IoT** \uc911\uc18c\uae30\uc5c5
> \uc9c0\uc6d0, LGU+ '**IoT** \uc720\ub9dd \uc911\uc18c\uae30\uc5c5
> \ubaa8\uc5ec\ub77c', "**IoT** \ub85c \uace0\uce35\ube4c\ub529
> \uc5d0\ub108\uc9c0 \uc18c\ube44 80% \u2193",
> \uc2dc\ud050\ub9ac\ud2f0\ud50c\ub7ab\ud3fc-\uc774\ub354\ube14\uc720\ube44\uc5e0
> **IoT** \ubcf4\uc548 \ud1a0\ud138\uc194\ub8e8\uc158 \uac1c\ubc1c, NIA,
> **IoT** \uc735\ud569 \uc2e0\uc0b0\uc5c5 \uc9c0\uc6d0 \uc704\ud574 135\uc5b5
> \uaddc\ubaa8 \ud22c\uc790, **IoT** \ub9db\uc744 \uc54c\uac8c \ub41c
> \ud1b5\uc2e03\uc0ac, '\uc0c1\uc0dd'\uc774 \uacb0\uad6d \ubbf8\ub798
> \ub9cc\ub4e0\ub2e4, \uc0bc\uc131\uc804\uc790, \ubd81\ubbf8 `<b>IoT</b>
> \uc2a4\ub9c8\ud2b8 \ube4c\ub529 \ucf00\uc5b4 \uc2dc\uc2a4\ud15c`
> \uacf5\ub7b5...\ud604\uc9c0 \ub370\uc778\ud2b8\ub9ac\uc640 \ud611\uc5c5,
> \ud30c\uc6cc\ubcf4\uc774\uc2a4, \uc74c\uc131\uc778\uc2dd **IoT**
> \uc2a4\ub9c8\ud2b8 \uc2a4\uc704\uce58 \ucd9c\uc2dc, \uace0\ub824\ub300,
> \uc815\ubcf4\ud1b5\uc2e0 \uace0\ub3c4\ud654\ud55c '**IoT**
> \ucea0\ud37c\uc2a4' \ubcc0\uc2e0,

... and so on.

I can't find any solutions concerning this problem.


Answer: Its not because of BeautifulSoup its because the way you print the node. doing
something like:

    
    
    print editData
    

will call the function `__repr__()` of the node, which is implemented by
returning the content as ascii string.

try this:

    
    
    for span in editData:
        print span.text
    

and you should see the unicode text (the for-in loop is because the result is
a list of nodes and not a single node so we print them all).

