Issue with scraping data with foreign characters
Question: I have written a python script to scrape data from some chinese site.
According to its head charset is "gb2312" and have also checked it with
"chardet.detect()" python library that its correct but still i get wrong
characters. Here is my code:

    
    
    import csv
    import requests
    import cssselect
    from lxml import html
    
    url = "http://www.example.com/"
    header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36'}
    
    mainPage = requests.get(url, headers = header)
    tree = html.fromstring(mainPage.content)
    
    with open('jd(test).csv', 'a', newline='') as csvfile:
        csvwriter = csv.writer(csvfile, delimiter=',')
        main_info = ["URL", "CAT-Name Original", "CAT-Name Converted"]
        csvwriter.writerow(main_info)
    
        for link in tree.cssselect("div#allsort div div.m div.mc dl dd em a"):
            urls = [link.get('href')]
            text = link.text
    
            print (urls)
    
            convertedText = text.encode('gb2312')
    
            print (text)
            print (convertedText)
    
            row_info = [urls, text, convertedText]
    
            csvwriter.writerow(row_info)
    

OUTPUT:

    
    
    ['http://example.com/']
    戏曲综艺
    b'\xcf\xb7\xc7\xfa\xd7\xdb\xd2\xd5'
    


Answer: What you ask cannot work; encoding is for character sets only, you want
**translation**.

You can get it using [py-translate](http://pythonhosted.org/py-
translate/devs/api.html#translator-co-routine), which is an interface to
Google but apparently free to use.

    
    
    Python 3.4.3 (default, Mar 27 2015, 02:30:53) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from translate import translator
    >>> translator('zh-Tw', 'en', '戏曲综艺')
    [[['Opera Arts', '戏曲综艺', 0]], 'zh-TW']
    >>>
    

I am not wholly familiar with the tool, so you'll better check out the
license.

> what do i need to do to get it translated without using any third party?

By definition, if you do not want to use _any third party_ , your only option
is to _learn Chinese_. In some outfits that might be a workable option. Case
in point: a firm I worked for needed some Chinese translations. Then more. And
more. First they hired a translator. Then a Chinese intern, and organized a
Chinese course for two developers. Depending on the amount of data, it could
be economically convenient.

But I think you want some kind of _free (as in beer) tool_. The problem being
that most tools are either quite amateurish and will at best spew
[Engrish](https://en.wikipedia.org/wiki/Engrish), or are **not free** in some
form or other; they might be trials, or "personal use". The fact is that
people developing these tools need to put in _a lot of work_ , and they
understandably seek some sort of return on their investment. You should at
least ask yourself, _what should I give back?_ , and, _what am I giving back
(knowingly or not)?_. Unfortunately - until a post-scarcity singularity - this
applies to _everything_.

You can try e.g. the [Baidu service](http://translate.baidu.com/) that has no
"reasonable or personal use" limit that I can readily see, but has his ideas
on your privacy ("Baidu hereby reminds users that the content you input into
Baidu Translate _will not be considered as your private personal
information_."). As long as you do not translate anything...
_controversial_... there should be no problems.

