UnicodeEncodeError Only When Script is Run as a Subprocess
Question: I'm running my main script in Python 3.5 using the Spyder IDE, and I want to
import functions from a script that happens to only work in Python 3.4. So I
was recommended to run this second script as a subprocess like so:

    
    
    import subprocess
    cmd = [r'c:\python34\pythonw.exe', r'C:\users\John\Desktop\scraper.py']
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = p.communicate()
    print(stdout)
    print(stderr)
    

The script being called is an example from NikolaiT's Web engine scraper:

    
    
    # -*- coding: utf-8 -*-
    
    import sys
    from GoogleScraper import scrape_with_config, GoogleSearchError
    from GoogleScraper.database import ScraperSearch, SERP, Link
    
    def basic_usage():
        # See in the config.cfg file for possible values
        config = {
            'SCRAPING': {
                'use_own_ip': 'True',
                'search_engines': 'baidu',
                'num_pages_for_keyword': 3
            },
            'keyword': '苹果',
            'SELENIUM': {
                'sel_browser': 'chrome',
            },
            'GLOBAL': {
                'do_caching': 'False'
            }
        }
    
        try:
            sqlalchemy_session = scrape_with_config(config)
        except GoogleSearchError as e:
            print(e)
    
        # let's inspect what we got
        link_list = []
        for serp in sqlalchemy_session.serps:
            #print(serp)
            for link in serp.links:
                #print(link)
                link_list.append(link.link)
        return link_list
    
    
    
    links = basic_usage()
    
    print("test")
    for link in links:
        print(link)
    

This script works just fine when running it in Python 3.4's IDLE IDE, but when
running it as a subprocess as above, I get the following UnicodeEncodeError
printed from my main script:

\python34\scripts\googlescraper\GoogleScraper\caching.py", line 413, in
parse_all_cached_files store_serp_result(serp, self.config)

File "c:\python34\scripts\googlescraper\GoogleScraper\output_converter.py",
line 123, in store_serp_result pprint.pprint(data)

File "c:\python34\lib\pprint.py", line 52, in pprint printer.pprint(object)

File "c:\python34\lib\pprint.py", line 139, in pprint self._format(object,
self._stream, 0, 0, {}, 0)

File "c:\python34\lib\pprint.py", line 193, in _format allowance + 1, context,
level)

File "c:\python34\lib\pprint.py", line 268, in _format write(rep)

File "c:\python34\lib\encodings\cp1252.py", line 19, in encode return
codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError:
\'charmap\' codec can\'t encode characters in position 1-2: character maps to
undefined'

Why would this only happen when running it indirectly? Thanks for any help,
clarification questions, or suggestions for improvement in framing my
question.


Answer: In short, your issue is as follows:

  * Python IDE fully supports UTF-8 encoding, and that's why your invoked script runs fine there.
  * On the other hand, when opening a subprocess, since you are on Windows, you are using Windows1252 charset by default, which doesn't support part of your output.

Quick solution: If you invoke the script as `python foobar.py` on Windows
command line, you can run `chcp 65001` before invoking the script; if you use
Spyder IDE, there should be a setting that allows you to set file / project
encoding to UTF-8.

(You may also try adding `# -*- coding: utf-8 -*-` to the top of your main
Python script.)

