Python I/O: Mixing The Datatypes
Question: I'm writing a small script which merges a host of JSON files in one directory
into a single file. Trouble is, I'm not entirely sure when my data is in which
state. TypeErrors abound. Here's the script;

    
    
    import glob
    import json
    import codecs
    
    reader = codecs.getreader("utf-8")
    
    for file in glob.glob("/Users/me/Scripts/BagOfJson/*.json"):
    #Aha, as binary here
    with open(file, "rb") as infile:
        data = json.load(reader(infile))
        #If I print(data) here, looks like good ol' JSON
    
        with open("test.json", "wb") as outfile:
            json.dump(data, outfile, sort_keys = True, indent = 2, ensure_ascii = False)
        #Crash
    

This script results in the following error;

    
    
    TypeError: a bytes-like object is required, not 'str'
    

Which is caused by the json.dump line.

Naive me just deletes the 'b' in 'wb' for the outfile open. That doesn't do
the trick.

Maybe this is a lesson to me to use the shell for testing, and making use of
the type() python function. Still, I'd love if someone can clear up for me the
logic behind these data swaps. I wish it could all be strings...


Answer: If this is Python 3, removing the `b` (binary mode) to open the file in _text
mode_ should work just fine. You probably want to specify the encoding
explicitly:

    
    
    with open("test.json", "w", encoding='utf8') as outfile:
        json.dump(data, outfile, sort_keys = True, indent = 2, ensure_ascii = False)
    

rather than rely on a default.

You shouldn't really use `codecs.getreader()`. The standard `open()` function
can handle UTF-8 files just fine; just open the file in text mode and specify
the encoding again:

    
    
    import glob
    import json
    
    for file in glob.glob("/Users/me/Scripts/BagOfJson/*.json"):
        with open(file, "r", encoding='utf8') as infile:
            data = json.load(infile)
            with open("test.json", "w", encoding='utf8') as outfile:
                json.dump(data, outfile, sort_keys = True, indent = 2, ensure_ascii = False)
    

The above will still re-create `test.json` for each file in the `*.json` glob;
you can't really put multiple JSON documents in the same file anyway (unless
you specifically create [JSONLines files](http://jsonlines.org/), which you
are not doing here because you are using `indent`).

You'd need to write to a new filename and move the new back to the `file`
filename if you wanted to re-format all JSON files in the glob.

