Random comma inserted at character 8192 in python "json" result called from node.js
Question: I'm a JS developer just learning python. This is my first time trying to use
node (v6.7.0) and python (v2.7.1) together. I'm using restify with python-
runner as a bridge to my python virtualenv. My python script uses a RAKE NLP
keyword-extraction package.

I can't figure out for the life of me why my return data in **server.js**
inserts a random comma at character 8192 and roughly multiples of. There's no
pattern except the location; Sometimes it's in the middle of the object key
string other times in the value, othertimes after the comma separating the
object pairs. This completely breaks the JSON.parse() on the return data.
Example outputs below. When I run the script from a python shell, this doesn't
happen.

I seriously can't figure out why this is happening, any experienced devs have
any ideas?

_Sample output in browser_

    
    
    [..., {...ate': 1.0, 'intended recipient': 4.,0, 'correc...}, ...]
    

_Sample output in python shell_

    
    
    [..., {...ate': 1.0, 'intended recipient': 4.0, 'correc...}, ...]
    

**DISREGARD ANY DISCREPANCIES REGARDING OBJECT CONVERSION AND HANDLING IN THE
FILES BELOW. THE CODE HAS BEEN SIMPLIFIED TO SHOWCASE THE ISSUE**

**server.js**

    
    
    var restify = require('restify');
    var py = require('python-runner');
    
    var server = restify.createServer({...});
    
    server.get('/keyword-extraction', function( req, res, next ) {
    
        py.execScript(__dirname + '/keyword-extraction.py', {
            bin: '.py/bin/python'
        })
        .then( function( data ) {
            fData = JSON.parse(data); <---- ERROR
            res.json(fData);
        })
        .catch( function( err ) {...});
    
        return next();
    });
    
    server.listen(8001, 'localhost', function() {...});
    

**keyword-extraction.py**

    
    
    import csv
    import json
    import RAKE
    
    f = open( 'emails.csv', 'rb' )
    f.readline() # skip line containing col names
    
    outputData = []
    
    try:
        reader = csv.reader(f)
    
        for row in reader:
            email = {}
            emailBody = row[7]
    
            Rake = RAKE.Rake('SmartStoplist.txt')
    
            rakeOutput = Rake.run(emailBody)        
    
            for tuple in rakeOutput:
    
                email[tuple[0]] = tuple[1]
    
            outputData.append(email)
    
    finally:
        file.close()
    
        print( json.dumps(outputData))
    


Answer: This looks suspiciously like a bug related to size of some buffer, since 8192
is a power of two.

The main thing here is to isolate exactly where the failure is occurring. If I
were debugging this, I would

  1. Take a closer look at the output from `json.dumps`, by printing several characters on either side of position 8191, ideally the integer character code (unicode, ASCII, or whatever). 

  2. If that looks OK, I would try capturing the output from the python script as a file and read that directly in the node server (i.e. don't run a python script). 

  3. If that works, then create a python script that takes that file and outputs it without manipulation and have your node server execute that python script instead of the one it is using now.

That should help you figure out where the problem is occurring. From comments,
I suspect that this is essentially a bug that you cannot control, unless you
can increase the python buffer size enough to guarantee your data will never
blow the buffer. 8K is pretty small, so that might be a realistic solution.

If that is inadequate, then you might consider processing the data on the the
node server, to remove every character at `n * 8192`, if you can consistently
rely on that. Good luck.

