python json encode - Missing ( and ' - urllib2.open() is ending in HTTP Error 400
Question: I am encoding a 40kb dictionary of dictionaries and lists into json and then
pushing it over http to a nosql database. I've used both jsonpickle.encode and
json.dumps modules to encode my dictionary's content, but both are leading to
an http error. I tried manually CURLing the problematic section of code with a
result of the error "-bash: syntax error near unexpected token `('"

Here is some example code:

    
    
    import urllib2 , jsonpickle
    url = "http://amazonaws.com/server/%s/_create" % item
    data = jsonpickle.encode( some_dict_of_dicts_and_lists ) # also tried json.dumps here.
    try:
        req = urllib2.Request ( url , data , { 'Content-Type' : 'application/json' } )
        f = urllib2.urlopen ( req )
    except Exception as e:
        print "Error: %s" %e
    

The above works for entering certain sections of my dictionary into my nosql
db. However, this prints "Error: HTTP Error 400: Bad Request" when I send over
other sections of my dictionary. This means to me that something is not
getting encoded properly in the data variable/string. To get CURL's response
on this problem, I tried CURLing the following code:

    
    
    item_id = item_dictionary [ 'id' ]
    data = jsonpickle.encode( some_dict_of_dicts_and_lists ) # also tried json.dumps here.
    
    command = 'curl -XPOST "http://amazonaws.com/server/%s/_create" -d '"%s"' % ( nsn_id , data )
    os.system(command)
    

This produces the error "sh: -c: line 0: syntax error near unexpected token
`('"  

If I try to manually type the curl command into the command line, I get the
following set of errors:  
curl: (6) Could not resolve host: material; nodename nor servname provided, or
not known  
curl: (6) Could not resolve host: items; nodename nor servname provided, or
not known  
curl: (6) Could not resolve host: of; nodename nor servname provided, or not
known  
curl: (3) [globbing] unmatched close brace/bracket at pos 146  
{"error":"MapperParsingException[Failed to parse]; nested:
JsonParseException[Unexpected end-of-input in VALUE_STRING\n at [Source:
[B@2ff246ab; line: 1, column: 6519]]; ","status":400}  

So:  
1) Is there a better way to make sure the the json encode process captures
makes all of the parentheses and single quotes are formatted with a '\' in
front of them? Should I do a replace on all of them?  

2) Is there a good way to figure out the cause of this error in a more
detailed way using urllib2 or do I need to do CURL from the command line in
order to get the bash type error?


Answer: To answer both your questions:

1) `json.dumps` \- assuming that you pass it valid input - will always create
a string representing a valid JSON object. If you pass invalid input, it will
throw an exception. Python's `json` module is extremely widely used and if
there were a bug that could cause it to spit out an invalidly formatted JSON
dictionary then this would definitely be known (and fixed) by now.

2) You're getting the HTTPError because the web server is returning a page
with a [HTTP Status
Code](http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html) of 400 in
response to your request. There was probably also some content in the page
body, though, and `urllib2` hasn't thrown that away. You can call the
`.read()` method of the HTTPError to get any content that was on the page,
rather than just seeing the status code like you are at the moment (which is
sometimes not enough to understand what you've done wrong).

Most web APIs will return you some sort of useful explanation when spitting
out a `400 - Bad Request` error, such as a message telling you what parameter
you missed out in your request. Putting `print e.read()` in your exception
block will let you see this, and will probably allow you to solve your current
problem.

