Convert string of list of dictionary to Python DataFrame
Question: I have a .JSON file which is around 3GB. I would like to read this JSON data
and load it to pandas data frames. Below is what i did so far..

Step 1: Read JSON file

    
    
    import pandas as pd
    with open('MyFile.json', 'r') as f:
        data = f.readlines()
    

Step2: just take one component, since data is huge and i want to see how it
looks

    
    
    cp = data[0:1]
    print(cp)
    
    ['{"reviewerID": "AO94DHGC771SJ", "asin": "0528881469", "reviewerName": "amazdnu", "helpful": [0, 0], "reviewText": "some review text...", "overall": 5.0, "summary": "Gotta have GPS!", "unixReviewTime": 1370131200, "reviewTime": "06 2, 2013"}\n']
    

Step3: to remove new line('\n') character

    
    
    while ix<len(t):
        t[ix]=t[ix].rstrip("\n")
        ix+=1
    

Questions:

  1. Why this JSON data is in string? Am I making any mistakes?
  2. How do I convert it into dictionary?

What I tried?

  1. I tried `b=dict(zip(t[0::2],t[1::2]))`, but get - 'dict' object not callable

  2. Tried joining, but did not work though

Can any one please help me? Thanks!


Answer: Why haven't you tried `pandas.read_json`?

    
    
    import pandas as pd
    df = pd.read_json('MyFile.json')
    

Works for the example you posted!

    
    
    In[82]: i = '{"reviewerID": "AO94DHGC771SJ", "asin": "0528881469", "reviewerName": "amazdnu", "helpful": [0, 0], "reviewText": "some review text...", "overall": 5.0, "summary": "Gotta have GPS!", "unixReviewTime": 1370131200, "reviewTime": "06 2, 2013"}'
    
    In[83]: pd.read_json(i)
    Out[83]: 
            asin  helpful  overall           reviewText  reviewTime     reviewerID reviewerName          summary  unixReviewTime
    0  528881469        0        5  some review text...  06 2, 2013  AO94DHGC771SJ      amazdnu  Gotta have GPS!      1370131200
    

