How to convert this nested JSON file in pandas df?
I have to deal with this database for a project:
In particular, I need to obtain a pandas df to formatting these data as input for a neural network in an NLP task. The Json format is the following:
json file
├── &quot;data&quot;
│   └── [i]
│       ├── &quot;paragraphs&quot;
│       │   └── [j]
│       │       ├── &quot;context&quot;: &quot;paragraph text&quot;
│       │       └── &quot;qas&quot;
│       │           └── [k]
│       │               ├── &quot;answers&quot;
│       │               │   └── [l]
│       │               │       ├── &quot;answer_start&quot;: N
│       │               │       └── &quot;text&quot;: &quot;answer&quot;
│       │               ├── &quot;id&quot;: &quot;&amp;lt;uuid&amp;gt;&quot;
│       │               └── &quot;question&quot;: &quot;paragraph question?&quot;
│       └── &quot;title&quot;: &quot;document id&quot;
└── &quot;version&quot;: 1.1

I tried hard using .json_normalize method but I can't get any results. I've noticed that most of my attempts (those which don't end with an error) end with recognizing as indexes just &quot;data&quot; and &quot;version&quot; and as the only object the rest of the text, as here:
f = open(&quot;SQuAD_it-test.json&quot;, &quot;r&quot;,encoding=&quot;Latin-1&quot; )

data = json.load(f)

df = pd.json_normalize(data)
df.sample(1)


                                         data         version
0   [{'paragraphs': [{'qas': [{'question': 'Quando...   1.1

And if I try to visualize more samples an error occurs to me which tells the population is only 1.
My desired output is something like this, selecting the indexes to use which can be at different level of the tree:
df.sample(5)

        title    context   question  text  answer_start
        str1     str6      str11     str16    N1
        str2     str7      str12     str17    N2
        str3     str8      str13     str13    N3
        str4     str9      str14     str18    N4
        str5     str10     str15     str19    N5

I've also worked on the argument of .json_normalize.
But I'm not able to fully comprehend the explanation. Could you help me?

Since the given json has many nested fileds, we can use record_path and meta arguments to get the desired dataframe:
df = pd.json_normalize(data, record_path=['data', 'paragraphs', 'qas', 'answers'], 
     meta=[['data','title'], ['data', 'paragraphs','context'], 
     ['data', 'paragraphs', 'qas','question']])

Note that the output keys will not be in the exact order as given in the desired output table. Also, keys will have slightly different (fully qualified) names.
print(df.keys())

Ouput:
Index(['text', 'answer_start', 'data.title', 'data.paragraphs.context',
   'data.paragraphs.qas.question'],
  dtype='object')

