How to split json dataset and save it?
I have taken a json dataset. Dataset name is v2_OpenEnded_mscoco_train2014_questions.json How can I split some of the data from the dataset and save the split data into another json file?
This is the example of my dataset:

{&quot;image_id&quot;: 426004, &quot;question&quot;: &quot;How many buns are on the plate?&quot;,
&quot;question_id&quot;: 426004002}, {&quot;image_id&quot;: 92846, &quot;question&quot;: &quot;What is
the color of the vase without flowers?&quot;, &quot;question_id&quot;: 92846000},
{&quot;image_id&quot;: 92846, &quot;question&quot;: &quot;Is there anything red in this
photo?&quot;, &quot;question_id&quot;: 92846002}, {&quot;image_id&quot;: 92846, &quot;question&quot;:
&quot;What does that vase represent?&quot;, &quot;question_id&quot;: 92846003},
{&quot;image_id&quot;: 262166, &quot;question&quot;: &quot;What color is the couch?&quot;,
&quot;question_id&quot;: 262166002}, {&quot;image_id&quot;: 262166, &quot;question&quot;: &quot;How many
seats are available?&quot;, &quot;question_id&quot;: 262166003}

I have about 443,757 data in my dataset. I want to split the dataset into 400 different dataset each having 100 data. How can I automate this using python?
It will be a great help if this can be done with pytorch.

Try this:
n_rows = 100
current_data = []
for i, e in enumerate(data):
    if i % n_rows == 0 and i &gt; 0:
        with open(f'dataset_{i - n_rows}-{i}.json', 'w') as f:
            json.dump(current_data, f)
        current_data = []
    
    current_data.append(e)

data is a list with jsons which you have. We iterate over it and every n_rows lines are written in a new file. The last several rows aren't written.
