Loading json file using torchtext
I'm working on the dailydialog dataset, which I've converted into a
JSON file which looks something like this:
[{&quot;response&quot;: &quot;You know that is tempting but is really not good for our fitness.&quot;, &quot;message&quot;: &quot;Say, Jim, how about going for a few beers after dinner?&quot;}, {&quot;response&quot;: &quot;Do you really think so? I don't. It will just make us fat and act silly. Remember last time?&quot;, &quot;message&quot;: &quot;What do you mean? It will help us to relax.&quot;}, {&quot;response&quot;: &quot;I suggest a walk over to the gym where we can play singsong and meet some of our friends.&quot;, &quot;message&quot;: &quot;I guess you are right. But what shall we do? I don't feel like sitting at home.&quot;}, {&quot;response&quot;: &quot;Sounds great to me! If they are willing, we could ask them to go dancing with us.That is excellent exercise and fun, too.&quot;, &quot;message&quot;: &quot;That's a good idea. I hear Mary and Sally often go there to play pingpong.Perhaps we can make a foursome with them.&quot;}, {&quot;response&quot;: &quot;All right.&quot;, &quot;message&quot;: &quot;Please lie down over there.&quot;}]
So, each item has two keys - response and message.
This is my first time using PyTorch, so I was following a few online available resources. These are the relevant snippets of my code:
def tokenize_en(text):
    return [tok.text for tok in spacy_en.tokenizer(text)]

src = Field(tokenize = tokenize_en, 
            init_token = '&lt;sos&gt;', 
            eos_token = '&lt;eos&gt;', 
            lower = True)

fields = {'response': ('r', src)}

train_data, test_data, validation_data = TabularDataset.splits(     
                                        path = 'FilePath',
                                        train = 'trainset.json',
                                        test = 'testset.json',
                                        validation = 'validationset.json',
                                        format = 'json',
                                        fields = fields        
)

Although no errors are raised, despite having many items in my JSON file, the train, test and validation datasets strangely have only 1 example each, as seen in this image:
Image Showing the length of train_data, test_data and validation_data
I'd be really grateful if someone could point out the error to me.
Edit: I found out that the whole file is being treated as a single text string due to lack of indents in the file. But if I indent the JSON file, the TabularDataset function throws a JSONDecodeError to me, suggesting it can no more decode the file. How can I get rid of this problem?

I think the code is alright, but the issue is with your JSON file. Can you try removing the square brackets(&quot;[]&quot;) at the beginning and the end of the file?
Probably that is the reason that Your Python file is reading it as one single object.
