Understanding TypeError: '<' not supported between instances of 'Example' and 'Example'
I am working on a project of text simplification using a multi-head attention transformer model. For the same, I am using torchtext for tokenisation and numericalization. The dataset contains two aligned files for training and two aligned files for testing. In the training files, one file contains the complex sentences while the other contains the corresponding simplified sentences.

I read the files as such:

training_sentences = open(path + "train.en" , encoding = "utf-8").read().split("\n")
target_sentences = open(path + "train.sen" , encoding = "utf-8").read().split("\n")


Next, I tokenised them as such:

complicated = spacy.load('en')
simple = spacy.load('en')

def tokenize_complicated(sentence):
   return [tok.text for tok in complicated.tokenizer(sentence)]

def tokenize_simple(sentence):
    return [tok.text for tok in simple.tokenizer(sentence)]

C_TEXT = Field(tokenize=tokenize_complicated, fix_length = 100)
S_TEXT = Field(tokenize=tokenize_simple, fix_length = 100, init_token = "&lt;sos&gt;", eos_token = "&lt;eos&gt;")


I then converted into TabularDataset object of torchtext.

import pandas as pd
raw_data = {'Complicated' : [line for line in training_sentences], 
        'Simple': [line for line in target_sentences]}

df = pd.DataFrame(raw_data, columns=["Complicated", "Simple"])

df.to_csv("df.csv", index=False)
data_fields = [('Complicated', C_TEXT), ('Simple', S_TEXT)]

train = torchtext.data.TabularDataset.splits(path='./', train = "df.csv", format='csv', fields=data_fields, skip_header = True)


And then created vocabulary

C_TEXT.build_vocab(train)
S_TEXT.build_vocab(train)


However, on doing so I got this error: 


  TypeError: '&lt;' not supported between instances of 'Example' and
  'Example'


On searching, I came across this solution here and the error disappeared. However, I am not understanding whether this makes the model take only one instance or it takes all of the dataset?
I would like to know the significance of the index [0] so that I can manipulate it effectively for my model.

In my case I solved the issue passing a sort_key and sort_within_batch arg to True as follows:
BATCH_SIZE = 64
train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data),
    device = device,
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.src),
    sort_within_batch=True
)

Good luck
