in [ ]from collections import counter build word to index vocabulary token_counter counter([token for review in tokenized_train for token in review]vocab_map {item[ ]index+ for indexitem in enumerate(dict(token_counteritems())max_index np max(list(vocab_map values())vocab_map['pad_index' vocab_map['not_found_index'max_index+ vocab_size len(vocab_mapview vocabulary size and part of the vocabulary map print('vocabulary size:'vocab_sizeprint('sample slice of vocabulary map:'dict(list(vocab_map items())[ : ])vocabulary size sample slice of vocabulary map{'martyrdom' 'palmira' 'servility' 'gardening' 'melodramatically' 'renfro' 'carlin' 'overtly' 'rend' 'anticlimactic' in this case we have used all the terms in our vocabularyyou can easily filter and use more relevant terms here (based on their frequencyby using the most_common(countfunction from counter and taking the first count terms from the list of unique terms in the training corpus we will now encode the tokenized text reviews based on the previous vocab_map besides thiswe will also encode the text sentiment class labels into numeric representations in [ ]from keras preprocessing import sequence from sklearn preprocessing import labelencoder get max length of train corpus and initialize label encoder le labelencoder(num_classes= positive - negative - max_len np max([len(reviewfor review in tokenized_train]#train reviews data corpus convert tokenized text reviews to numeric vectors train_x [[vocab_map[tokenfor token in tokenized_reviewfor tokenized_review in tokenized_traintrain_x sequence pad_sequences(train_xmaxlen=max_lenpad #train prediction class labels convert text sentiment labels (negative\positiveto binary encodings ( / train_y le fit_transform(train_sentiments#test reviews data corpus convert tokenized text reviews to numeric vectors test_x [[vocab_map[tokenif vocab_map get(tokenelse vocab_map['not_found_index'for token in tokenized_reviewfor tokenized_review in tokenized_testtest_x sequence pad_sequences(test_xmaxlen=max_len