Reason for adding 1 to word index for sequence modeling I notice in many of the tutorials 1 is added to the word_index. For example considering a sample code snippet inspired from Tensorflow's tutorial for NMT https://www.tensorflow.org/tutorials/text/nmt_with_attention :
import tensorflow as tf
sample_input = [&quot;sample sentence 1&quot;, &quot;sample sentence 2&quot;]
lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
lang_tokenizer.fit_on_texts(sample_input)
vocab_inp_size = len(lang_tokenizer.word_index)+1

I dont understand the reason for adding 1 to the word_index dictionary. Wont adding a random 1 affect the prediction. Any suggestions will be helpful
First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected.
Then, why adding 1 ?
Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned incrementally. Therefore, the greatest token ID in word_index is len(word_index). Therefore, we need vocabulary of size len(word_index) + 1 to be able to index up to the greatest token ID.
Update: note that adding 1 to the vocabulary size has nothing to do with out of vocabulary words: the words that are not pretrained are encoded as the out-of-vocabulary token (oov_token) if it was provided when building the tokenizer, or ignored if not. The oov token, if provided, has index 1.