In the Keras Tokenizer class, what exactly does word_index signify? I'm trying to really understand Tokenizing and Vectorizing text in machine learning, and am looking really hard into the Keras Tokenizer class. I get the mechanics of how it's used, but I'd like to really know more about it. So for example, there's this common usage:

tokens = Tokenizer(num_words=SOME_NUMBER)
tokens.fit_on_texts(texts)


tokens returns a word_index, which maps words to some number.
Are the words all words in texts, or are they maxed at SOME_NUMBER?
And are the dict values for word_index the frequency of each word, or just the order of the word?
Are the words all words in texts, or are they maxed at SOME_NUMBER?


Yes, it will be maxed at the most common SOME_NUMBER-1 words.


  And are the dict values for word_index the frequency of each word, or just the order of the word?


It is just the index of the word in the dictionary.

You can read more here in the documentation.