working with text data listing using keras for word-level one-hot encoding from keras preprocessing text import tokenizer creates tokenizerconfigured to only take into account the , most common words samples ['the cat sat on the mat ''the dog ate my homework 'builds the word index tokenizer tokenizer(num_words= tokenizer fit_on_texts(samplessequences tokenizer texts_to_sequences(samplesturns strings into lists of integer indices one_hot_results tokenizer texts_to_matrix(samplesmode='binary'word_index tokenizer word_index print('found % unique tokens len(word_index)you could also directly get the one-hot binary representations vectorization modes other than one-hot encoding are supported by this tokenizer how you can recover the word index that was computed variant of one-hot encoding is the so-called one-hot hashing trickwhich you can use when the number of unique tokens in your vocabulary is too large to handle explicitly instead of explicitly assigning an index to each word and keeping reference of these indices in dictionaryyou can hash words into vectors of fixed size this is typically done with very lightweight hashing function the main advantage of this method is that it does away with maintaining an explicit word indexwhich saves memory and allows online encoding of the data (you can generate token vectors right awaybefore you've seen all of the available datathe one drawback of this approach is that it' susceptible to hash collisionstwo different words may end up with the same hashand subsequently any machine-learning model looking at these hashes won' be able to tell the difference between these words the likelihood of hash collisions decreases when the dimensionality of the hashing space is much larger than the total number of unique tokens being hashed listing word-level one-hot encoding wit hashing rick ( oy examplesamples ['the cat sat on the mat ''the dog ate my homework 'dimensionality max_length results np zeros((len(samples)max_lengthdimensionality)for isample in enumerate(samples)for jword in list(enumerate(sample split()))[:max_length]index abs(hash(word)dimensionality results[ijindex stores the words as vectors of size , if you have close to , words (or more)you'll see many hash collisionswhich will decrease the accuracy of this encoding method licensed to hashes the word into random integer index between and ,