understanding -grams and bag-of-words word -grams are groups of (or fewerconsecutive words that you can extract from sentence the same concept may also be applied to characters instead of words here' simple example consider the sentence the cat sat on the mat it may be decomposed into the following set of -grams{"the""the cat""cat""cat sat""sat""sat on""on""on the""the""the mat""mat"it may also be decomposed into the following set of -grams{"the""the cat""cat""cat sat""the cat sat""sat""sat on""on""cat sat on""on the""the""sat on the""the mat""mat""on the mat"such set is called bag-of- -grams or bag-of- -gramsrespectively the term bag here refers to the fact that you're dealing with set of tokens rather than list or sequencethe tokens have no specific order this family of tokenization methods is called bag-of-words because bag-of-words isn' an order-preserving tokenization method (the tokens generated are understood as setnot sequenceand the general structure of the sentences is lost)it tends to be used in shallow language-processing models rather than in deep-learning models extracting -grams is form of feature engineeringand deep learning does away with this kind of rigidbrittle approachreplacing it with hierarchical feature learning one-dimensional convnets and recurrent neural networksintroduced later in this are capable of learning representations for groups of words and characters without being explicitly told about the existence of such groupsby looking at continuous word or character sequences for this reasonwe won' cover -grams any further in this book but do keep in mind that they're powerfulunavoidable feature-engineering tool when using lightweightshallow text-processing models such as logistic regression and random forests one-hot encoding of words and characters one-hot encoding is the most commonmost basic way to turn token into vector you saw it in action in the initial imdb and reuters examples in (done with wordsin that caseit consists of associating unique integer index with every word and then turning this integer index into binary vector of size (the size of the vocabulary)the vector is all zeros except for the th entrywhich is of courseone-hot encoding can be done at the character levelas well to unambiguously drive home what one-hot encoding is and how to implement itlistings and show two toy examplesone for wordsthe other for characters licensed to