generative deep learning how do you generate sequence datathe universal way to generate sequence data in deep learning is to train network (usually an rnn or convnetto predict the next token or next few tokens in sequenceusing the previous tokens as input for instancegiven the input "the cat is on the ma,the network is trained to predict the target tthe next character as usual when working with text datatokens are typically words or charactersand any network that can model the probability of the next token given the previous ones is called language model language model captures the latent space of languageits statistical structure once you have such trained language modelyou can sample from it (generate new sequences)you feed it an initial string of text (called conditioning data)ask it to generate the next character or the next word (you can even generate several tokens at once)add the generated output back to the input dataand repeat the process many times (see figure this loop allows you to generate sequences of arbitrary length that reflect the structure of the data on which the model was trainedsequences that look almost like human-written sentences in the example we present in this sectionyou'll take lstm layerfeed it strings of characters extracted from text corpusand train it to predict character the output of the model will be softmax over all possible charactersa probability distribution for the next character this lstm is called character-level neural language model initial text probability distribution for the next character initial text sampled next character the cat sat on the language model sampling strategy the cat sat on the ma language model sampling strategy figure the process of character-by-character text generation using language model the importance of the sampling strategy when generating textthe way you choose the next character is crucially important naive approach is greedy samplingconsisting of always choosing the most likely next character but such an approach results in repetitivepredictable strings that don' look like coherent language more interesting approach makes slightly more surprising choicesit introduces randomness in the sampling processby sampling from the probability distribution for the next character this is called stochastic sampling (recall that stochasticity is what we call randomness in this fieldin such setupif has probability of being the next characteraccording to the modelyou'll choose it licensed to