text generation with lstm of the time note that greedy sampling can be also cast as sampling from probability distributionone where certain character has probability and all others have probability sampling probabilistically from the softmax output of the model is neatit allows even unlikely characters to be sampled some of the timegenerating more interestinglooking sentences and sometimes showing creativity by coming up with newrealisticsounding words that didn' occur in the training data but there' one issue with this strategyit doesn' offer way to control the amount of randomness in the sampling process why would you want more or less randomnessconsider an extreme casepure random samplingwhere you draw the next character from uniform probability distributionand every character is equally likely this scheme has maximum randomnessin other wordsthis probability distribution has maximum entropy naturallyit won' produce anything interesting at the other extremegreedy sampling doesn' produce anything interestingeitherand has no randomnessthe corresponding probability distribution has minimum entropy sampling from the "realprobability distribution--the distribution that is output by the model' softmax function--constitutes an intermediate point between these two extremes but there are many other intermediate points of higher or lower entropy that you may want to explore less entropy will give the generated sequences more predictable structure (and thus they will potentially be more realistic looking)whereas more entropy will result in more surprising and creative sequences when sampling from generative modelsit' always good to explore different amounts of randomness in the generation process because we--humans--are the ultimate judges of how interesting the generated data isinterestingness is highly subjectiveand there' no telling in advance where the point of optimal entropy lies in order to control the amount of stochasticity in the sampling processwe'll introduce parameter called the softmax temperature that characterizes the entropy of the probability distribution used for samplingit characterizes how surprising or predictable the choice of the next character will be given temperature valuea new probability distribution is computed from the original one (the softmax output of the modelby reweighting it in the following way listing reweight ing probability dist ribut ion different emperat ure import numpy as np def reweight_distribution(original_distributiontemperature= )distribution np log(original_distributiontemperature distribution np exp(distributionreturn distribution np sum(distributionoriginal_distribution is numpy array of probability values that must sum to temperature is factor quantifying the entropy of the output distribution licensed to returns reweighted version of the original distribution the sum of the distribution may no longer be so you divide it by its sum to obtain the new distribution