title: Change 'temperature' in RNN to generate text
body: <p>I recently followed a <a href="http://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/" rel="nofollow noreferrer">tutorial</a> to make a RNN to generate text:
  I copied the python code exactly and sort of understand it as well. 
  My model has been trained for 20 epochs and it produces a long repetitive loop of 3 words:</p>
  
  <pre><code>"and the wour and the wour and the wour..."
  </code></pre>
  
  <p>I have read in Andrej Kaparthy's <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noreferrer">blog</a> that changing the temperature of the RNN changes its confidence:</p>
  
  <blockquote>
    <p>Decreasing the temperature from 1 to some lower number (e.g. 0.5) makes the RNN more confident, but also more conservative in its samples. </p>
  </blockquote>
  
  <p>I want to change this temperature level to reduce the confidence of the RNN in order for it to create new patterns, but as this is my first Machine Learning project I don't know how. </p>
  
  <p>Here is my Python/keras code:</p>
  
  <p>Generating Text File:</p>
  
  <pre><code># Generate Text
  import sys
  import numpy
  from keras.models import Sequential
  from keras.layers import Dense
  from keras.layers import Dropout
  from keras.layers import LSTM
  from keras.callbacks import ModelCheckpoint
  from keras.utils import np_utils
  
  filename = "king_lear.txt"
  raw_text = open(filename).read()
  raw_text = raw_text.lower()
  
  chars = sorted(list(set(raw_text)))
  char_to_int = dict((c, i) for i, c in enumerate(chars))
  int_to_char = dict((i, c) for i, c in enumerate(chars))
  
  n_chars = len(raw_text)
  n_vocab = len(chars)
  print "Total Characters: ", n_chars
  print "Total Vocab: ", n_vocab
  
  seq_length = 100
  dataX = []
  dataY = []
  for i in range(0, n_chars - seq_length, 1):
      seq_in = raw_text[i:i + seq_length]
      seq_out = raw_text[i + seq_length]
      dataX.append([char_to_int[char] for char in seq_in])
      dataY.append(char_to_int[seq_out])
  n_patterns = len(dataX)
  print "Total Patterns: ", n_patterns
  
  X = numpy.reshape(dataX, (n_patterns, seq_length, 1))
  
  X = X / float(n_vocab)
  
  y = np_utils.to_categorical(dataY)
  
  model = Sequential()
  model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2]), return_sequences=True))
  model.add(Dropout(0.2))
  model.add(LSTM(256))
  model.add(Dropout(0.2))
  model.add(Dense(y.shape[1], activation='softmax'))
  
  filename = "weights-improvement-08-2.0298-bigger.hdf5"
  model.load_weights(filename)
  model.compile(loss='categorical_crossentropy', optimizer='adam')
  
  start = numpy.random.randint(0, len(dataX)-1)
  pattern = dataX[start]
  print "Seed:"
  print "\"", ''.join([int_to_char[value] for value in pattern]), "\""
  
  for i in range(60):
      x = numpy.reshape(pattern, (1, len(pattern), 1))
      x = x / float(n_vocab)
      prediction = model.predict(x, verbose=0)
      index = numpy.argmax(prediction)
      result = int_to_char[index]
      seq_in = [int_to_char[value] for value in pattern]
      sys.stdout.write(result)
      pattern.append(index)
      pattern = pattern[1:len(pattern)]
  
  print "\nDone."
  </code></pre>
  
  <p>Learning File:</p>
  
  <pre><code># Learn Sentences
  import numpy
  from keras.models import Sequential
  from keras.layers import Dense
  from keras.layers import Dropout
  from keras.layers import LSTM
  from keras.callbacks import ModelCheckpoint
  from keras.utils import np_utils
  
  
  
  filename = "king_lear.txt"
  raw_text = open(filename).read()
  raw_text = raw_text.lower()
  
  chars = sorted(list(set(raw_text)))
  char_to_int = dict((c, i) for i, c in enumerate(chars))
  
  n_chars = len(raw_text)
  n_vocab = len(chars)
  print "Total Characters: ", n_chars
  print "Total Vocab: ", n_vocab
  
  seq_length = 100
  dataX = []
  dataY = []
  for i in range(0, n_chars - seq_length, 1):
      seq_in = raw_text[i:i + seq_length]
      seq_out = raw_text[i + seq_length]
      dataX.append([char_to_int[char] for char in seq_in])
      dataY.append(char_to_int[seq_out])
  n_patterns = len(dataX)
  print "Total Patterns: ", n_patterns
  
  X = numpy.reshape(dataX, (n_patterns, seq_length, 1))
  
  X = X / float(n_vocab)
  
  y = np_utils.to_categorical(dataY)
  
  model = Sequential()
  model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2]), return_sequences=True))
  model.add(Dropout(0.2))
  model.add(LSTM(256))
  model.add(Dropout(0.2))
  model.add(Dense(y.shape[1], activation='softmax'))
  model.compile(loss='categorical_crossentropy', optimizer='adam')
  
  filepath="weights-improvement-{epoch:02d}-{loss:.4f}-bigger.hdf5"
  checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
  callbacks_list = [checkpoint]
  
  model.fit(X, y, epochs=50, batch_size=64, callbacks=callbacks_list)
  </code></pre>
  
  <p>Please help me to do this. If there is anything wrong with this post, don't hesitate to correct me as it is my first question. 
  Thank you very much.</p>