# Text generation with an RNN

This tutorial demonstrates how to generate text using a character-based RNN. We will work with a dataset of Shakespeare's writing from Andrej Karpathy's The Unreasonable Effectiveness of Recurrent Neural Networks. Given a sequence of characters from this data ("Shakespear"), train a model to predict the next character in the sequence ("e"). Longer sequences of text can be generated by calling the model repeatedly.

This tutorial includes runnable code implemented using tf.keras and eager execution. The following is sample output when the model in this tutorial trained for 30 epochs, and started with the string "Q":

QUEENE:
Thus by All bids the man against the word,
Which are so weak of care, by old care done;
And the precipitation through the bleeding throne.

BISHOP OF ELY:
Marry, and will, my lord, to weep in such a one were prettiest;
Yet now I was adopted heir
Of the world's lamentable day,
To watch the next way with his father with his face?

ESCALUS:
The cause why then we are all resolved more sons.

VOLUMNIA:
O, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, it is no sin it should be dead,
And love and pale as any will to that word.

QUEEN ELIZABETH:
But how long have I heard the soul for this world,
And show his hands of life be proved to stand.

PETRUCHIO:
I say he look'd on, if I must be content
To stay him from the fatal of our country's bliss.
His lordship pluck'd from this sentence then for prey,
And then let us twain, being the moon,
were she such a case as fills m


While some of the sentences are grammatical, most do not make sense. The model has not learned the meaning of words, but consider:

• The model is character-based. When training started, the model did not know how to spell an English word, or that words were even a unit of text.

• The structure of the output resembles a play—blocks of text generally begin with a speaker name, in all capital letters similar to the dataset.

• As demonstrated below, the model is trained on small batches of text (100 characters each), and is still able to generate a longer sequence of text with coherent structure.

## Setup

### Import TensorFlow and other libraries

import tensorflow as tf

import numpy as np
import os
import time


Change the following line to run this code on your own data.

path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')

Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt
1122304/1115394 [==============================] - 0s 0us/step



First, look in the text:

# Read, then decode for py2 compat.
# length of text is the number of characters in it
print ('Length of text: {} characters'.format(len(text)))

Length of text: 1115394 characters


# Take a look at the first 250 characters in text
print(text[:250])

First Citizen:
Before we proceed any further, hear me speak.

All:
Speak, speak.

First Citizen:
You are all resolved rather to die than to famish?

All:
Resolved. resolved.

First Citizen:
First, you know Caius Marcius is chief enemy to the people.


# The unique characters in the file
vocab = sorted(set(text))
print ('{} unique characters'.format(len(vocab)))

65 unique characters



## Process the text

### Vectorize the text

Before training, we need to map strings to a numerical representation. Create two lookup tables: one mapping characters to numbers, and another for numbers to characters.

# Creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(vocab)}
idx2char = np.array(vocab)

text_as_int = np.array([char2idx[c] for c in text])


Now we have an integer representation for each character. Notice that we mapped the character as indexes from 0 to len(unique).

print('{')
for char,_ in zip(char2idx, range(20)):
print('  {:4s}: {:3d},'.format(repr(char), char2idx[char]))
print('  ...\n}')

{
'\n':   0,
' ' :   1,
'!' :   2,



## Train the model

At this point the problem can be treated as a standard classification problem. Given the previous RNN state, and the input this time step, predict the class of the next character.

### Attach an optimizer, and a loss function

The standard tf.keras.losses.sparse_categorical_crossentropy loss function works in this case because it is applied across the last dimension of the predictions.

Because our model returns logits, we need to set the from_logits flag.

def loss(labels, logits):
return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)

example_batch_loss  = loss(target_example_batch, example_batch_predictions)
print("Prediction shape: ", example_batch_predictions.shape, " # (batch_size, sequence_length, vocab_size)")
print("scalar_loss:      ", example_batch_loss.numpy().mean())

Prediction shape:  (64, 100, 65)  # (batch_size, sequence_length, vocab_size)
scalar_loss:       4.176197



Configure the training procedure using the tf.keras.Model.compile method. We'll use tf.keras.optimizers.Adam with default arguments and the loss function.

model.compile(optimizer='adam', loss=loss)


### Configure checkpoints

Use a tf.keras.callbacks.ModelCheckpoint to ensure that checkpoints are saved during training:

# Directory where the checkpoints will be saved
checkpoint_dir = './training_checkpoints'
# Name of the checkpoint files
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")

checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(
filepath=checkpoint_prefix,
save_weights_only=True)


### Execute the training

To keep training time reasonable, use 10 epochs to train the model. In Colab, set the runtime to GPU for faster training.

EPOCHS=10

history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])

Epoch 1/10
172/172 [==============================] - 5s 28ms/step - loss: 2.6474
Epoch 2/10
172/172 [==============================] - 5s 27ms/step - loss: 1.9480
Epoch 3/10
172/172 [==============================] - 5s 27ms/step - loss: 1.6822
Epoch 4/10
172/172 [==============================] - 5s 27ms/step - loss: 1.5370
Epoch 5/10
172/172 [==============================] - 5s 27ms/step - loss: 1.4514
Epoch 6/10
172/172 [==============================] - 5s 27ms/step - loss: 1.3921
Epoch 7/10
172/172 [==============================] - 5s 27ms/step - loss: 1.3464
Epoch 8/10
172/172 [==============================] - 5s 27ms/step - loss: 1.3089
Epoch 9/10
172/172 [==============================] - 5s 27ms/step - loss: 1.2737
Epoch 10/10
172/172 [==============================] - 5s 27ms/step - loss: 1.2414



## Generate text

### Restore the latest checkpoint

To keep this prediction step simple, use a batch size of 1.

Because of the way the RNN state is passed from timestep to timestep, the model only accepts a fixed batch size once built.

To run the model with a different batch_size, we need to rebuild the model and restore the weights from the checkpoint.

tf.train.latest_checkpoint(checkpoint_dir)

'./training_checkpoints/ckpt_10'

model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)

model.build(tf.TensorShape([1, None]))

model.summary()

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_1 (Embedding)      (1, None, 256)            16640
_________________________________________________________________
gru_1 (GRU)                  (1, None, 1024)           3938304
_________________________________________________________________
dense_1 (Dense)              (1, None, 65)             66625
=================================================================
Total params: 4,021,569
Trainable params: 4,021,569
Non-trainable params: 0
_________________________________________________________________



### The prediction loop

The following code block generates the text:

• It Starts by choosing a start string, initializing the RNN state and setting the number of characters to generate.

• Get the prediction distribution of the next character using the start string and the RNN state.

• Then, use a categorical distribution to calculate the index of the predicted character. Use this predicted character as our next input to the model.

• The RNN state returned by the model is fed back into the model so that it now has more context, instead than only one character. After predicting the next character, the modified RNN states are again fed back into the model, which is how it learns as it gets more context from the previously predicted characters.

Looking at the generated text, you'll see the model knows when to capitalize, make paragraphs and imitates a Shakespeare-like writing vocabulary. With the small number of training epochs, it has not yet learned to form coherent sentences.

def generate_text(model, start_string):
# Evaluation step (generating text using the learned model)

# Number of characters to generate
num_generate = 1000

# Converting our start string to numbers (vectorizing)
input_eval = [char2idx[s] for s in start_string]
input_eval = tf.expand_dims(input_eval, 0)

# Empty string to store our results
text_generated = []

# Low temperatures results in more predictable text.
# Higher temperatures results in more surprising text.
# Experiment to find the best setting.
temperature = 1.0

# Here batch size == 1
model.reset_states()
for i in range(num_generate):
predictions = model(input_eval)
# remove the batch dimension
predictions = tf.squeeze(predictions, 0)

# using a categorical distribution to predict the character returned by the model
predictions = predictions / temperature
predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()

# We pass the predicted character as the next input to the model
# along with the previous hidden state
input_eval = tf.expand_dims([predicted_id], 0)

text_generated.append(idx2char[predicted_id])

return (start_string + ''.join(text_generated))

print(generate_text(model, start_string=u"ROMEO: "))

ROMEO: Jesuehousady, having
greens, let's we move, and be
prosed that love I passing him!

HENRY BOLINGBROKE:
Master little I have in arms.

Shepherd:
You live love; stand fortent, he thanks?
If I did stop me none; and prayed was' not to gue applain
And fill it once. perour will Nature understand'st.

KING HENRY VI:
Has
His prometimes in speak ip, and speeds me to their betterer class.
Whato you then away. Nay, constable.

ANGELO:
Alonest of yourself; it ce found most traitor,
Even in twenty court-shining, sir. For what thou dislet it,
Receive it to me; heaven and march once again?

Lord:
Of his 'langually, get against the early while he chaseizes
Thwick soon conkements Isand on some poison of substifies and the best,
And to be thourtredess. What king; let me set ouf you talk off alise,
With nothing honour, gentleman!
I pray you, he not?

MERCUTIO:
What art all mun
Bride the caring wrong, were rustle a chewhing brief.

RATCABELLA:
They are they king, pretty countryman by guests
Upon the rige:



The easiest thing you can do to improve the results it to train it for longer (try EPOCHS=30).

You can also experiment with a different start string, or try adding another RNN layer to improve the model's accuracy, or adjusting the temperature parameter to generate more or less random predictions.

The above training procedure is simple, but does not give you much control.

So now that you've seen how to run the model manually let's unpack the training loop, and implement it ourselves. This gives a starting point if, for example, to implement curriculum learning to help stabilize the model's open-loop output.

We will use tf.GradientTape to track the gradients. You can learn more about this approach by reading the eager execution guide.

The procedure works as follows:

• First, reset the RNN state. We do this by calling the tf.keras.Model.reset_states method.

• Next, iterate over the dataset (batch by batch) and calculate the predictions associated with each.

• Open a tf.GradientTape, and calculate the predictions and loss in that context.

• Calculate the gradients of the loss with respect to the model variables using the tf.GradientTape.grads method.

• Finally, take a step downwards by using the optimizer's tf.train.Optimizer.apply_gradients method.

model = build_model(
vocab_size = len(vocab),
embedding_dim=embedding_dim,
rnn_units=rnn_units,
batch_size=BATCH_SIZE)

WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-0.embeddings
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-2.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-2.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.cell.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-0.embeddings
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-2.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-2.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.cell.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.


optimizer = tf.keras.optimizers.Adam()

@tf.function
def train_step(inp, target):
predictions = model(inp)
loss = tf.reduce_mean(
tf.keras.losses.sparse_categorical_crossentropy(
target, predictions, from_logits=True))

return loss

# Training step
EPOCHS = 10

for epoch in range(EPOCHS):
start = time.time()

# resetting the hidden state at the start of every epoch
model.reset_states()

for (batch_n, (inp, target)) in enumerate(dataset):
loss = train_step(inp, target)

if batch_n % 100 == 0:
template = 'Epoch {} Batch {} Loss {}'
print(template.format(epoch+1, batch_n, loss))

# saving (checkpoint) the model every 5 epochs
if (epoch + 1) % 5 == 0:
model.save_weights(checkpoint_prefix.format(epoch=epoch))

print ('Epoch {} Loss {:.4f}'.format(epoch+1, loss))
print ('Time taken for 1 epoch {} sec\n'.format(time.time() - start))

model.save_weights(checkpoint_prefix.format(epoch=epoch))

Epoch 1 Batch 0 Loss 4.174383640289307
Epoch 1 Batch 100 Loss 2.3455469608306885
Epoch 1 Loss 2.1354
Time taken for 1 epoch 6.386500358581543 sec

Epoch 2 Batch 0 Loss 2.143766403198242
Epoch 2 Batch 100 Loss 1.9540011882781982
Epoch 2 Loss 1.7758
Time taken for 1 epoch 5.211035490036011 sec

Epoch 3 Batch 0 Loss 1.7939282655715942
Epoch 3 Batch 100 Loss 1.697733759880066
Epoch 3 Loss 1.6339
Time taken for 1 epoch 5.233696222305298 sec

Epoch 4 Batch 0 Loss 1.589755654335022
Epoch 4 Batch 100 Loss 1.5560765266418457
Epoch 4 Loss 1.5129
Time taken for 1 epoch 5.385164499282837 sec

Epoch 5 Batch 0 Loss 1.4754937887191772
Epoch 5 Batch 100 Loss 1.4611799716949463
Epoch 5 Loss 1.4449
Time taken for 1 epoch 5.53707480430603 sec

Epoch 6 Batch 0 Loss 1.3802833557128906
Epoch 6 Batch 100 Loss 1.384648084640503
Epoch 6 Loss 1.3626
Time taken for 1 epoch 5.327985763549805 sec

Epoch 7 Batch 0 Loss 1.3318533897399902
Epoch 7 Batch 100 Loss 1.370686650276184
Epoch 7 Loss 1.3535
Time taken for 1 epoch 5.156204462051392 sec

Epoch 8 Batch 0 Loss 1.2854591608047485
Epoch 8 Batch 100 Loss 1.336053490638733
Epoch 8 Loss 1.3254
Time taken for 1 epoch 5.2365453243255615 sec

Epoch 9 Batch 0 Loss 1.2265124320983887
Epoch 9 Batch 100 Loss 1.280550241470337
Epoch 9 Loss 1.2670
Time taken for 1 epoch 5.247822999954224 sec

Epoch 10 Batch 0 Loss 1.187045931816101
Epoch 10 Batch 100 Loss 1.2388993501663208
Epoch 10 Loss 1.2459
Time taken for 1 epoch 5.328867673873901 sec
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.44707006216049194, "perplexity": 13129.284775374732}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2020-40/segments/1600402130615.94/warc/CC-MAIN-20201001030529-20201001060529-00202.warc.gz"}