# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: U4-S3-DNN (Python3)
#     language: python
#     name: u4-s3-dnn
# ---

# <img align="left" src="https://lever-client-logos.s3.amazonaws.com/864372b1-534c-480e-acd5-9711f850815c-1524247202159.png" width=200>
# <br></br>
# <br></br>
#
# ## *Data Science Unit 4 Sprint 3 Assignment 1*
#
# # Recurrent Neural Networks and Long Short Term Memory (LSTM)
#
# ![Monkey at a typewriter](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Chimpanzee_seated_at_typewriter.jpg/603px-Chimpanzee_seated_at_typewriter.jpg)
#
# It is said that [infinite monkeys typing for an infinite amount of time](https://en.wikipedia.org/wiki/Infinite_monkey_theorem) will eventually type, among other things, the complete works of Wiliam Shakespeare. Let's see if we can get there a bit faster, with the power of Recurrent Neural Networks and LSTM.
#
# This text file contains the complete works of Shakespeare: https://www.gutenberg.org/files/100/100-0.txt
#
# Use it as training data for an RNN - you can keep it simple and train character level, and that is suggested as an initial approach.
#
# Then, use that trained RNN to generate Shakespearean-ish text. Your goal - a function that can take, as an argument, the size of text (e.g. number of characters or lines) to generate, and returns generated text of that size.
#
# Note - Shakespeare wrote an awful lot. It's OK, especially initially, to sample/use smaller data and parameters, so you can have a tighter feedback loop when you're trying to get things running. Then, once you've got a proof of concept - start pushing it more!
#
# Resource: 
# - https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/

# + colab={} colab_type="code" id="Ltj1je1fp5rO"
with open('./data/shakespeare.txt', encoding='utf-8') as file:
  text = file.read().lower()
print('text length:', len(text))
# -

# Take a look at the first 250 characters in text
print(text[:250])

# +
# get unique chars only

import numpy as np
chars = list(set(text))

num_chars = len(chars)
txt_data_size = len(text)

print("unique characters : ", num_chars)
print("txt_data_size : ", txt_data_size)

# +
# before training, we need to map strings to a numerical rep
# create two dicts/tables 
# one maping characters to numbers, 

# one hot encode
char_to_int = dict((c, i) for i, c in enumerate(chars)) # "enumerate" retruns index and value. Convert it to dictionary
int_to_char = dict((i, c) for i, c in enumerate(chars))
print(char_to_int)
print("----------------------------------------------------")
print(int_to_char)
print("----------------------------------------------------")
# integer encode input data
integer_encoded = [char_to_int[i] for i in text] # "integer_encoded" is a list which has a sequence converted from an original data to integers.
print(integer_encoded)
print("----------------------------------------------------")
print("data length : ", len(integer_encoded))

# +
# prepare the dataset of input to output pairs encoded as integers
# we must split the data up into subseqences with a set length

# this sets the max num of chars for each subsequence
# recall: txt_data_size already determined
seq_length = 100

dataX = []
dataY = []

for i in range(0, txt_data_size - seq_length, 1): # 0, 5584464 - 100 , 1
  seq_in = text[i:i + seq_length] # 0:0 + 100
  seq_out = text[i + seq_length] # 0 + 100
  
  dataX.append([char_to_int[char] for char in seq_in])
  dataY.append(char_to_int[seq_out])

  
# this now tells us that we have X number of training patterns
n_patterns = len(dataX)
print('Total patters:', n_patterns) 

# +
# now that our training data is prepared
# we have to transform it so it is suitable for use with Keras


# 1. reshape X
X = np.reshape(dataX, (n_patterns, seq_length, 1))
# -

X 

# 2. normalize to make it easier
# makes the patterns easier to learn by the LSTM network that uses 
# the sigmoid activation function by default.
X = X / float(num_chars)

# +
from keras.utils import np_utils

# 3. one hot encode the output variable
# to predict the probability of each of the unique chars
# rather than force it to predict pricisely the next character
y = np_utils.to_categorical(dataY)
y 
# -



# +
# create the LSTM model using the ENTIRE dataset

# couple things:
# - we defined a single hidden LSTM layer with 128 memory units
# - dropout prob of 20
# - output is a Dense layer, activation is softmax to ouput a prob pred for each of the unique chars

# think of the problem as a single char classification problem
# with [unique chars] number of classes
# as such, optimizing the log loss here using Adam for speed

from keras.models import Sequential
from keras.layers import Dense, LSTM, Activation, Dropout
from keras.optimizers import Adam

model = Sequential()
model.add(LSTM(128, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
print(model.summary())
# -

# fit model to data
history = model.fit(X, y, epochs=5, batch_size=128, verbose=True)

plt.plot(history.history['loss'])
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train'], loc='upper left')
plt.show()

# ## Create predictions

# +
# 1. pick a random starting subsequence as input
start = np.random.randint(0, len(dataX)-1)
pattern = dataX[start]

print('Start:', start)
print('Pattern:', pattern)
print('Seed sequence:', "\"", ''.join([int_to_char[value] for value in pattern]), "\"")
# -

# create output with the starting seed sequence
out = [int_to_char[value] for value in pattern]
print(out)

# +
# 2. then generate the next char
# then update the seed seq to add the generated char to the end
# delete the first char

# basically, putting all the above code together into a loop
import sys

for i in range(1000):
    # transform for use in Keras & normalize
    x = np.reshape(pattern, (1, len(pattern), 1))
    x = x / float(num_chars)
    
    # predict & get index location as int
    prediction = model.predict(x, verbose=0)
    index = np.argmax(prediction)
    
    # convert the index int to a char
    result = int_to_char[index]
    
    # continue along seq in
    seq_in = [int_to_char[value] for value in pattern]
    
    # append the predicted char to the output/generated text
    out.append(result)
    
    # update pattern
    pattern.append(index)
    pattern = pattern[1:len(pattern)]
    
print("\nDone")
# -

print('Generated Text:', out)

# + [markdown] colab_type="text" id="zE4a4O7Bp5x1"
# # Resources and Stretch Goals

# + [markdown] colab_type="text" id="uT3UV3gap9H6"
# ## Stretch goals:
# - Refine the training and generation of text to be able to ask for different genres/styles of Shakespearean text (e.g. plays versus sonnets)
# - Train a classification model that takes text and returns which work of Shakespeare it is most likely to be from
# - Make it more performant! Many possible routes here - lean on Keras, optimize the code, and/or use more resources (AWS, etc.)
# - Revisit the news example from class, and improve it - use categories or tags to refine the model/generation, or train a news classifier
# - Run on bigger, better data
#
# ## Resources:
# - [The Unreasonable Effectiveness of Recurrent Neural Networks](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) - a seminal writeup demonstrating a simple but effective character-level NLP RNN
# - [Simple NumPy implementation of RNN](https://github.com/JY-Yoon/RNN-Implementation-using-NumPy/blob/master/RNN%20Implementation%20using%20NumPy.ipynb) - Python 3 version of the code from "Unreasonable Effectiveness"
# - [TensorFlow RNN Tutorial](https://github.com/tensorflow/models/tree/master/tutorials/rnn) - code for training a RNN on the Penn Tree Bank language dataset
# - [4 part tutorial on RNN](http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/) - relates RNN to the vanishing gradient problem, and provides example implementation
# - [RNN training tips and tricks](https://github.com/karpathy/char-rnn#tips-and-tricks) - some rules of thumb for parameterizing and training your RNN

StarcoderdataJupyterScriptsDedupFiltered