# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: "\u201CDeepLearning\u201D"
#     language: python
#     name: dlr
# ---

# # Character level language model

# In this notebook we demonstrate the power of recurrent neural networks in learning a character level language model.
#
# In the example below we  use a list of dutch cities as input and we  generate new city names by learning the character level patterns in the existing names. The model generates new sequences of characters using the patterns in the input sequence.
#
#
# We will use tensorflow to create the model. The below architecture can be used to learn the patterns in other sequences too. Feel free to experiment with your own dataset and modify the architecture of the neural network.

# ### What is a Recurrent Neural Network?

# A recurrent neural network is a neural network that has an internal state/memory. This internal state stores information on the previous examples the cell has seen, and may affect the outcome the cell will output at the next timestep. As the internal state updated through the training, RNNs are very efficient in learning patterns in sequences of data. RNNs are often used when the order of the values in the dataset matters and has importance in finding a value of a given sample. 
#
# A classic use case  is a timeseries of stock market data. A naive example is an indicator that gives a prediction whether a value of a stock will go up or down based on the current change of the stock value and changes in  the past. This means the indicator needs to be able to define long term patterns in the sequence. 
#
# Another popular use case is sentiment analysis. In order to define the sentiment of a sentence(positive, negative, neutral) it is essential to learn dependencies between the words. e.g.: "I am happy" - "I am not happy". The word "happy" has a positive sentiment but used with the word "not" makes it negative.
#
# Sequence to sequence transformations such as language translation and speech recognition are other examples where RNNs have been applied with a great success.
#

# The figure below represents an abstract overview of how an RNN works. At timestep t X<sub>t</sub> is fed into the network that will output  some h<sub>t</sub> and also update its internal states based on the value it has seen. We move on to timestep t+1 and the process is repeated. 
#

# <img width="10%" src="rnn_rolled.png"></img>
# <caption>Recurrent Neural Network src: http://colah.github.io/posts/2015-08-Understanding-LSTMs/</caption>

# The figure below is visualisation of an RNN network unrolled. A basic RNN cell has two inputs:
# * X<sub>t</sub> - our data at timestep t
# * S<sub>t-1</sub> - the state of the network at timestep t-1

# <img src="rnn_unrolled.png"></img>
# <caption>Recurrent Neural Network src: http://colah.github.io/posts/2015-08-Understanding-LSTMs/</caption>

# In the example below we will be working with an LSTM(Long Short Term Memory)  cell. 

# An Long Short Term Memory has the following inputs:
#     * Hidden state: hidden cell state of the RNN at timestep t-1
# $$ a^{\langle t-1 \rangle} $$
#     * Cell state: cell state of the RNN at timestep t-1
# $$ c^{\langle t-1 \rangle} $$
# the following gates:
#     * Forget gate: how much of the cell state coming from the previous time step the network will ignore
# $$\Gamma_f^{\langle t \rangle} = \sigma(W_f[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_f)\tag{1} $$
#     * Keep gate: how much of the new proposed cell state the network will keep
# $$\Gamma_u^{\langle t \rangle} = \sigma(W_u[a^{\langle t-1 \rangle}, x^{\{t\}}] + b_u)\tag{2} $$
#     * Output gate: how much of the calculated output will be returned
# $$ \Gamma_o^{\langle t \rangle}=  \sigma(W_o[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_o)\tag{5}$$ 
# performs the following calculations:
#     * Proposed cell state: calculated from the hidden state of the previous step and x
# $$ \tilde{c}^{\langle t \rangle} = \tanh(W_c[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_c)\tag{3} $$
#     * New cell state: element wise product of forget gate and the last cell state +  element wise product of forget gate and proposed cell state
# $$ c^{\langle t \rangle} = \Gamma_f^{\langle t \rangle}* c^{\langle t-1 \rangle} + \Gamma_u^{\langle t \rangle} *\tilde{c}^{\langle t \rangle} \tag{4} $$

# To get a good understanding of how LSTM works I would recommend <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/"> colah's blog.</a> 

# ##### Import packages

import numpy as np
import pandas as pd
import tensorflow as tf

# ### 1 - Reading input dataset

# First we import the dataset, and count the number of characters that will be our vocabulary size. The goal is to train an RNN network that given an input sequence (t, t+1, t+2.... t+n) will return a probability distribution for the value of t+n+1. We consider every character as a class and our neural network will be optimized to classify an input sequence.

file = 'nl_cities.csv'
data = open(file, 'r').read().lower()
chars = list(set(data))
num_classes = vocab_size = len(chars)

with open(file) as f:
    datarows = f.readlines()
datarows = [x.lower().strip() for x in datarows]
np.random.shuffle(datarows)

print("Number of characters in dataset", vocab_size)

# ##### Creating a map from a character to a class id, and a map from a class id to a character

# Our network will take one-hot encodings as an input at every time step. A one-hot encoding is a representation that makes training of the network easier.
# Imagine there are four classes: {0, 1, 2, 3} and our dataset is the following: <br>
# 2 <br>
# 2 <br>
# 1 <br>
# 3 <br>
# Their one-hot encoding would like the following: <br>
# [0, 0, 1, 0] <br>
# [0, 0, 1, 0] <br>
# [0, 1, 0, 0] <br>
# [0, 0, 0, 1] <br>

# We create two dictionaries to help us during the implementation: </br>
# * a dictionary from class_id to character
# * and a dictionary from character to class_id

char_to_class_id = { ch:i for i,ch in enumerate(sorted(chars)) }
class_id_to_char = { i:ch for i,ch in enumerate(sorted(chars)) }


# ### 2 - Building the model

# In tensorflow first you define the graph of your model then you execute it by running it in a session.
# When working with tensorflow I would recommend the following steps:
# 1. Define your input and target placeholders.
# 2. Build up the graph of your model.
# 3. Define a cost function that will measure how well your model performs against your objectives
# 4. Define an optimizer to minimize your cost function
# 5. Execute your model by feeding in the input data and target data to the placeholders
#

# #### Create placeholders for input and target data

# In tensorflow placeholders are used to define data sources to your graph.
# Our graph model will have two datasource:
#     * X - a number of sequences with a maximum length of max_timesteps
#     * Y - same as X shifted by to the right

def create_placeholders(length, batch_size=1):
    """
    Creates the placeholder for the input date and the target data.
    batch_size - Number of data rows, default to 1
    max_timesteps - The maximum length of sequence
    
    return X,Y placeholders
    """
    X = tf.placeholder(tf.int32, [batch_size, length])
    Y = tf.placeholder(tf.int32, [batch_size, length])
    
    return X,Y    


# ##### Create model - forward propagation

def create_graph(X, Y, state_size, num_classes, batch_size):
    """
    Constructs the graph of the model
    Creates the placeholder for the hidden state and cell state
    batch_size - Number of data rows
    state_size - state_size of the RNN Unit
    num_classes - number of classes we are predicting
    parameters - matrices containing weights
    
    return cell_state, hidden_state, current_state, predictions, total_loss
    """
    #Create one hot representaion from input X placeholder
    inputs_series = tf.one_hot(X, num_classes)
    
    #Create lstm_cell
    lstm_cell = tf.contrib.rnn.BasicLSTMCell(state_size, state_is_tuple=True)
    
    #Create placeholder for cell state and hidden state
    cell_state = tf.placeholder(tf.float32, [batch_size, state_size])
    hidden_state = tf.placeholder(tf.float32, [batch_size, state_size])
    #LSTMStateTuple represent the state of the cell
    rnn_tuple_state = tf.nn.rnn_cell.LSTMStateTuple(cell_state, hidden_state)
    
    #Unroll the cell to a max_length connected rnn cells
    #The length of the timeseries is dynamically determined during runtime, as every datarow has different length
    outputs, current_state = tf.nn.dynamic_rnn(lstm_cell,
                                               inputs_series,
                                               initial_state=rnn_tuple_state,dtype=tf.float32)
    #Determine length of sequence
    length = tf.shape(X)[1]
    
    #outputs will have a shape of batch_size X state_size
    #we define and out matrix of shape state_size, num_classes
    # outputs * out_weight will result in an output of the desired shape
    out_weight = tf.get_variable('out_weight', [state_size, num_classes])
    out_bias = tf.get_variable('out_bias', [num_classes])


    logits = tf.reshape(tf.matmul(tf.reshape(outputs, [-1, state_size]), out_weight) + out_bias,[batch_size, length, num_classes])
    
    #Create prediction for sampling purposes
    predictions = tf.nn.softmax(logits)
    #Calculate loss
    losses = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=Y, logits=logits)
    #Calculate total loss as average over timesteps
    total_loss = tf.reduce_mean(losses)
    
    return cell_state, hidden_state, current_state, predictions, total_loss


def run(training_steps=35000, learning_rate=0.05, state_size=5, sample_size=10, every=2000, batch_size=1):
    assert batch_size == 1, "batch_size greater then 1 not supported yet"
    # Get placeholders for X and Y
    # We will leave max_timesteps undefined, it will be evaluated during runtime
    X, Y = create_placeholders(None)
    #
    cell_state, hidden_state, current_state, predictions, total_loss = create_graph(X, Y, state_size, num_classes, batch_size)
    
    #Define train step
    train_step = tf.train.AdamOptimizer(learning_rate).minimize(total_loss)
    with tf.Session() as sess:
        #Initialize variables
        init = tf.global_variables_initializer()
        sess.run(init)        
        
        for step in range(1, training_steps+1):
            #Zero Initialize the hidden and cell state of the lstm
            _current_state = np.zeros((2, batch_size, state_size))
            row_index = step % len(datarows)

            X_train = [-1] + [char_to_class_id[ch] for ch in datarows[row_index]] 
            Y_train = X_train[1:] + [char_to_class_id["\n"]]

            # Reshape data to get 1x28 shaped element
            batch_x = np.expand_dims(np.array(X_train), axis=0)
            batch_y = np.expand_dims(np.array(Y_train), axis=0)
            
            
            
            cost, _current_state, _ = sess.run([total_loss, current_state, train_step],
                                                           feed_dict={
                                                                       X: batch_x,
                                                                       Y: batch_y,
                                                                       cell_state: _current_state[0],
                                                                       hidden_state: _current_state[1]})
                
            #Print Loss and sample from trained grapd
            if step % every == 0:
                print("Step " + str(step) + ", Loss= " + \
                      "{:.4f}".format(cost))
                print("list of sampled characters:")
                for sample in range(sample_size):
                    _current_sample_state = np.zeros((2, batch_size, state_size))
                    #sample a prediction
                    idx = -1
                    newline_character = char_to_class_id['\n']
                    counter = 0
                    indices = []
                    X_eval = [-1]
                    X_eval = np.expand_dims(np.array(X_eval), axis=0)
                    while (idx != newline_character and counter != 50):
#                         np.random.seed(counter+sample) 
                        pred_out, _current_sample_state = sess.run([predictions, current_state],
                                                                   feed_dict={
                                                                    X: X_eval, 
                                                                    cell_state: _current_sample_state[0],
                                                                    hidden_state: _current_sample_state[1]})
                        pred_probs = pred_out[0][0]

                        #Sample a character using the output probability distribution
                        idx = np.random.choice(np.arange(0,vocab_size), p = pred_out.ravel())
                        #Append sampled character to a list
                        character = class_id_to_char[idx]
                        indices.append(idx)
                        #set sampled characted as an input in the next timestep
                        X_eval = [idx]
                        X_eval = np.expand_dims(np.array(X_eval), axis=0)
                        counter += 1
                    print(''.join([class_id_to_char[i] for i in indices]).strip())
                
        print("Optimization Finished!")
    return sess

run()



StarcoderdataJupyterScriptsDedupFiltered