"""
<center><h2>Cumulative Sum Prediction Problem</h2></center>
<br>
<div style="font-family:verdana; word-spacing:1.9px;">
The problem is defined as a sequence of random values between 0 and 1. This sequence is taken as input for the problem with each number provided once per time step. <br>A binary label (0 or 1) is associated with each input. The output values are all 0. Once the cumulative sum of the input values in the sequence exceeds a threshold, then the output value flips from 0 to 1.<br><br>
A threshold of one quarter (1/4) the sequence length is used. For example, below is a sequence
of 10 input time steps (X):<br><br>
    <blockquote>0.63144003 0.29414551 0.91587952 0.95189228 0.32195638 0.60742236 0.83895793 0.18023048
    0.84762691 0.29165514</blockquote><br>
    The corresponding classification output (y) would be:<br><br>
    <blockquote>0 0 0 1 1 1 1 1 1 1</blockquote><br>
</div>

![image.png](attachment:image.png)
"""
"""
<h3>The Bidirectional LSTM</h3>
<div style="font-family:verdana; word-spacing:1.9px;">
Bidirectional LSTMs focus on the problem of getting the most out of the input sequence by stepping through input time steps in both the forward and backward directions.<br> In practice, this architecture involves duplicating the first recurrent layer in the network so that there are now two layers side-by-side, then providing the input sequence as-is as input to the first layer and providing a reversed copy of the input sequence to the second. <br>This approach was developed some time ago as a general approach for improving the performance of Recurrent Neural Networks (RNNs).
</div>

![image.png](attachment:image.png)

<div style="font-family:verdana; word-spacing:1.9px;">
Bidirectional LSTMs are a small step on top of this capability.<br> Specifically, Bidirectional LSTMs are supported in Keras via the Bidirectional layer wrapper that essentially merges the output from two parallel LSTMs, one with input processed forward and one with output processed backwards.<br><br>
    The Bidirectional wrapper layer also allows you to specify the merge mode; that is how the forward and backward outputs should be combined before being passed on to the next layer. The options are:
    <ul>
        <li>‘sum’: The outputs are added together.
            <li>‘mul’: The outputs are multiplied together.
            <li>‘concat’: The outputs are concatenated together (the default), providing double the
number of outputs to the next layer.
            <li>‘ave’: The average of the outputs is taken.
    </ul>
    
 The default mode is to concatenate, and this is the method often used in studies of bidirectional LSTMs. <br>In general, it might be a good idea to test each of the merge modes on your problem to see if you can improve upon the concatenate default option.
</div>
"""
"""
<center><h3>Importing Libraries
"""
import numpy as np
from random import random
from numpy import cumsum
from numpy import array_equal

from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense
from keras.layers import TimeDistributed
from keras.layers import Bidirectional
"""
<center><h3>Sequence Generation
"""
# create a cumulative sum sequence
def get_sequence(n_timesteps):
    # create a sequence of random numbers in [0,1]
    X = np.array([random() for _ in range(n_timesteps)])
    # calculate cut-off value to change class values
    limit = n_timesteps/4.0
    # determine the class outcome for each item in cumulative sequence
    y = np.array([0 if x < limit else 1 for x in cumsum(X)])
    return X, y


# create multiple samples of cumulative sum sequences
def get_sequences(n_sequences, n_timesteps):
    seqX, seqY = list(), list()
    # create and store sequences
    for _ in range(n_sequences):
        X, y = get_sequence(n_timesteps)
        seqX.append(X)
        seqY.append(y)
    # reshape input and output for lstm
    seqX = np.array(seqX).reshape(n_sequences, n_timesteps, 1)
    seqY = np.array(seqY).reshape(n_sequences, n_timesteps, 1)
    return seqX, seqY

X, y = get_sequence(10)
print(X)
print(y)
"""
<center><h3>Modelling</h3>
"""
"""
<h3>Bidirectional
"""
n_timesteps = 10


# define LSTM
model = Sequential()
model.add(Bidirectional(LSTM(50, return_sequences = True), input_shape=(n_timesteps, 1)))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()
# train LSTM
X, y = get_sequences(50000, n_timesteps)
X_test, y_test = get_sequences(5000, n_timesteps)
model.fit(X, y, epochs=1, batch_size=10, validation_data=(X_test, y_test))
"""
<center><h3>Predicting sequences
"""
# make predictions
for _ in range(10):
    X1, y1 = get_sequences(1, n_timesteps)
    yhat = model.predict_classes(X1, verbose=0)
    exp, pred = y1.reshape(n_timesteps), yhat.reshape(n_timesteps) 
    print('y=%s, yhat=%s, correct=%s' % (exp, pred, array_equal(exp,pred)))
"""
<h3>Forward LSTM
"""
# define Forward LSTM
model = Sequential()
model.add(LSTM(50, return_sequences = True, input_shape=(n_timesteps, 1)))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()

model.fit(X, y, epochs=1, batch_size=10, validation_data=(X_test, y_test))
# make predictions
for _ in range(10):
    X1, y1 = get_sequences(1, n_timesteps)
    yhat = model.predict_classes(X1, verbose=0)
    exp, pred = y1.reshape(n_timesteps), yhat.reshape(n_timesteps) 
    print('y=%s, yhat=%s, correct=%s' % (exp, pred, array_equal(exp,pred)))
"""
<h3>Backward LSTM
"""
# define Backward LSTM
model = Sequential()
model.add(LSTM(50, return_sequences = True, input_shape=(n_timesteps, 1), go_backwards=True))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()

model.fit(X, y, epochs=1, batch_size=10, validation_data=(X_test, y_test))