Training an LSTM to track sine waves I'm experimenting (read: playing around) with LSTMs on Keras. 
I want to train an LSTM network so it would "track" sine waves, that is, given sine waves with different wave length, phases and lengths, it would output the rest of wave. 
In a sense, a many-to-many problem.

I decided that the network would need to "track" 5 sine waves simultaneously. 
I would generate random sine waves and use the 50 last data points as the required output of the NN.

My code is a follows:

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed,Lambda, Dropout
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(5, return_sequences=True))
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values



print(model.summary())

model.compile(loss='mean_squared_error',
              optimizer='adam')


def make_line(length):
    shift= np.random.random()
    wavelength = 5+10*np.random.random()
    a=np.arange(length)
    answer=np.sin(a/wavelength+shift)
    return answer




def make_data(seq_num,seq_len,dim):
    data=np.array([]).reshape(0,seq_len,dim)
    for i in range (seq_num):
        mini_data=np.array([]).reshape(0,seq_len)
        for j in range (dim):
            line = make_line(seq_len)
            line=line.reshape(1,seq_len)            
            mini_data=np.append(mini_data,line,axis=0)
        mini_data=np.swapaxes(mini_data,1,0)
        mini_data=mini_data.reshape(1,seq_len,dim)      
        data=np.append(data,mini_data,axis=0)
    return (data)



def train_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50     
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:] # all but last 50      
        y_train = (data[:, -50:, :]) # last 50      
        yield x_train, y_train


model.fit_generator(train_generator(), steps_per_epoch=30, epochs=100, verbose=1,validation_data=train_generator(),validation_steps=30)
model.save('vi2h_model.h5')


I then try to generate data and use the model to predict the continuation of a sine wave:

def data_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:]    
        y_train = (data[:, -50:, :])                
        return x_train, y_train


x,y= data_generator()   

model = load_model('vi2h_model.h5')
pred=model.predict (x)
np.save ('pred.npy',pred)
np.save ('y.npy',y)
np.save ('x.npy',x)


When inspecting the ability of the network to predict the continuation of a sine wave (comparing y[0,:,0] to pred[0,:,0] for example, in order to compare the first sine wave of the first generated datum), see that my network performed very badly:

 

(Edit: legend: The predicted vs actual 50 last points in a sample sine wave (blue and orange, respectively), placed after the part of the wave used as input for the NN)

What did I do wrong? And how should I my code to correctly track sine waves?
Many thanks!
I have added a dense layer to the network:

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(5)))  # &lt;--- This is new :)
model.add(Activation('linear')) 
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values


This has provided some improvement :



These are two sine waves and the predictions of the last 50 data points of them.
As can be seen, the results have improved, with some of them being okay approximations of the sine waves. Other times... not so much.

Still a lot better than the previous results.

I then tried another approach: Instead of having one (multilayered) LSTM, in which I ignore all but the 50 inputs (in a sense trying to predict from each one of the 50 last values the value located 50 places after it... which might not be an ideal approach) I split the network into 2 LSTMs, with one receiving the input and "injecting" its results via a dense layer into the second LTSM layer which generates the output.

features_num=5 

model.add(LSTM(40, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(40, return_sequences=False))

model.add(Dense(80))
model.add(Activation('tanh')) 
model.add(RepeatVector(50)) # 50 output points

model.add(LSTM(40, return_sequences=True))
model.add(LSTM(40, return_sequences=True))

model.add(TimeDistributed(Dense(features_num)))
model.add(Activation('linear')) 


Results are, for the most part, better: