title: Avoiding overfitting in LSTM
body: <p>I have written code using Keras and TensorFlow to recognize a pattern in a cyclic dataset. The thing which I worried about was overfitting and how to avoid from being overfitted. Now, from the loss value and accuracy, it seems I have become overfitted. The code is below:</p>
  
  <pre><code>#importing libraries
  import numpy as np
  import pandas as pd
  import os
  import math
  import matplotlib.pyplot as plt
  import tensorflow as tf
  from sklearn.preprocessing import MinMaxScaler
  from sklearn.model_selection import train_test_split
  from sklearn import preprocessing
  from pandas import read_csv
  from matplotlib import pyplot
  import tensorflow.keras.backend as K
  from keras.layers import Dense, Activation
  from keras.layers.recurrent import LSTM
  from keras import backend as K
  from keras.models import Sequential
  from sklearn.metrics import mean_squared_error
  from keras.layers import InputLayer
  
  # Reading dataset
  df = pd.read_excel("concate35w270.xlsx")
  df = df.astype('float32')
  df.head()
  
  
  scaler = MinMaxScaler(feature_range= (0,1))
  df = scaler.fit_transform(df)
  
  x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.25, random_state = 0)
  
  #Looking last time step
  
  def create_dataset(dataset, look_back=1):
      dataX, dataY = [], []
      for i in range(len(dataset)-look_back-1):
          a = dataset[i:(i+look_back), 0]
          dataX.append(a)
          dataY.append(dataset[i + look_back, 0])
      return np.array(dataX), np.array(dataY
  # Reshaping dataset
  # reshape into X=t and Y=t+1
  look_back = 1
  trainX, trainY = create_dataset(train, look_back)
  testX, testY = create_dataset(test, look_back)
  
  look_back = 1
  trainX, trainY = create_dataset(train, look_back)
  testX, testY = create_dataset(test, look_back)
  
  
  # reshape input to be [samples, time steps, features]
  trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
  testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
  
  
  
  
  # Network Architecture 
  
  # create and fit the LSTM network
  model = tf.keras.models.Sequential()
  model.add(tf.keras.layers.LSTM(1, input_shape=(1, look_back)))
  model.add(tf.keras.layers.Flatten())
  model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
  #model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
  model.add(tf.keras.layers.Dense(1, activation = 'sigmoid'))
  
  
  
  def coeff_determination(y_true, y_pred):
      SS_res =  K.sum(K.square( y_true-y_pred ))
      SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
      return ( 1 - SS_res/(SS_tot + K.epsilon()) )
  
  model.compile(optimizer='sgd', 
           loss='mse',
           metrics = [coeff_determination])
  
  
  
  model.fit(trainX, trainY, epochs = 30)
  
  </code></pre>
  
  <p>After starting to fit model using the training data set, I saw this massages from the machine:</p>
  
  <pre><code>Epoch 1/30
  702543/702543 [==============================] - 64s 91us/sample - loss: 0.0376 - coeff_determination: 0.4673
  Epoch 2/30
  702543/702543 [==============================] - 61s 86us/sample - loss: 0.0015 - coeff_determination: 0.9791
  Epoch 3/30
  702543/702543 [==============================] - 60s 86us/sample - loss: 0.0014 - coeff_determination: 0.9802
  Epoch 4/30
  702543/702543 [==============================] - 64s 91us/sample - loss: 0.0013 - coeff_determination: 0.9812
  Epoch 5/30
  702543/702543 [==============================] - 68s 97us/sample - loss: 0.0013 - coeff_determination: 0.9820
  Epoch 6/30
  702543/702543 [==============================] - 67s 96us/sample - loss: 0.0012 - coeff_determination: 0.9827
  Epoch 7/30
  702543/702543 [==============================] - 67s 95us/sample - loss: 0.0012 - coeff_determination: 0.9834
  </code></pre>
  
  <p>I guess I should define a penalty to avoid from overfitting, How can I do that?
  All help will be appreciated. </p>