title: Non-linear multivariate time-series response prediction using RNN
body: <p>I am trying to predict the hygrothermal response of a wall, given the interior and exterior climate. Based on literature research, I believe this should be possible with RNN but I have not been able to get good accuracy.</p>
  
  <p>The dataset has 12 input features (time-series of exterior and interior climate data) and 10 output features (time-series of hygrothermal response), both containing hourly values for 10 years. This data was created with hygrothermal simulation software, there is no missing data. </p>
  
  <p>Dataset features:
  <a href="https://i.stack.imgur.com/vFRhx.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/vFRhx.jpg" alt="Input features"></a></p>
  
  <p>Dataset targets:
  <a href="https://i.stack.imgur.com/LDEQQ.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/LDEQQ.jpg" alt="Output features"></a></p>
  
  <p>Unlike most time-series prediction problems, I want to predict the response for the full length of the input features time-series at each time-step, rather than the subsequent values of a time-series (eg financial time-series prediction). I have not been able to find similar prediction problems (in similar or other fields), so if you know of one, references are very welcome.</p>
  
  <hr>
  
  <p>I think this should be possible with RNN, so I am currently using LSTM from Keras. Before training, I preprocess my data the following way:</p>
  
  <ol>
  <li>Discard first year of data, as the first time steps of the hygrothermal response of the wall is influenced by the initial temperature and relative humidity.</li>
  <li>Split into training and testing set. Training set contains the first 8 years of data, the test set contains the remaining 2 years.</li>
  <li>Normalise training set (zero mean, unit variance) using <code>StandardScaler</code> from Sklearn. Normalise test set analogously using mean an variance from training set.</li>
  </ol>
  
  <p>This results in: <code>X_train.shape = (1, 61320, 12)</code>, <code>y_train.shape = (1, 61320, 10)</code>, <code>X_test.shape = (1, 17520, 12)</code>, <code>y_test.shape = (1, 17520, 10)</code></p>
  
  <p>As these are long time-series, I use stateful LSTM and cut the time-series as explained <a href="https://github.com/ahstat/deep-learning/blob/master/rnn/4_lagging_and_stateful.py" rel="noreferrer">here</a>, using the <code>stateful_cut()</code> function. I only have 1 sample, so <code>batch_size</code> is 1. For <code>T_after_cut</code> I have tried 24 and 120 (24*5); 24 appears to give better results. This results in <code>X_train.shape = (2555, 24, 12)</code>, <code>y_train.shape = (2555, 24, 10)</code>, <code>X_test.shape = (730, 24, 12)</code>, <code>y_test.shape = (730, 24, 10)</code>.</p>
  
  <p>Next, I build and train the LSTM model as follows:</p>
  
  <pre><code>model = Sequential()
  model.add(LSTM(128, 
                 batch_input_shape=(batch_size,T_after_cut,features), 
                 return_sequences=True,
                 stateful=True,
                 ))
  model.addTimeDistributed(Dense(targets)))
  model.compile(loss='mean_squared_error', optimizer=Adam())
  
  model.fit(X_train, y_train, epochs=100, batch_size=batch=batch_size, verbose=2, shuffle=False)
  </code></pre>
  
  <p>Unfortunately, I don't get accurate prediction results; not even for the training set, thus the model has high bias. </p>
  
  <p><a href="https://i.stack.imgur.com/5hSVV.jpg" rel="noreferrer">The prediction results of the LSTM model for all targets</a></p>
  
  <hr>
  
  <p>How can I improve my model? I have already tried the following:</p>
  
  <ol>
  <li>Not discarding the first year of the dataset -> no significant difference</li>
  <li>Differentiating the input features time-series (subtract previous value from current value) -> slightly worse results</li>
  <li>Up to four stacked LSTM layers, all with the same hyperparameters -> no significant difference in results but longer training time</li>
  <li>Dropout layer after LSTM layer (though this is usually used to reduce variance and my model has high bias) -> slightly better results, but difference might not be statistically significant</li>
  </ol>
  
  <p>Am I doing something wrong with the stateful LSTM? Do I need to try different RNN models? Should I preprocess the data differently?</p>
  
  <p>Furthermore, training is very slow: about 4 hours for the model above. Hence I am reluctant to do an extensive hyperparameter gridsearch...</p>