title: Forecasting time series data with PyBrain Neural Networks
body: <h1>Problem</h1>
  
  <p>I am trying to use 5 years of consecutive, historical data to forecast values for the following year. </p>
  
  <h2>Data Structure</h2>
  
  <p>My input data <em>input_04_08</em> looks like this where the first column is the day of the year (1 to 365) and the second column is the recorded input.</p>
  
  <pre><code>1,2
  2,2
  3,0
  4,0
  5,0
  </code></pre>
  
  <p>My output data <em>output_04_08</em> looks like this, a single column with the recorded output on that day of the year.</p>
  
  <pre><code>27.6
  28.9
  0
  0
  0
  </code></pre>
  
  <p>I then normalise the values between 0 and 1 so the first sample given to the network would look like</p>
  
  <pre><code>Number of training patterns:  1825
  Input and output dimensions:  2 1
  First sample (input, target):
  [ 0.00273973  0.04      ] [ 0.02185273]
  </code></pre>
  
  <h1>Approach(s)</h1>
  
  <h2>Feed Forward Network</h2>
  
  <p>I have implemented the following code in PyBrain</p>
  
  <pre class="lang-python prettyprint-override"><code>input_04_08 = numpy.loadtxt('./data/input_04_08.csv', delimiter=',')
  input_09 = numpy.loadtxt('./data/input_09.csv', delimiter=',')
  output_04_08 = numpy.loadtxt('./data/output_04_08.csv', delimiter=',')
  output_09 = numpy.loadtxt('./data/output_09.csv', delimiter=',')
  
  input_04_08 = input_04_08 / input_04_08.max(axis=0)
  input_09 = input_09 / input_09.max(axis=0)
  output_04_08 = output_04_08 / output_04_08.max(axis=0)
  output_09 = output_09 / output_09.max(axis=0)
  ds = SupervisedDataSet(2, 1)
  
  for x in range(0, 1825):
      ds.addSample(input_04_08[x], output_04_08[x])
  
  n = FeedForwardNetwork()
  inLayer = LinearLayer(2)
  hiddenLayer = TanhLayer(25)
  outLayer = LinearLayer(1)
  n.addInputModule(inLayer)
  n.addModule(hiddenLayer)
  n.addOutputModule(outLayer)
  in_to_hidden = FullConnection(inLayer, hiddenLayer)
  hidden_to_out = FullConnection(hiddenLayer, outLayer)
  n.addConnection(in_to_hidden)
  n.addConnection(hidden_to_out)
  n.sortModules()
  
  trainer = BackpropTrainer(n, ds, learningrate=0.01, momentum=0.1)
  
  for epoch in range(0, 100000000): 
      if epoch % 10000000 == 0:
          error = trainer.train()  
          print 'Epoch: ', epoch
          print 'Error: ', error
  
  
  result = numpy.array([n.activate(x) for x in input_09])
  </code></pre>
  
  <p>and this gave me the following result with <strong>final error of 0.00153840123381</strong></p>
  
  <p><a href="https://i.stack.imgur.com/WQ9Tt.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WQ9Tt.png" alt="enter image description here"></a></p>
  
  <p>Admittedly, this looks good. However, having read more about LSTM (Long Short-Term Memory) neural networks, and there applicability to time series data, I am trying to build one. </p>
  
  <h2>LSTM Network</h2>
  
  <p>Below is my code</p>
  
  <pre class="lang-python prettyprint-override"><code>input_04_08 = numpy.loadtxt('./data/input_04_08.csv', delimiter=',')
  input_09 = numpy.loadtxt('./data/input_09.csv', delimiter=',')
  output_04_08 = numpy.loadtxt('./data/output_04_08.csv', delimiter=',')
  output_09 = numpy.loadtxt('./data/output_09.csv', delimiter=',')
  
  input_04_08 = input_04_08 / input_04_08.max(axis=0)
  input_09 = input_09 / input_09.max(axis=0)
  output_04_08 = output_04_08 / output_04_08.max(axis=0)
  output_09 = output_09 / output_09.max(axis=0)
  ds = SequentialDataSet(2, 1)
  
  for x in range(0, 1825):
      ds.newSequence()
      ds.appendLinked(input_04_08[x], output_04_08[x])
  
  
  fnn = buildNetwork( ds.indim, 25, ds.outdim, hiddenclass=LSTMLayer, bias=True, recurrent=True)
  trainer = BackpropTrainer(fnn, ds, learningrate=0.01, momentum=0.1)
  
  for epoch in range(0, 10000000): 
      if epoch % 100000 == 0:
          error = trainer.train()  
          print 'Epoch: ', epoch
          print 'Error: ', error
  
  result = numpy.array([fnn.activate(x) for x in input_09])
  </code></pre>
  
  <p>This results in a <strong>final error of 0.000939719502501</strong>, but this time, when I feed the test data, the output plot looks terrible.</p>
  
  <p><a href="https://i.stack.imgur.com/XmvTU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XmvTU.png" alt="enter image description here"></a></p>
  
  <h1>Possible Problems</h1>
  
  <p>I have looked around here at pretty much all the PyBrain questions, these stood out, but haven't helped me figure things out</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/24597544/training-an-lstm-neural-network-to-forecast-time-series-in-pybrain-python">Training an LSTM neural network to forecast time series in pybrain, python</a></li>
  <li><a href="https://stackoverflow.com/questions/4230853/time-series-prediction-via-neural-networks">Time Series Prediction via Neural Networks</a></li>
  <li><a href="https://stackoverflow.com/questions/3584077/time-series-forecasting-eventually-with-python">Time series forecasting (eventually with python)</a></li>
  </ul>
  
  <p>I have read a few blog posts, these helped further my understanding a bit, but obviously not enough</p>
  
  <ul>
  <li><a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">Understanding LSTM Networks</a></li>
  <li><a href="http://blog.terminal.com/demistifying-long-short-term-memory-lstm-recurrent-neural-networks/" rel="nofollow noreferrer">Demystifying LSTM neural networks</a></li>
  </ul>
  
  <p>Naturally, I have also gone through the PyBrain docs but couldn't find much to help with the sequential dataset bar <a href="http://pybrain.org/docs/tutorial/datasets.html#sequentialdataset" rel="nofollow noreferrer">here</a>.</p>
  
  <p>Any ideas/tips/direction would be welcome.</p>