title: Neural Network System Identification
body: <p>I am trying to implement a Neural Network to identify a Nonlinear System. I have implemented a very simple system in simulink and on the basis of examples of its input and output I would like to have the NN to mimic its behaviour. The system is the following<img src="https://i.stack.imgur.com/WfIDr.jpg" alt="enter image description here"></p>
  
  <p>The schema on the top (the one with the random number as inputs) generates the training set, while the second one (with the ramp as input) is used to check whether the NN behaves properly.
  The network is a two layer network with a hidden layer of <em>tanh</em> nodes and a linear output layer. I have implemented two versions of the network, one in which stochastic gradient descent is used (update at each step) and another one in which the update is performed in batch. At the moment I am focusing on the iterative solution.
  I have also implemented the train-validate-test procedure described in <a href="https://stackoverflow.com/questions/25889637/how-to-use-k-fold-validation-in-a-neural-network/" title="this post">this post</a>.</p>
  
  <p>Here the parameters of the network:</p>
  
  <ul>
  <li><strong>learning rate:</strong>  1e-3;</li>
  <li><strong>number of hidden units:</strong> 15</li>
  <li><strong>number of samples:</strong> 10002</li>
  <li><strong>max epochs:</strong> 1000</li>
  <li><strong>error threshold:</strong> 0.07</li>
  </ul>
  
  <p>I use the 80% of the example set to train the NN, 10% to compute the error and the remaining 10% to produce the output you can see below. However, I cannot make the network generalize the system for different types of inputs. </p>
  
  <p><img src="https://i.stack.imgur.com/tfGKm.jpg" alt="output during the last training cycle"></p>
  
  <p><img src="https://i.stack.imgur.com/KAt2G.jpg" alt="test against remaining samples and different input"></p>
  
  <p>The first two plots show the training phase. The one on the left is the output at the last iteration of the training procedure, the one on the right the plot of the mean value of the error at the end of each training epoch. The last two plots shows the performance of the network on unseen data. The plot on the right is obtained using as input data from the examples set not used in <em>any</em> way for training. The one on the right is generated using a ramp as input and comparing it with the output of the system for the same input.</p>
  
  <p>What am I doing wrong? 
  The full code can be found <a href="http://pastebin.com/VsFBxZ8L" rel="nofollow noreferrer">here</a></p>
  
  <p>Regards</p>
  
  <p>Andrea</p>