title: Keras Regression to approximate function (goal: loss < 1e-7)
body: <p>I'm working on a neural network which approximates a function f(X)=y, with X a vector  [x0, .., xn] and y in [-inf, +inf]. This approximated function needs to have an accuracy (sum of errors) around 1e-8. In fact, I need my neural network to overfit.</p>
  
  <p>X is composed of random points in the interval -500 and 500. Before putting these points into the input layer I normalized them between [0, 1].</p>
  
  <p>I use keras as follow:</p>
  
  <pre><code>dimension = 10 #example
  
  self.model = Sequential()
  self.model.add(Dense(128, input_shape=(dimension,), init='uniform',  activation='relu'))
  self.model.add(Dropout(.2))
  self.model.add(Activation("linear"))
  self.model.add(Dense(64, init='uniform', activation='relu'))
  self.model.add(Activation("linear"))
  self.model.add(Dense(64, init='uniform', activation='relu'))
  self.model.add(Dense(1))
  
  X_scaler = preprocessing.MinMaxScaler(feature_range=(0, 1))
  y_scaler = preprocessing.MinMaxScaler(feature_range=(0, 1))
  
  X_scaled = (X_scaler.fit_transform(train_dataset))
  y_scaled = (y_scaler.fit_transform(train_labels))
  
  self.model.compile(loss='mse', optimizer='adam')
  self.model.fit(X_scaled, y_scaled, epochs=10000, batch_size=10, verbose=1)
  </code></pre>
  
  <p>I tried different NN, first [n] -> [2] -> [1] with Relu activation function, then [n] -> [128] -> [64] -> [1].
  I tried the SGB Optimizer and I slowly increase the learning rate from 1e-9 to 0.1.
  I also tried without normalized the data but, in this case, the loss is very high.</p>
  
  <p>My best loss (MSE) is 0.037 with the current setup but i'm far from my goal (1e-8).</p>
  
  <p>First, I would like to know if I did something wrong. I'm in the good way ?
  If not, how can I reach my goal ?</p>
  
  <p>Thanks you very much</p>
  
  <hr>
  
  <p><strong><em>Try #2</em></strong></p>
  
  <p>I tried this new configuration:</p>
  
  <pre><code>model = Sequential()
  model.add(Dense(128, input_shape=(10,), init='uniform', activation='relu'))
  model.add(Dropout(.2))
  model.add(Dense(64, init='uniform', activation='relu'))
  model.add(Dense(64, init='uniform', activation='relu'))
  model.add(Dense(1, init='uniform', activation='sigmoid'))
  </code></pre>
  
  <p>On a sample of 50 elements, batch_size at 10 and during 100000 epochs. 
  I get a loss around 1e-4.</p>
  
  <hr>
  
  <p><strong><em>Try #3</em></strong></p>
  
  <pre><code>model.add(Dense(128, input_shape=(10,), activation='tanh'))
  model.add(Dense(64,  activation='tanh'))
  model.add(Dense(1, activation='sigmoid'))
  </code></pre>
  
  <p>batch_size=1000
  epochs=1e5</p>
  
  <p>result: Loss around 1.e-7</p>