Error in Neural Network To design a neural network based predictor for f(x) = sin(x) I designed the following network. Input ----&gt; Dense(num_neurons=1) ---&gt; Relu() ----&gt; Dense(num_neurons=100) ---&gt; Relu() ---&gt; Dense(num_neurons=1) ----&gt; Relu()--&gt; Output. 

The choice of loss was L2 and optimizer was standard gradient descent. The predictor was trained on X_train = numpy.arange(0.0, 314.1, 0.1) and Y_train = numpy.sin(X_train). It is subsequently tested on X_test= numpy.arange(-10.0, 10.0, 0.001) and Y_test = numpy.sin(X_test) The predictor however performs badly on the test data. What could have gone wrong?

The training data is too small to train the network. Adding more data especially when X &lt; 0

Relu activations used in the network are wrong. Changing the activation in the output node from Relu to Linear should help network learn the function better.

Network is not deep enough. Adding more hidden units/layers to the network will help the network generalize

L2 loss is not a good metric to measure loss when regressing for sin(x)

Gradient descent is not the best optimizer for this problem. Adam optimizer would be a better option.

This type of neural network architecture can fundamentally not learn a function like sin(x).
Since we know that sin(x) can have -ve values as well,
Relu will kill all those mercilessly..

Also we can't have relu just before the output layer as it doesn't makes any sense..(Atleast I have never seen this architecture before..)

Also you can make that 100 drop to around 40

Check this link also