How to test if my implementation of back propagation neural Network is correct I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have noticed during training test of my network:

Specification of the implementation:


A data set containing almost 100000 raw containing (3 variable as input, the sinus of the sum of those three variables as expected output).
The network does have 7 layers, all the layers use the sigmoid activation function


When I run the back propagation training process:


The minimum of costs of the error is found at the fourth iteration (The minimum cost of error is 140, is it normal? I was expecting much less than that)
After the fourth iteration the costs of the error start increasing (I don't know if it is normal or not?)
Actually the implementation was correct, 

The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers which causes the vanishing gradient problem.

When I have decreased the ANN layers to 3 the cost of error was widely reduced besides of that the learning process was faster.

Another common solution is to use RELU or ELU or SELU in the hidden layers of the neural network instead of the sigmoid function