What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification
The problem faced with back propagation with activation functions that has a max limit used in a Deep Neural Network (i.e. more then 2 layer network) is that the deeper the network is the quicker your back propagation will degrade.

Take for example your tanh activation function


If say you backpropagate from the output layer to the hidden layer and your tanh derivative is 0.25 then the next layer will only be limited by 0.25 from your tanh function your derived. Since you compute the next layer's derivative which in this case can be 0.5 of the total input and the previous layer's derivative totals to 0.25 then your derivative would be a total of 0.125. By the time the back propagation reaches the input layer, the weights would change by such a small fraction that it does not even matter. 

This is why the ReLu function was such a big breakthrough as it does not have a max limit. There should be methods to counter this effect but ReLu is way more effective.