Why is there a sigmoid function in the hidden layer of a neural network?
Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e

layer_output = Weights.layer_input + bias


Consider a 2 layer neural network, the outputs from layer one will be:
    x2 = W1*x1 + b1
Now we pass the same input to the second layer, which will be

x3 = W2x*2 + b2
Also x2 = W1*x1 + b1
Substituting back, we have:
x3 = W2(W1*x1 + b1) + b2
x3 = (W2W1)*x1 + (W2*b1 + b2)
x3 = W*x1 + b


Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.

So what is the solution?

We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:


ReLU
Sigmoid
tanh
Softmax


and there are a lot more of them.

Yay! Our network is no more linear!

We have a lot of different non linear functions, and each of them serve a different purpose.

For example, ReLU is simple and computationally cheap.
    ReLU(x) = max(0, x)
Sigmoid outputs are between 0 and 1.
tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1
Softmax is usually used if you want to represent any vector as a discrete probability distribution.

Hope you are having a great day!