Properly using activation functions of neural network I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear. 

Input Layer:

From my understanding the first step in the neural network is to weight 
inputs by picking the "best" linear function.

Methods used for finding optimal weights (may not be relevant to problem): 


   For finding optimal weights solving quadratic minimization problem is what I usually do, perhaps by finding global minimum of the convex (quadratic error $||Ax-b||^2$ in this case, $A$ being basis matrix and $b$ being ideal vector) function (since every critical point is global minimum there) or orthogonal projection equations (since error is orthogonal to the column space of basis, and their dot product gives us the equation - $A^T(Ax-b$), equating it to $x$: $x = (A^TA)^{-1}A^Tb$.




Once best linear coefficients are found (say $m$ as slope and $c$ as bias), next step is to weight the inputs by evaluating them into the function $f(x)=mx+c$. 

Hidden Layer:

After all inputs are weighted, they must be summed up which gives us a constant number.

I understand that this constant number must then be inputted to some activation function (perhaps sigmoid: $f(x)=\frac{1}{(1+e^{-x})}$ or reLU: $f(x) = max(0, x)$.

Representation of simple neural network with single hidden layer having sigmoid as activation function:  



Picture reference



Problem:

But constants that are evaluated in these activation functions obviously output constants again, so how can they be utilized for data prediction? 

For example, say sum of all weighted inputs is $15$, then its evaluation in sigmoid function will be $\frac{1}{(1-e^{-15})} = 0.99999969409$. That constant number can't be utilized for data prediction (classification/regression) so what are next steps to take? If activation function returns a constant number, how can the data be predicted for different input variables? 

Am I having incorrect perspective for activation functions?
How is data predicted from activation functions? (considering that it returns constants on weighted input sum). 


You should consider the fact that the label of your input data is going to be predicted by the network. Moreover, the outputs of the network usually represent the probability of belonging to each class. The label is not predicted by the activation function, it is predicted by the network. Take a network as a mapper which can map the inputs to outputs. The activations are used to add non-linearity in order to approximate complicated non-linear mappings.


  What's the point of multiple hidden layers?


As you can read here, the purpose of adding those is that you can learn different complicated regions. Another interpretation can be this that by adding more layers you can learn more complicated mappings which have non-linear behaviour.

About the global minimum, because you can not see the cost function which is the error based on the weights, due to the fact that it is not possible to visualise it because it has so many dimensions, you should use gradient-descent based algorithms.