in particulareach of the components of the input is represented as node in the input layer ( in the hidden layer ( there are : nodeseach of which is associated with pair of variables (zawith values :hi (xi and : ( hidden layer = link between nodes ( and xi with weight hi signifies that the value of depends on the value of xi via the function hi finallythe output layer ( represents the value (xqj= note that the arrows on the graph remind us that the sequence of the computations is executed from left to rightor from the input layer through to the output layer in practicewe do not know the collection of functions { hi }because they depend on the unknown gin the unlikely event that gis linearthen all of the ( )( one-dimensional functions will be linear as well howeverin generalwe should expect that each of the functions in { hi is nonlinear unfortunatelytheorem only asserts the existence of { hi }and does not tell us how to construct these nonlinear functions one way out of this predicament is to replace these ( )( unknown functions with much larger number of known nonlinear functions called activation functions for examplea logistic activation function is - ( ( exp(- )we then hope that such networkbuilt from sufficiently large number of activation functionswill have similar representational capacity as the neural network in figure with ( )( functions in generalwe wish to use the simplest activation functions that will allow us to build learner with large representational capacity and low training cost the logistic function is merely one possible choice for an activation function from among infinite possibilities figure shows small selection of activation functions with different regularity or smoothness properties heaviside or unit step rectified linear unit (relu logistic - { - { - - ( exp(- )figure some common activation functions (zwith their defining formulas and plots the logistic function is an example of sigmoid (that isan -shapedfunction some books define the logistic function as ( (in terms of our definitionin addition to choosing the type and number of activation functions in neural networkwe can improve its representational capacity in another important wayintroduce more hidden layers in the next section we explore this possibility in detail activation functions derive their name from models of neuron' response when exposed to chemical or electric stimuli activation functions