example (density estimationestimating the density of some random feature is the prototypical unsupervised learning taskwhich we tackled in section using gaussian mixture models we can view gaussian mixture model with components and common scale parameter as neural network with two hidden layerssimilar to the one on figure in particularif the activation function in the first hidden layer is of the form ( with ( :exp(- /( )) ps then the density value (xis computed viaz (xa> ( ) ( )where is column vector of onesw is matrix of zerosand is the softmax function we identify the column vector with the location parameters[ ]of the gaussian mixture and with the weights of the mixture note the unusual activation function of the output layer -it requires the value of from the first hidden layer and from the second hidden layer there are number of key design characteristics of feed-forward network firstwe need to choose the activation function(ssecondwe need to choose the loss function for the training of the network as we shall explain in the next sectionthe most common choices are the relu activation function and the cross-entropy loss cruciallywe need to carefully construct the network architecture -the number of connections among the nodes in different layers and the overall number of layers of the network for exampleif the connections from one layer to the next are pruned (called sparse connectivityand the links share the same weight values {wl, (called parameter sharingfor all {(ij| }then the weight matrices will be sparse and toeplitz intuitivelythe parameter sharing and sparse connectivity can speed up the training of the networkbecause there are fewer parameters to learnand the toeplitz structure permits quick computation of the matrix-vector products in algorithm an important example of such network is the convolution neural network (cnn)in which some or all of the network layers encode the linear operation of convolutionnetwork architecture convolution neural network wl al- wl al- where [ ] : xk yi- + as discussed in example convolution matrix is special type of sparse toeplitz matrixand its action on vector of learning parameters can be evaluated quickly via the fast fourier transform cnns are particularly suited to image processing problemsbecause their convolution layers closely mimic the neurological properties of the visual cortex in particularthe cortex partitions the visual field into many small regions and assigns group of neurons to every such region moreoversome of these groups of neurons respond only to the presence of particular features (for exampleedgesthis neurological property is naturally modeled via convolution layers in the neural network specificallysuppose that the input image is given by an matrix of pixels nowdefine matrix (sometimes called kernelwhere is generally taken to be or thenthe convolution layer output can be calculated using the discrete convolution