to come from mixture of (usually gaussiandistributionsand the objective is to recover the parameters of the mixture distributions from the data the principal tool for parameter estimation in mixture models is the em algorithm section discusses more heuristic approach to clusteringwhere the data are grouped according to certain "cluster centers"whose positions are found by solving an optimization problem section describes how clusters can be constructed in hierarchical manner finallyin section we discuss the unsupervised learning technique called principal component analysis (pca)which is an important tool for reducing the dimensionality of the data we will revisit various unsupervised learning techniques in subsequent on supervised learning for examplecross-entropy training loss minimization will be important in logistic regression (section and classification ()and pca can be used for variable selection and dimensionality reductionto make models easier to train and increase their predictive powersee sections and risk and loss in unsupervised learning in unsupervised learningthe training data :{ xn only consists of (what are usually assumed to beindependent copies of feature vector xthere is no response data suppose our objective is to learn the unknown pdf of based on an outcome { xn of the training data convenientlywe can follow the same line of reasoning as for supervised learningdiscussed in sections table gives summary of definitions for the case of unsupervised learning compare this with table for the supervised case similar to supervised learningwe wish to find function gwhich is now probability density (continuous or discrete)that best approximates the pdf in terms of minimizing risk `( : lossf ( ) ( ))( where loss is loss function in ( )we already encountered the kullback-leibler risk `( : ln cross-entropy risk (xe ln (xe ln (xg( ( if is class of functions that contains then minimizing the kullback-leibler risk over will yield the (correctminimizer of coursethe problem is that minimization of ( depends on which is generally not known howeversince the term ln (xdoes not depend on git plays no role in the minimization of the kullback-leibler risk by removing this termwe obtain the cross-entropy risk (for discrete replace the integral with sum)`( :- ln (xz (xln (xdx ( thusminimizing the cross-entropy risk ( over all gagain gives the minimizer provided that unfortunatelysolving ( is also infeasible in generalas it still