Another type of probabilistic model is the Gaussian Process (GP) which is frequently used for regression tasks. By definition, a GP is a collection of random variables, any finite number of which have a joint Gaussian distribution . The working principle of GP is similar to the Bayesian linear models mentioned above, where we first assume a GP prior by specifying the mean (in most cases, we assume zero mean) and covariance functions and then fit this prior to the observed data by maximizing the marginal likelihood with respect to its parameters. Some commonly used covariance functions include squared exponential (SE), Matérn, rational quadratic (RQ), etc. In practice, GP can be effectively utilized to incorporate prior information into functions or integrated with DL models to improve their predictive performance. Additionally, GP is often used as a surrogate model for Bayesian optimization along with an acquisition function. To give a concrete example, say we want to learn a function 𝑓: 𝒳 → 𝒴 to the observed data, i.e. 𝑦 ̂= 𝑓(𝒙), using a supervised learning model with hyperparameters 𝛾. In most cases, hyperparameters are specified before training and we need to manually tune them if the model performance is unsatisfactory. Here, with Bayesian optimization, we treat 𝛾 as the input to a GP and the model performance 𝑠 (e.g. classification accuracy) as its output, i.e. 𝑠(𝛾) ∼ 𝒢𝒫(0, 𝑘(𝛾, 𝛾 ′ )) where 𝑘 is the covariance function of GP. After fitting the GP, for each 𝛾, we can sample from this GP to obtain the mean performance 𝜇 𝑠 = 𝜇(𝛾) and its uncertainty 𝜎 𝑠 = 𝜎(𝛾). We then use the acquisition function 𝑔 to determine the next set of hyperparameters we want to choose. A common choice of acquisition function is the upper confidence bound (UCB) as given below (Some other popular acquisition functions include probability of improvement (PI), expected improvement (EI), etc.):