--- abstract: 'Multi-output regression problems are commonly encountered in science and engineering. In particular, multi-output Gaussian processes have been emerged as a promising tool for modeling these complex systems since they can exploit the inherent correlations and provide reliable uncertainty estimates. In many applications, however, acquiring the data is expensive and safety concerns might arise (e.g. robotics, engineering). We propose a safe active learning approach for multi-output Gaussian process regression. This approach queries the most informative data or output taking the relatedness between the regressors and safety constraints into account. We prove the effectiveness of our approach by providing theoretical analysis and by demonstrating empirical results on simulated datasets and on a real-world engineering dataset. On all datasets, our approach shows improved convergence compared to its competitors.' bibliography: - 'ref.bib' --- \[ \] Introduction {#section-intro} ============ Active learning (AL) selects the most informative data sequentially according to previous measurements and an acquisition function [@krause08a; @journals/corr/abs-1112-5745; @AAAI1611879]. The objective is to optimize a model without labeling unnecessary data. The problem setup is closely related to Bayesian optimization, i.e. BO [@brochu2010tutorial], which optimizes a black-box function with limited exploration. In various scenarios, safety concerns are also critical during the exploration phase. For instance, movements of a machine are not supposed to crash any objects. A system should avoid generating high pressure, high temperature, or explosion. Safe learning addresses this by incorporating and learning safety constraints [@pmlr-v37-sui15]. @10.1007/978-3-319-23461-8_9 and @NEURIPS2018_b197ffde combine safety considerations with AL so that the data selection is done only in the determined safe domain. These works, however, rarely considered multi-output (MO) regression problems, despite them commonly encountered in science, engineering and medicine [@xu2019survey; @zhang2021survey; @LIU2018102]. In such problems, it is possible to consider individual tasks or outputs independently, but the plausibly shared mechanisms are ignored, and the performances or data efficiency might be deteriorated. @AAAI1611879 dealt with AL on MO models but focused on efficient computation of AL with large datasets and safe exploration was not addressed. We consider safe AL for MO regression models that exploit the correlations. In particular, we focus on problems in which different output components may not be synchronously observed (e.g. due to different measuring cost or difficulty). MO Gaussian processes (GPs) are natural candidates for these problems [@NIPS2007_66368270; @JMLR:v12:alvarez11a; @alvarez2012kernels; @vanderwilk2020framework], due to their capability of capturing the correlations among different outputs and of quantifying the uncertainty. In our work, we consider as main model the Linear Model of Coregionalization (LMC, @etde5214736), in which each output is modeled as a weighted sum of shared latent functions. Each latent function is drawn from a GP. Later on, we extend the theoretical analysis also to the convolution process [@higdon2002space; @JMLR:v12:alvarez11a] in which each latent function is additionally convolved by an output-specific smoothing kernel. To the best of our knowledge, this is the first framework about safe AL for MOGP regression. Our contributions can be summarized as follows: - We formulate an acquisition function for safe active learning in the MOGP framework that allows asynchronous measurements. - We provide theoretical analysis of the safe AL algorithm in our framework, particularly we derive a convergence rate to the algorithm. - We demonstrate the performance and superiority to state-of-the-art competitors on a real-world engineering dataset. The overview of this papers is as follows. In section \[section-related\_work\], we briefly review the related works. In section \[section-methods\], we introduce our algorithm. We discuss the theory behind our algorithm in section \[section-thms\], and validate empirically its usefulness in section \[section-experiments\]. Finally, section \[section-conclusion\] concludes our work. Related Work {#section-related_work} ============ AL has been extensively investigated for classification tasks [@Hoi_ICML2006; @5206627; @journals/corr/abs-1112-5745; @GCAI2019:On_Robustness_of_Active; @pmlr-v130-shi21a], but less literature addresses AL in the regression setting [@krause08a; @GarnettOH2013]. The problem setup is closely related to BO [@brochu2010tutorial]. While AL and BO both consider limited exploration, the goals are very different. AL aims to obtain a well performing model, usually with characteristic of overall precision, but BO only finds an optimum, e.g. a configuration of best performance or lowest cost. In a BO problem, the model quality for points far away from the optimum is not important and can be really bad. For a more general problem in this line of research, i.e. optimizing under uncertainty, GPs, which are capable of making predictions under uncertainty, are often used as surrogate models [@brochu2010tutorial; @Srinivas_2012]. In recent years, the importance of safety considerations has led to a novel line of research ranging from Safe Bayesian optimization [@pmlr-v37-sui15; @Berkenkamp_2016; @berkenkamp2020bayesian] to safe AL in a static environment [@10.1007/978-3-319-23461-8_9] or dynamic systems [@NEURIPS2018_b197ffde]. None of these contributions, however, consider MO which is able to exploit correlations among outputs. Exploiting MO correlations has been shown successful in various applications [@casale2017joint; @LIU2018102; @s12911-020-1069-4]. State of the art MO models, in particular with GPs (also refer to @alvarez2012kernels and @vanderwilk2020framework for an overview over MOGPs), include the Linear Model of Coregionalization (LMC), a simple yet effective model [@etde5214736; @NIPS2007_66368270; @teh2005semiparametric], and one of its extensions, the convolution process, which further captures correlations in multiple outputs that vary in smoothness [@higdon2002space; @JMLR:v12:alvarez11a]. Complexity of GPs and MOGPs scales cubically with the number of observations [@3569]. Existing works focus a lot on approximation methods for large datasets with GPs [@pmlr-v5-titsias09a; @hensman2013gaussian] and with MOGPs [@JMLR:v12:alvarez11a; @Nguyen2014UAI_CMOGP; @vanderwilk2020framework]. In contrast to @AAAI1611879, both our simulation and our real-world dataset can be modeled with very few data points. We thus consider MOGPs without any sparse approximations, even though these methods could be incorporated into our approach. Very few works have tried to combine MO modeling with BO [@NIPS2013_f33ba15e] or AL [@AAAI1611879]. @NIPS2013_f33ba15e focused on transferring BO results between tasks, while @AAAI1611879 investigated efficient computation of AL for sparse MOGPs [@AAAI1611879]. To the best of our knowledge, none of the literature addressed safe data query or safe AL for MOGPs. Methods {#section-methods} ======= We first provide background on GPs and MOGPs, and different inference strategies. In a second step, we show how safe active learning can be applied over multiple outputs. GP Regression {#section-GP} ------------- #### Single-output A GP is a stochastic process where every finite subset follows a multivariate normal distribution. In GP regression, given observed data $\mathcal{D}=\{\bm{x}_{n} \in \mathbb{R}^D, y_{n} \in \mathbb{R}\}_{n=1}^{N}$, we specify a mean function $m: \mathbb{R}^D \rightarrow \mathbb{R}$ and a positive definite kernel function (covariance function) $k: \mathbb{R}^D \times \mathbb{R}^D \rightarrow \mathbb{R}$ as a GP prior for the function. The observations $y_n$ are assumed to be the functional values blurred by i.i.d. Gaussian noise. The model is formulated as $$g \sim \mathcal{GP} \left( m(\cdot), k(\cdot, \cdot) \right), y_{n}=g(\bm{x}_{n}) + \epsilon_{n}, \epsilon_{n} \sim \mathcal{N} \left( 0, \sigma^2 \right) .$$ The goal is to predict $g(\bm{x}_*)$ and its uncertainty for a new input $\bm{x}_{*}$. Assuming for simplicity a zero mean prior, $m \equiv 0$, the posterior is $p \left( g(\bm{x}_{*}) | \bm{x}_{*}, \mathcal{D} \right) = \mathcal{N} \left( \mu(\bm{x}_{*}), var(\bm{x}_{*}) \right) $, with $$\begin{aligned} \label{eqn-gp_mean} \mu(\bm{x}_{*}) &= K_{N*}^T \left( K_{NN} + \sigma^2 I \right)^{-1} (y_1, ..., y_N)^T, \\ \label{eqn-gp_variance} var(\bm{x}_{*}) &= k(\bm{x}_{*}, \bm{x}_{*}) - K_{N*}^T \left( K_{NN} + \sigma^2 I \right) ^{-1} K_{N*},\end{aligned}$$ where $K_{N*} \in \mathbb{R}^{N\times1}$ and $K_{NN} \in \mathbb{R}^{N \times N}$ are matrices with $[K_{N*}]_{i} = k(\bm{x}_{i}, \bm{x}_{*})$ and $[K_{NN}]_{i, j} = k(\bm{x}_{i}, \bm{x}_{j})$. For further details, please see @3569. #### Multi-output (MO) We consider LMC as our main model [@etde5214736]. Here we have $\bm{y}_n=\bm{f}(\bm{x}_n)+\bm{\epsilon}_n = W\bm{g}(\bm{x}_n)+\bm{\epsilon}_n \in \mathbb{R}^{P}$ with i.i.d. noise $[\bm{\epsilon}_n]_{p} \sim \mathcal{N} \left( 0, \sigma_{p}^{2} \right) $ for $p=1, 2, ..., P$, linear transformation $W \in \mathbb{R}^{P \times L}$, and latent GPs $g_l(\cdot) = [\bm{g}(\cdot)]_{l} \sim \mathcal{GP} \left( 0, k_l(\cdot, \cdot) \right) $ for $l=1, ..., L$. Throughout this paper, we further assume finite $P$, finite $L$, bounded $k_l(\cdot, \cdot)$, and each element of $W$ bounded by a constant. Let $f_p(\cdot) = [\bm{f}(\cdot)]_{p}$. In this model, $\{f_{p}(\bm{x})\}_{p=1}^{P}$ is also a GP where every finite subset has zero mean and covariance $cov \left( f_{p}(\bm{x}), f_{p'}(\bm{x}') \right)=$ $\sum_{l=1}^{L} W_{pl}W_{p'l} k_{l}(\bm{x}, \bm{x}') \eqqcolon \eta_{p, p'}(\bm{x}, \bm{x}'). $ Let $\bm{Y}$ denote the collection of observations $\{\bm{y}_{n} \in \mathbb{R}^{P}\}_{n=1}^{N}$, $\bm{X}$ denote $\{\bm{x}_{n}\}_{n=1}^{N}$, and $\mathcal{D} = \{ \bm{X}, \bm{Y} \}$. Let $y_{pn}$ be the $p$-th component of the $n$-th observation, i.e. $y_{pn} = [\bm{y}_{n}]_{p}$. The posterior $p(\bm{f}(\bm{x}_{*}) | \bm{x}_{*}, \mathcal{D})$ is a multivariate Gaussian $\mathcal{N}(\mu(\bm{x}_{*}), \Sigma(\bm{x}_{*}))$ with $$\begin{aligned} \label{eqn-mo_posterior_mu} \mu(\bm{x}_{*}) &= \Omega_{N*}^T \left( \Omega_{NN} + {\text{diag}}(\{\sigma_i^2\}_{p=1}^{P}) \otimes I_N \right) ^{-1} \bm{Y}, \\ \label{eqn-mo_posterior_cov} \Sigma(\bm{x}_{*}) &= \Omega_{**} - \Omega_{N*}^T \left( \Omega_{NN} + {\text{diag}}(\{\sigma_i^2\}_{p=1}^{P}) \otimes I_N \right) ^{-1} \Omega_{N*},\end{aligned}$$ where $\otimes$ denotes the Kronecker product, $\Omega_{**}$, $\Omega_{N*}$ and $\Omega_{NN}$ are gram matrices of kernel $\eta_{p, p'}(\cdot, \cdot)$. See our supplementary section \[supp\_section-MOGP\] for full expression of the matrices and for the derivation of this posterior. Notice that $\bm{Y}$ can be ordered differently, but the corresponding permutation needs to be applied to the current $\Omega_{N*}$, $\Omega_{NN}$ and ${\text{diag}}(\{\sigma_i^2\}_{p=1}^{P}) \otimes I_N$. As the permutation matrices cancel each other out, the posterior stays in the same form with only different indexing. #### Partially observed MO In the previous section, each observation of $\bm{Y}$ has every component observed for every input $\bm{X}$. In the following, we assume that some components can be omitted to save measuring costs as well as computational costs due to smaller $\Omega_{\cdot \cdot}$. Let $N_p$ be the number of outputs with $p$-th component observed and $N_{sum} = \sum_{p=1}^{P} N_p$. If the output is fully observed, we can see that $N_1=...=N_P=N$ and $N_{sum} = PN$. For clarification, we define a reindexing bijection that maps the original index pairs to scalar indices (i.e. the scheme concatenates the outputs over all components into an one-dimensional vector), and the non-observed components are assigned with negative or zero indices. With this bijection, we can consider all observed output components by looking only at the positive indices ranging from $1$ to $N_{sum}$. The notation of outputs now becomes $\bm{Y}_{\phi} = \{y_{p_{k}n_{k}}\}_{k=1}^{N_{sum}}$, where $\phi:(p, n) \rightarrow k$ is a re-indexing bijection with $(p_{k}, n_{k}) = \phi^{-1}(k)$. The output domain of $\phi$ is $\mathbb{Z} \cap [-NP+N_{sum}+1, N_{sum}]$ , where $\{1, ..., N_{sum}\}$ are the new indices of all observed output components. Notice that the notation is adapted from the fully observed scenario, so $\phi$ is dependent of $N$ (i.e. we clearly have $N_p \leq N, \forall p$). However, we omit $N$ in the notation for simplicity. In addition, the corresponding rows and columns of the gram matrices are also omitted, and when we make predictions for one output component, the notation becomes as follows: $$\begin{aligned} \label{eqn-mo_posterior_mu_poo} \mu(\bm{x}_{*}, p_*) &= [\Omega_{N_{sum}*}]_{all,p*}^T \widehat{\Omega}_{N_{sum}N_{sum}}^{-1} \bm{Y}_{\phi}, \\ \label{eqn-mo_posterior_cov_poo} \begin{split} \Sigma(\bm{x}_{*}, p_*) &= \eta_{p_*, p_*}(\bm{x}_{*}, \bm{x}_{*})\\ &- [\Omega_{N_{sum}*}]_{all,p*}^T \widehat{\Omega}_{N_{sum}N_{sum}}^{-1} [\Omega_{N_{sum}*}]_{all,p*}, \end{split}\end{aligned}$$ where $\widehat{\Omega}_{N_{sum}N_{sum}} = \Omega_{N_{sum}N_{sum}} + diag(\{\sigma_{p_k}^2\}_{k=1}^{N_{sum}})$. Further notice that we omit the components without changing the order, so $\{\sigma_{p_k}^2\}_{k=1}^{N_{sum}}$ is actually $\{\sigma_1^2\}_{i=1}^{N_1}$ followed by $\{\sigma_2^2\}_{i=1}^{N_2}$ and so on. #### MOGPs v.s. Multiple independent single-output GPs Concatenating $P$ single output GPs without modeling the output correlations is equivalent to a MOGP with $W = I_p$, i.e. $\eta_{p, p} = k_p$ and $\eta_{p, p'} \equiv 0$ for $p \neq p'$. Notice that in this case, the gram matrix $\Omega_{N_{sum},N_{sum}}$ and the inverse with noise variances have only non-zero components on the diagonal subblocks corresponding to $\eta_{p, p}$. The cross-covariance $\Omega_{N_{sum},*}$ has only non-zero components at $(iN+p, p)$ entries, $i=0,...,P-1$, and thus the posterior is identical to squeezing the posteriors of individual GPs into one vector/matrix. See supplementary section \[supp\_section-MOGP\] for full matrix expression. On the other hand, if $W \neq I_P$, information can flow between the components which can ultimately lead to more accurate predictions and smaller uncertainty estimates. Inference with Hyperparameters ------------------------------ The choice of kernel(s) and noise variance(s) allows the model to express various patterns learned from the data. This, however, requires the tuning of hyperparameters, jointly denoted by $\bm{\theta}$. #### Type II maximum likelihood estimation A simple way is to select hyperparameters that maximize the log marginal likelihood. Mathematical detail is provided in supplementary section \[supp\_section-exp\_detail\_hyperparameters\_tuning\_detail\]. Note that GPs are not scalable to large datasets without any approximations such as sparse variational inference [@pmlr-v5-titsias09a; @hensman2013gaussian]. Such kind of approximation techniques could also be incorporated into our MOGP model [@Nguyen2014UAI_CMOGP; @vanderwilk2020framework]. #### Bayesian treatment Maximum likelihood estimation can suffer from overfitting problems. This in particular holds true for the low-data regime in which we are operating. On the contrary, we can assign prior distributions over the hyperparameters and compute the predictive GP posterior over all possible hyperparameters. This inference is then an integral over the hyperparameters, which is intractable. We either need to perform approximate inference [@titsias2014doubly] or resort to Monte Carlo sampling. In our work, we apply the latter. We use Hamiltonian Monte Carlo (HMC) [@betancourt2018conceptual; @brooks2011handbook] as our sampling method. Extensions to sparse GPs also exist [@hensman2015mcmc]. We refer to section \[supp\_section-exp\_detail\_bayesian\_treatment\] for mathematical detail. Safe AL {#section-safety_AL} ------- Our algorithm extends the work of @NEURIPS2018_b197ffde to the multi-output scenario. The general goal of AL is to obtain good models with as few data points as possible. AL methods are especially important when it is expensive to measure training data (e.g. expensive to hire an expert or run large devices). The model performance can be quantified e.g. by uncertainty or by RMSE. Here we introduce the algorithm we use, and then in section \[section-thms\] we show that the uncertainty of the model decreases to zero with the safe AL. #### Pool-based AL AL is a sequential learning scheme that allows us to query only the most informative data for a problem. In each learning iteration, we are given an observed dataset $\mathcal{D}$ and a pool set $\mathcal{D}_{pool}$ containing candidate points that can be queried. We query a new observation $\bm{y}_a(\bm{x}_a)$ from the pool according to an acquisition function $\alpha, \alpha(\cdot) \in \mathbb{R}$ such that $\bm{x}_a = \text{argmax}_{\bm{x}} \{\alpha(\bm{x}, \mathcal{D}) | \bm{x} \in \mathcal{D}_{pool}\}$. The acquisition function determines the gain of acquiring each candidate without access to the actual $\bm{y}$ value corresponding to this candidate. In a real application, data that are not queried would not be measured. After the query, the corresponding new measurement $\bm{y}_a$ will be provided. Therefore, the observed and pool sets become $\mathcal{D} \cup \{\bm{x}_a, \bm{y}_a\}$ and $\mathcal{D}_{pool} \setminus \{\bm{x}_a, \bm{y}_a\}$ respectively, and the new iteration is conducted with the updated datasets. When the outputs are partially observed, $\mathcal{D}_{pool} = \{((\bm{x}, p), y_p)\}$ and the query problem is $(\bm{x}_a, p_a) = \text{argmax}_{\bm{x}, p} \{\alpha(\bm{x}, p, \mathcal{D}) | (\bm{x}, p) \in \mathcal{D}_{pool}\}$ and the corresponding $[\bm{y}_a]_{p_a}$ is returned (see algorithm \[alg-SAL\]). A pool set is usually a finite set and we also focus on finite pools in this work. We consider finite pool assumption not a limiting factor. In practice, many datasets are either finite by nature or can be easily discretized in this way [@Kumar_Gupta_ALQueryStrategies]. From a theoretical point of view, we focus on compact datasets in the next sections (as assumed in previous literature). Given commonly used kernels such as a squared exponential kernel or Mat<span>é</span>rn kernels (see supplementary section \[supp\_section-MOGP\]), such a space can always be described by finite discretization with arbitrarily small error [@Srinivas_2012]. #### Acquisition function Commonly used acquisition function includes differential entropy [@krause08a; @10.1007/978-3-319-23461-8_9] and expected information gain [@krause08a; @journals/corr/abs-1112-5745]. We use predictive entropy as our acquisition function: $$\label{eqn-acquisition_func} \alpha(\cdot, \mathcal{D}) = H(\cdot | \mathcal{D}) = \frac{1}{2} \log(|\Sigma|) + \frac12 R\log(2 \pi e).$$ For fully observed outputs, $R=P$ and $\Sigma$ is the covariance from eq. . For partially observed outputs, $R=1$ and $\Sigma$ is the variance from eq. . Note that a close form entropy can be obtained because the GP posterior is normal. However, with a Bayesian treatment scheme, the entropy is an intractable integral, and it is unrealistic in practice to compute the integral for each candidate sample. Hence, we further approximate the Bayesian treatment posterior (eq. ) as a Gaussian distribution using moment matching. See supplementary section \[supp\_section-exp\_detail\_entropy\_approximation\] for detail. Maximization of the entropy with respect to input variables is an optimization problem independent of the constant term given in the formula. Therefore, this acquisition function is actually equivalent to $\log(|\Sigma|)$ and also to $|\Sigma|$ because we further know that $\log$ is strictly increasing. #### Safety condition An important goal of safe AL is to ensure that the data are queried with safety consideration. Therefore, in addition to the observations, $\bm{Y}$, we assume to have safety values $\bm{Z} \subseteq \mathbb{R}$ described by a function $h: \mathbb{R}^{D} \rightarrow \mathbb{R}$. We assume $h$ has a GP prior, then the predictive distribution $p(h(\bm{x_*}) | \bm{x_*}, \bm{X}, \bm{Z})$ can be used to determine the safety condition probabilistically. Here $p(h(\bm{x_*}) | \bm{x_*}, \bm{X}, \bm{Z})$ is a normal distribution with mean and variance later denoted by $\mu_h(\bm{x})$ and $var_h(\bm{x})$ (computed with eq. eq. ). We let $\xi(\bm{x} | \mathcal{D})$ denote the safety probability at $\bm{x}$. For instance, the safety values may be temperature that should not exceed a threshold $z_{max}$, then $$\label{def-safety_p} \xi(\bm{x} | \mathcal{D}) \coloneqq \int_{-\infty}^{z_{max}} \mathcal{N} \left( z | \mu_h(\bm{x}), var_h(\bm{x}) \right) dz$$ would be the safety probability at $\bm{x}$. If we define safety as the values above a threshold $z_{min}$, then the safety probability would be the integral of the same distribution over $z_{min}$ to infinity. We denote $z_{min}$ or $z_{max}$ jointly by $z_{bar}$, and let $z_{mode}$ be a boolean variable controlling whether the threshold is an upper bound or lower bound. Then we adjust the notation $\xi(\bm{x} | \mathcal{D})$ to $\xi(\bm{x} | z_{bar}, z_{mode}, \mathcal{D})$, indicating that the safety probability is actually conditioned on the safety setup. Furthermore, notice that the safety values are not observed by the main MOGP model, but could easily be included in future work. #### Acquisition of safe AL Assume that $\bm{x}$ is safe when the corresponding safety probability is greater than $1-\delta$ for a small $\delta \in (0, 1]$ [@10.1007/978-3-319-23461-8_9; @NEURIPS2018_b197ffde]), then our query problem becomes $$\label{prob-safe_al}\begin{array}{c} (\bm{x}_a, p_a) =\text{argmax}_{\bm{x}, p} \{\alpha(\bm{x}, p, \mathcal{D}) | \bm{x} \in \mathcal{D}_{pool}\} \\ s.t.\ \ \xi(\bm{x}_a | z_{bar}, z_{mode}, \mathcal{D}) > 1-\delta. \end{array}$$ If the output is fully queried, the index $p_a$ can be omitted. Notice that we consider finite $\mathcal{D}_{pool}$, which indicates that problem \[prob-safe\_al\] can be solved by computing the safety probability and acquisition score of every candidate point. We first exclude points failing the constraint and then selecting the $(\bm{x}_a, p_a)$ pair with maximal acquisition score. If there are multiple pairs with the same maximal acquisition score, one of them would be selected ranomly, but this is in principle not going to happen for our acquisition function which gives floating numbers numerically. In principle, the same acquisition function could be applied using independent single output GPs and optimizing over all outputs simultaneously. However, as discussed in section \[section-GP\], the observation of the $p$-th output component would then have no effect on the posterior of any other component, leading to a suboptimal selection strategy. The datasets $\mathcal{D}_{\cdot}$ now represents the collections $\{(\bm{x}, \bm{y}, z)\}$ or $\{((\bm{x}, p), y_p, z)\}$. Under the partially observed output setting, multiple queries of different output components at the same input $\bm{x}$ may result in duplicate queries of the corresponding safety output $z$. When the observation noise is close to zero, such duplicate queries would result in almost identical rows or/and columns in the gram matrix and thus make the matrix non-invertible (see eq. -). However, this rarely happens in practice, and we also did not experience this in our experiments. $\delta \in (0,1], z_{bar}, z_{mode}, \mathcal{D}_0, \mathcal{D}_{pool}$ (disjoint) Given $\mathcal{D}_i$, optimize/sample hyperparameters for 1.5em $\bm{f}$ (main MOGP model) and 1.5em $h$ (safety GP model for querying, see eq. \[def-safety\_p\]) Query according to eq. \[prob-safe\_al\]: 1.5em $\mathcal{D}_{new} \gets \{\bm{x}_a, \bm{y}_a, z_a \}$ or $\mathcal{D}_{new} \gets \{\bm{x}_a, y_{pa}, z_a \}$ $\mathcal{D}_{i+1} \gets \mathcal{D}_i \cup \mathcal{D}_{new}$, $\mathcal{D}_{pool} \gets \mathcal{D}_{pool} \setminus \mathcal{D}_{new}$\ Complexity {#section-complexity} ---------- In each AL iteration, it is required to compute the marginal likelihood of GP models and the predictive uncertainty on a pool set. If we wish to evaluate the model performance, e.g. RMSE, on a test set, then we would also compute the prediction on this test set. The overall complexity is the number of AL iteration times the complexity of each iteration. Computation of the marginal likelihood is for model training or hyperparameters sampling. This computation is dominated by the inversion of covariance matrix (eq. ), which scales with $\mathcal{O}\left( N_{sum}^3 \right)$. If each output dimension is modeled independently by a single-output GP, the joint model of all output scales with $\mathcal{O}\left( N_1^3 + ... + N_P^3 \right)$. In a Bayesian treatment, the number of samples create linear burden multiplied to this complexity. To perform an inference on $N_{eval}$ independent points, i.e. compute the mean and covariance of different output channels at each point but not the covariance among different data points, the complexity is $\mathcal{O}\left( N_{sum}^3 \right) + \mathcal{O}\left( N_{eval} N_{sum}^2 \right)$ with a MOGP. With independent GPs, it is $\mathcal{O}\left( N_1^3 + ... + N_P^3 \right) + \mathcal{O}\left( N_{eval} N_1^2 + ... + N_{eval} N_P^2 \right)$. The first term is from matrix inversion while the second from matrix multiplication. Many AL scenarios consider very few data points, where the term with $N_{eval}$ might dominate, otherwise this term is negligible and can be omitted. This term can be further reduced as the predictions can also be performed in parallel, e.g. with a GPU. The cubic complexity is one of the main weaknesses of standard GP methods. However, such implementations can still scale up to thousands of observations which is sufficient for many practical AL scenarios in which measuring a single outcome can already be expensive and/or time consuming. For applications that require larger datasets, we can either use approximate sparse solutions based on optimization [@titsias2014doubly] or based on Monte Carlo sampling [@hensman2015mcmc], or use more customized implementations that built on conjugate gradients and multi-GPU parallelization [@wang_wilson2019neurips]. However, in this scenario, one might also need to consider batch AL [@krause08a; @AAAI1611879; @kirsch2019batchbald] which lies outside the scope of this paper. Asymptotic Convergence Analysis {#section-thms} =============================== The goal of this section is to obtain a convergence guarantee of the algorithm by extending Theorems 2, 3 in @NEURIPS2018_b197ffde and theorem 5 in @Srinivas_2012 to our MO framework. Even though our main focus is on the scenario with partially observed outputs, the following theoretical analysis holds for a more general setting, namely fully observed and partially observed outputs. Notice that when the outputs are fully observed, all output components are predicted at the same time, and the conditioning structure is not exactly the same as obtaining all components of the same point sequentially in the partially observed manner. Uncertainty Bound ----------------- We start from bounding the predictive uncertainty of multiple AL iteration by a mutual information term. We are using the following notation in this section: given $k-1$ observations $\{y_{p_{i}n_{i}}\}_{i=1}^{k-1}$ (partially observed outputs) or $\{\bm{y}_{i}\}_{i=1}^{k-1}$ (fully observed outputs) obtained from our acquisition function (without safety constraint), for any point $\hat{\bm{x}}_{\cdot}$ and any index $\hat{p}_k$, let $\Sigma_{k-1}(\hat{\bm{x}}_{n_k}, \hat{p}_k)$ be the predictive variance of $f_{p_k}(\hat{\bm{x}}_{n_k}) | \{y_{p_{i}n_{i}}\}_{i=1}^{k-1}$ for partially observed outputs, and $\Sigma_{k-1}(\hat{\bm{x}}_{k})$ the predictive covariance of $\bm{f}(\hat{\bm{x}}_{k}) | \{\bm{y}_{i}\}_{i=1}^{k-1}$ for fully observed outputs. In addition, let $\Sigma_{0}(\hat{\bm{x}}_{n_1}, p_1)=\eta_{p_{n_1}}(\hat{\bm{x}}_{n_1}, \hat{\bm{x}}_{n_1})$ and $\Sigma_{0}(\hat{\bm{x}}_{1})=\bm{\eta}(\hat{\bm{x}}_{1}, \hat{\bm{x}}_{1})$ be the corresponding prior (co)variance. In the first step, our main goal is to bound the predictive (co)variances. Similar to @NEURIPS2018_b197ffde, we first use the mutual information (also see supplementary-lemma \[supp\_lemma-mutual\_information\]) to bound the predictive (co)variance of the MO model. Notice that $W$ is finite dimensional, implies that if each element of $W$ is bounded then there exists a constant $sup\{bound \ of \ |W_{p,l}| \}$ bounding all elements at the same time. This lemma (and the next theorem) only holds with an acquisition function returning points with maximum determinant of predictive variance. \[lemma-predictive\_cov\_bound\] If $k_l(\cdot, \cdot)$ and $W$ are bounded, let $\hat{v}>0$ be a bound of all kernels $k_l(\cdot, \cdot)$, and let $\hat{w} > 0$ be a bound of all elements of $W$ (i.e. $0 \leq k_l(\cdot, \cdot) \leq \hat{v}, \forall l$ and $|W_{p,l}| \leq \hat{w}, \forall p, l$). Furthermore, let $\psi$ be an upper bound of $\{\sigma_p^2\}_{p=1}^{P}$. $\{(\bm{x}_{n_k}, p_k)\}$ or $\{\bm{x}_n\}$ is the dataset queried from our acquisition function and $\bm{Y}$ or $\bm{Y}_\phi$ is the collection of correponding observations. Given a fixed set of hyperparameters $\bm{\theta}$, then $$\begin{aligned} \frac{1}{N_{sum}}\sum_{k=1}^{N_{sum}}\Sigma_{k-1}(\cdot, \cdot) &\leq \frac{2C_1}{N_{sum}} I \left( \bm{Y}_{\phi}, \{ f_{p_k}(\bm{x}_{n_k}) \}_{k=1}^{N_{sum}} \right), \\ \frac{1}{N}\sum_{n=1}^{N}|\Sigma_{n-1}(\cdot)| &\leq \frac{2C_2}{N} I \left( \bm{Y}, \{ \bm{f}(\bm{x}_{n}) \}_{n=1}^{N} \right), \end{aligned}$$ where $C_1 = \frac{ L \hat{w}^2 \hat{v} }{ \log \left( 1+\frac{L \hat{w}^2 \hat{v}}{\psi} \right) }$ and $C_2 = \frac{(L \hat{w}^2 \hat{v})^P}{\log \left( 1 + \left( \frac{L \hat{w}^2 \hat{v}}{\psi} \right)^P \right) }$ are constants, and $I(\cdot, \cdot)$ is the notation of mutual information. $C_1$ and $C_2$ are the bounding coefficients. We provide the proof of this lemma in the supplementary material (section \[supp\_section-proof\_of\_main\_lemma1\], also see supplementary corollary \[supp\_ineq-determinant\_posterior\_cov\] and lemma \[supp\_lemma-omega\_bound\]). Notice that lemma 2 or 4 of @NEURIPS2018_b197ffde could not be applied directly for our result. The proof makes use of sequential conditioning of observations. In our setting, this conditioning chain involves multiple output components with varying noise levels, which is different from the setting in @NEURIPS2018_b197ffde. Our proof makes the assumptions that the elements of $W$ and the noise variances $\{\sigma_p^2\}_{p=1}^{P}$ are bounded. We deem these assumptions to be mild in practice, since most datasets are normalized within sensitive ranges. When $\psi$ is finite, $C_1$ and $C_2$ are bounded (note: $\lim_{t \rightarrow 0} \frac{t}{\log (1+t/\psi)}=\psi$). With finite $C_1$ and $C_2$, the predictive (co)variances are simply bounded by $\mathcal{O}\left( \frac{1}{N_{sum}} I \left( \bm{Y}_{\phi}, \{ f_{p_k}(\bm{x}_{n_k}) \}_{k=1}^{N_{sum}} \right) \right)$ and $\mathcal{O}\left( \frac{1}{N} I \left( \bm{Y}, \{ \bm{f}(\bm{x}_{n}) \}_{n=1}^{N} \right) \right)$. Convergence Guarantee --------------------- As the second step, we would like to show that the mutual information bound in lemma \[lemma-predictive\_cov\_bound\] converges to zero for $k_l(\cdot, \cdot)$ being some commonly used kernels. We hereby go beyond the work of @NEURIPS2018_b197ffde that only focused on squared exponential kernels. Once proven, we can use lemma \[lemma-predictive\_cov\_bound\] to conclude that the predictive uncertainty of the model decreases to zero with our AL scheme, which is a desired property. To achieve this, we use the maximum information gain, $\gamma$, which was introduced in @Srinivas_2012. We add few more notation in order to define the quantity $\gamma$. Let $\mathcal{X} \subseteq \mathbb{R}^D$ denote the input space, $\mathcal{Y} = \mathcal{Y}(\mathcal{X}) \subseteq \mathbb{R}^P$ the output space, $\pi_p: \mathbb{R}^P \rightarrow \mathbb{R}$ the projection mapping returning the $p$-th component, and $\mathcal{Y}_p(\mathcal{X}) \subseteq \mathbb{R}$ the set $\pi_p (\mathcal{Y})$. We consider $\gamma_p^{N_p}$ of GPs $ f_p \sim \mathcal{GP}(0, \eta_{p, p}) $, i.e. $\gamma_p^{N_p} = \max_{\mathcal{D} \subseteq \mathcal{Y}_p(\mathcal{X}): |\mathcal{D}|=N_p} I(\mathcal{D}, f_p)$. Notice that these are standard single-output GPs, and we can thus apply the theorems in @Srinivas_2012. The maximum information gain gives us the following theorem: \[thm-convergence\_guarantee\] Let $n$ be a unified expression of $N$ and $N_{sum}$. Let $\{\bm{\hat{x}_i}\}_{i=1}^{n}$ be $n$ arbitrary input points within a compact and convex domain $\mathcal{X}$, and, in a partial output setting, let $\{\hat{p}_i\}_{i=1}^{n}$ be $n$ arbitrary output component indices. Assume $k_l(\cdot, \cdot)$ and $W$ are bounded and $\eta_{p, p'}(\cdot, \cdot) \leq 1$ for any $p, p'$. We further let $\{\Sigma_{k-1}(\hat{\bm{x}}_k), \Sigma_{k-1}(\hat{\bm{x}}_k, \hat{p}_k)\}$ be the predictive (co)variances of $\bm{\hat{x}_k}$ conditioning on $k-1$ training data queried according to our acquisition function within domain $\mathcal{X}$ (without safety constraint). Given fixed hyperparameters $\bm{\theta}$, then $$\label{eqn-thm2_expresion} \frac{1}{n}\sum_{k=1}^{n}|\Sigma_{k-1}(\bm{\hat{x}_k})|, \ \frac{1}{n}\sum_{k=1}^{n} \Sigma_{k-1}(\bm{\hat{x}_k}, \hat{p}_k) \leq \mathcal{O} \left( \frac{1}{n} \sum_{p=1}^{P} \gamma_p^{N_p} \right).$$ Furthermore, if all of $k_l$ are $\nu$-Mat<span>é</span>rn kernel with $\nu > 1$ or are squared exponential kernel, then $$\begin{aligned} \frac{1}{n} \sum_{p=1}^{P} \gamma_p^{N_p} &\leq \mathcal{O} \left( n^{\frac{-2\nu}{2\nu+D(D+1)}}\log \ n \right),\text{ or}\\ \frac{1}{n} \sum_{p=1}^{P} \gamma_p^{N_p} &\leq \mathcal{O} \left( \frac{(log \ n)^{D+1}}{n} \right),\text{ respectively}. \end{aligned}$$ Here we sketch the idea of the proof. We consider the mutual informations in lemma \[lemma-predictive\_cov\_bound\] with respect to our GP prior, which is ${\frac{1}{2}\log \big| I_{N_{sum}} + \left( {\text{diag}}(\{\sigma_{p_k}^2\}_{k=1}^{N_{sum}}) \right) ^{-1} \Omega_{N_{sum}N_{sum}} \big|}$, where $\Omega_{N_{sum}N_{sum}}$ is from the dataset $\{(\bm{x}_{n_k}, p_k)\}$ queried with our acquisition function. We can apply Fischer's inequality in order to bound this term by ${\frac{1}{2} \sum_{p, N_p>0} \log \big| I_{N_{p}} + \sigma_p^{-2} \eta_{p,p}(\{\bm{x}_{n_k}\}_{k}, \{\bm{x}_{n_k}\}_{k}) \big|}$, which is the sum of $I(y_p, f_p)$ given our GP prior. Then we use the maximum information gain to obtain the first part of our theorem. For the second part of our theorem, we follow the analysis from @Srinivas_2012. We bound the eigenvalues of MO kernel by similar quantities as used in @Srinivas_2012 and extend their analysis to obtain the bound. This proof also holds when the data is fully observed. See supplementary section \[supp\_section-proof\_of\_thm2\] for details. Here $\lim_{n \rightarrow \infty} n^{\frac{-2\nu}{2\nu+D(D+1)}} \log \ n $ and $\lim_{n \rightarrow \infty} \frac{(\log \ n)^{D+1}}{n}$ are $0$ according to L'H<span>ô</span>pital's rule. This theorem tells us that the predictive uncertainty converges to zero given data points queried by our acquisition function (without safety constraint). As in @NEURIPS2018_b197ffde, we now add the safety constraint into the theorem to obtain the final result. Notice that eq. can be considered as a non-constraint optimization problem within a set $S=\{\bm{x} | \xi(\bm{x} | z_{bar}, z_{mode}, \mathcal{D}) > 1-\delta \}$, except that $S$ differs in every iteration of the algorithm. \[thm-convergence\_guarantee\_safe\] We use the same unified expression $n$ of $N$ and $N_{sum}$. Let $\{\bm{\hat{x}_i} \in S_i \}_{i=1}^{n}$ be $n$ arbitrary input data drawn from iteration-dependent safe regions $S_i \subseteq \mathcal{X}$, and let $\{\Sigma_{k-1}(\hat{\bm{x}}_k), \Sigma_{k-1}(\hat{\bm{x}}_k, \hat{p}_k)\}$ be the predictive (co)variance of $\bm{\hat{\bm{x}}_k}$ conditioning on $k-1$ training data queried according to eq. \[prob-safe\_al\]. The other notation remains the same. Assuming fixed hyperparameters $\bm{\theta}$ and the same bounded conditions to the kernel as previously, then, similar to theorem \[thm-convergence\_guarantee\], ${\frac{1}{n}\sum_{k=1}^{n}|\Sigma_{k-1}(\bm{\hat{x}_k})|}$ and ${\frac{1}{n}\sum_{k=1}^{n} \Sigma_{k-1}(\bm{\hat{x}_k}, \hat{p}_k)}$ are bounded by ${\mathcal{O} \left( \frac{1}{n} \sum_{p=1}^{P} \gamma_p^{N_p} \right)}$, where $\gamma_p^{N_p}$ has exactly the same definition as in theorem \[thm-convergence\_guarantee\] (maximum information gain on $\mathcal{X}$). In addition, $\frac{1}{n} \sum_{p=1}^{P} \gamma_p^{N_p}$ has the same bounds as stated in theorem \[thm-convergence\_guarantee\]. Notice that the safety constraint is only defined on $\bm{x}$ and does not affect the selection of output component indices $\hat{p}_i$. The key to the proof is to inspect the sets carefully and build up the same inequalities. Details are in the supplementary material (section \[supp\_section-proof\_of\_thm3\]). With theorem \[thm-convergence\_guarantee\_safe\], we have the asymptotic convergence guarantee of the safe AL querying for a LMC. Extension to Convolution Processes ---------------------------------- As the second multi-output model, we consider the convolution processes [@higdon2002space; @JMLR:v12:alvarez11a], another popular type of MOGP model. We describe the detail of this model in supplementary section \[supp\_section-extended\_thm\_convolution\_process\]. We show in theorem \[thm-convolution\_process\_sal\_bound\] that the convergence guarantee we previously got also exists for a convolution process. Empirical Result {#section-experiments} ================ \[h\] As we are the first safe AL framework for MOGP, to the best of our knowledge, we carefully select benchmark datasets and methods for our algorithm. We compare our method on 2 simulated and a real-world dataset. All experiments confirm that our novel approach reaches smaller error level under a fixed sample budget as its comparison partners while fulfilling the safety constraints. We compare our approach with two competitors: (i) MOGP with random selection (RS\_MOGP, @LIU2018102) to which we add a safety constraint, (ii) safe AL with single output models (AL\_indGPs). The AL\_indGPs is adapted from @NEURIPS2018_b197ffde by removing the dynamic structure of data and concatenating uncertainty of different outputs for data queries (equivalent to our algorithm with $L=P$ and $W=I_P$, see section \[section-GP\]). Notice that the outputs were partially observed and, in the AL\_indGPs setting, a query of the $p$-th output component has no effect on the GP for any other output component(s). In addition, we have another pipeline AL\_MOGP\_nosafe, which is identical to our main pipeline AL\_MOGP except that the query is done without any safety constraint. This pipeline serves as a safety comparison reference. All the inferences are performed with Bayesian treatment. This avoids overfitting problems of maximum likelihood estimation, especially with small amount of data with which our safe AL operates. We describe the numerical detail in supplementary section \[supp\_section-exp\_detail\]. The code is also available [^1]. In addition to the main experiments, we compare setup of partially observed output to setup of fully observed output, where the result is provided in supplementary section \[supp\_section-ablation\]. Dataset: simulation with sin & sigmoid -------------------------------------- We first performed experiments on a simulation dataset generated with mixture of sin and sigmoid functions. This dataset has $\bm{X} \subseteq \mathbb{R}$, $\bm{Y} \subseteq \mathbb{R}^2$ and safety values $\bm{Z} \subseteq \mathbb{R}$. We refer to section \[supp\_section-exp\_detail\_toydata\] for detail. In a safety critical environment, it is important that the safety model $h$ is robust enough, to ensure safe exploration throughout the whole learning process [@10.1007/978-3-319-23461-8_9]. This can be seen from supplementary table \[table-safety\_toy\], which demonstrates the precision of the safety models in this experiment. In addition, we compare the portions of safe points within all queries after the AL is finished. AL\_MOGP achieves 96.24% (standard error 0.47%) while AL\_MOGP\_nosafe reaches only 26.75% (std. err. 0.67%). This shows the effect of applying a safety constraint. We also report the portions for other pipelines: RS\_MOGP has 99.06% (std. err. 0.34%) and AL\_indGPs has 96.75% (std. err. 0.39%). Root mean squared error (RMSE) values are shown in figure \[fig1\]. We observe that our approach, AL\_MOGP, converges the fastest. To achieve an average RMSE $\leq 0.4$ (which is roughly where the improvements of our framework become slower), AL\_MOGP needs 24 points (13-th iteration), RS\_MOGP needs 42 points (31-th iteration), AL\_indGPs needs 38 points (27-th iteration). Here the RMSE is not reported for AL\_MOGP\_nosafe because we only evaluate on safe data while AL\_MOGP\_nosafe can explore non-safe regions. In summary, our simulations demonstrate that our new approach achieves a smaller test errror than its competitors for a fixed sample budget (figure \[fig1\]), while at the same time fulfilling the safety requirements. Dataset: MOGP samples --------------------- We generated another simulation dataset with MOGPs. This dataset has dimension $D=2$ and $P=4$. The safety threshold is the 20%-quantile as the lower bound. See section \[supp\_section-exp\_detail\_GPdata\] for detail. Precisions of safety models are presented in supplementary table \[table-safety\_GP\]. Portions of safe points within all queries are 98.80% (std. err. 0.34%) for AL\_MOGP, 99.69% (std. err. 0.13%) for RS\_MOGP, 99.19% (std. err. 0.24%) for AL\_indGPs and 78.26% (std. err. 2.87%) for AL\_MOGP\_nosafe. Notice that 80% of the data are safe in the set. Figure \[fig1\] demonstrates that our approach is able to achieve a comparable test error with fewer samples. An average RMSE $\leq 0.6$ needs 83 points (44-th iteration) for our method, AL\_MOGP, 101 points (62-th iteration) for RS\_MOGP, and 115 points (76-th iteration) for AL\_indGPs. Engine Emission (EngE) Dataset ------------------------------ This dataset measures temperature and various chemical substances of a gasoline engine [^2]. Measurements of different output channels vary in effort and cost e.g. due to the installment of measurement equipment or clean-up and re-installment after certain amount of usage. Consequently, the measuring processes, especially of the expensive outputs, benefit significantly from the capability of reducing the number of required samples. Therefore, our safe AL-MOGP framework is highly suitable as it reduces the number of measurements by active learning and by exploiting the correlations among the components. We actively learn a MOGP model over the outputs HC and O2 while considering it to be safety critical that the temperature stays below a certain threshold (unlike in the previous simulations in which we required the safety values to be above a certain threshold). For experimental details, please see our supplementary section \[supp\_section-exp\_detail\]. Precisions of safety models in this experiment are in supplementary table \[table-safety\_OWEO\]. Notice that different output channels vary in their complexity and the temperature channel can be considered easier to learn than the channels of the main model, HC and O2. The portions of safe points within all queries are 99.21% (std. err. 0.18%) for AL\_MOGP, 98.81% (std. err. 0.28%) for RS\_MOGP, 98.93% (std. err. 0.22%) for AL\_indGPs and 85.59% (std. err. 0.35%) for AL\_MOGP\_nosafe. Note that 80% training data are safe by design. Figure \[fig2\] demonstrates that our approach shows competitive performance to the benchmark methods and is able to achieve a comparable test error with fewer samples. In this experiment, an average RMSE $\leq 1.0$ needs 54 points (7-th iteration) for our method, AL\_MOGP, and 63 points (16-th iteration) for its single-output alternative, AL\_indGPs. Applying random selection (RS\_MOGP) requires several hundred points, which is beyond the scope of this experiment. Conclusion {#section-conclusion} ========== Our novel safe AL approach for MOGPs allows safe exploration of a system in a doubly data-efficient manner: by actively selecting informative queries and by additionally exploiting the correlation between outputs. Our theoretical analysis shows that using the determinant or entropy of predictive (co)variance as the acquisition function guarantees the convergence of MOGPs for two state-of-the-art kernels. Our empirical results also demonstrate the applicability of our framework on a real-world engineering dataset, hereby outperforming its competitors under a fixed sample budget. Besides engineering and robotics applications, we envision that safe AL for MO will also become important in the clinical setting, e.g. [@s12911-020-1069-4], in which data efficiency is often required due to budget costs and safety constraints might arise due to data privacy issues. Acknowledgements {#acknowledgements .unnumbered} ================ This work was supported by Bosch Center for Artificial Intelligence, which provided finacial support, computers and GPU clusters. The Bosch Group is carbon neutral. Administration, manufacturing and research activities do no longer leave a carbon footprint. This also includes GPU clusters on which the experiments have been performed. Overview {#overview .unnumbered} ======== The supplementary materials are overviewed as follows. In section \[supp\_section-MOGP\], we demonstrate the full expression of MOGP matrices and kernels we use. Section \[supp\_section-additional\_lemmas\] and section \[supp\_section-additional\_lemmas\_proof\] provides all the additional lemmas and their proofs we need for our theoretical analysis. In section \[supp\_section-extended\_thm\_convolution\_process\], we extend our theoretical analysis in section \[section-thms\] to another popular MOGP model. Section \[supp\_section-proof\_of\_main\_lemma1\], \[supp\_section-proof\_of\_thm2\], \[supp\_section-proof\_of\_thm3\], and \[supp\_section-proof\_of\_thm4\] are our proofs for lemma and theorems in the main paper. Finally, in section \[supp\_section-exp\_detail\] and \[supp\_section-ablation\], we describe our experiment in detail and show ablation study and additional figures. Multi-output Gaussian Process (MOGP) {#supp_section-MOGP} ==================================== Full expression of MOGP covariance ---------------------------------- Recall $\bm{X} = (\bm{x}_1, ..., \bm{x}_N)$, $\bm{Y} = (y_{11}, ..., y_{1N}, ..., y_{P1}, ..., y_{PN})^T$, and $$\eta_{p, p'}(\bm{x}, \bm{x}') \coloneqq \sum_{l=1}^{L} W_{pl}W_{p'l} k_{l}(\bm{x}, \bm{x}').$$ Notice that for all indices $p, p' \in \{1, ..., P\}$, $$\eta_{p, p'}(\bm{X}, \bm{X}) = \begin{pmatrix} \eta_{p, p'}(\bm{x}_1, \bm{x}_1) & \dots & \eta_{p, p'}(\bm{x}_1, \bm{x}_N) \\ & \vdots & \\ \eta_{p, p'}(\bm{x}_N, \bm{x}_1) & \dots & \eta_{p, p'}(\bm{x}_N, \bm{x}_N) \end{pmatrix} \in \mathbb{R}^{N \times N},$$ $$\eta_{p, p'}(\bm{X}, \cdot) = \begin{pmatrix} \eta_{p, p'}(\bm{x}_1, \cdot)\\ \vdots\\ \eta_{p, p'}(\bm{x}_N, \cdot) \end{pmatrix} \in \mathbb{R}^{N \times 1}.$$ Then $$\begin{gathered} \Omega_{NN} = \begin{pmatrix} \eta_{1, 1}(\bm{X}, \bm{X}) & \dots & \eta_{1, P}(\bm{X}, \bm{X}) \\ & \vdots & \\ \eta_{P, 1}(\bm{X}, \bm{X}) & \dots & \eta_{P, P}(\bm{X}, \bm{X}) \end{pmatrix} \in \mathbb{R}^{PN \times PN},\end{gathered}$$ $$\begin{gathered} \Omega_{N*} = \begin{pmatrix} \eta_{1, 1}(\bm{X}, \bm{x}_*) & \dots & \eta_{1, P}(\bm{X}, \bm{x}_*) \\ & \vdots & \\ \eta_{P, 1}(\bm{X}, \bm{x}_*) & \dots & \eta_{P, P}(\bm{X}, \bm{x}_*) \end{pmatrix} \in \mathbb{R}^{PN \times P},\end{gathered}$$ $$\begin{gathered} \Omega_{**} = \begin{pmatrix} \eta_{1, 1}(\bm{x}_*, \bm{x}_*) & \dots & \eta_{1, P}(\bm{x}_*, \bm{x}_*) \\ & \vdots & \\ \eta_{P, 1}(\bm{x}_*, \bm{x}_*) & \dots & \eta_{P, P}(\bm{x}_*, \bm{x}_*) \end{pmatrix} \in \mathbb{R}^{P \times P},\end{gathered}$$ Full expression of observation noise variances ---------------------------------------------- $$\begin{gathered} diag \left( \{\sigma_p^2\}_{p=1}^{P} \right) \otimes I_N = \begin{pmatrix} \sigma_1^2 I_N & & \bm{0}\\ & \ddots & \\ \bm{0} & & \sigma_P^2 I_N \end{pmatrix} \in \mathbb{R}^{PN \times PN},\end{gathered}$$ $$\begin{gathered} diag \left( \{\sigma_{p_k}^2\}_{k=1}^{N_{sum}} \right) = \begin{pmatrix} \sigma_{p_1}^2& & \bm{0}\\ & \ddots & \\ \bm{0} & & \sigma_{p_{N_{sum}}}^2 \end{pmatrix} = \begin{pmatrix} \sigma_{1}^2 I_{N_1}& & \bm{0}\\ & \ddots & \\ \bm{0} & & \sigma_{P}^2 I_{N_P} \end{pmatrix} \in \mathbb{R}^{N_{sum} \times N_{sum}}.\end{gathered}$$ Derivation of MOGP posterior ---------------------------- The GP prior tells us that $$\begin{gathered} \begin{pmatrix} \bm{f}(\bm{x_*}) \\ \bm{Y} \end{pmatrix} = \begin{pmatrix} f_1(\bm{x_*}) \\ \vdots \\ f_P(\bm{x_*}) \\ y_{11} \\ \vdots \\ y_{1N} \\ \vdots \\ y_{P1} \\ \vdots \\ y_{PN} \end{pmatrix} \sim \mathcal{N} \left( \bm{0}, \begin{pmatrix} \Omega_{**} & \Omega_{N*}^T \\ \Omega_{N*} & \Omega_{NN} + diag(\{\sigma_p^2\}_{p=1}^{P}) \otimes I_N \end{pmatrix} \right)\end{gathered}$$ From appendix A.2 of @3569, we then have $$\begin{gathered} \bm{f}(\bm{x}_*) | \bm{Y} \sim \mathcal{N}\left( \Omega_{N*}^T (\Omega_{NN} + diag(\{\sigma_i^2\}_{p=1}^{P}) \otimes I_N)^{-1} \bm{Y}, \Omega_{**} - \Omega_{N*}^T (\Omega_{NN} + diag(\{\sigma_i^2\}_{p=1}^{P}) \otimes I_N)^{-1} \Omega_{N*} \right)\end{gathered}$$ Commonly used kernels (for single-output GPs) --------------------------------------------- A kernel $k$ is said to be stationary if for all $\bm{x}, \bm{x}'$, $k(\bm{x}, \bm{x}')$ depends only on $\bm{x} - \bm{x}'$. We denote a stationary kernel also by $k(\bm{x} - \bm{x}')$. Furthermore, if given a norm $|\cdot|$, $k(\bm{x}, \bm{x}')$ depends only $|\bm{x}-\bm{x}'|$, then kernel $k$ is isotropic. In this case $k(\bm{x}, \bm{x}')$ is also denoted by $k(r)$ for $r \in \mathbb{R}^{+} \cup \{0\}$. We always use L2-norm and mainly consider isotropic kernels. #### $\nu$-Mat<span>é</span>rn kernel $\nu$ is the smoothing parameter. $$\begin{aligned} k(r)=\sigma_{kernel}^2\frac{2^{1-\nu}}{\Gamma(\nu)} \left( \frac{\sqrt{2\nu}r}{\rho} \right)^\nu K_\nu\left( \frac{\sqrt{2\nu}r}{\rho} \right),\end{aligned}$$ where $K_\nu (\cdot)$ is a modified Bessel function, scale $\sigma_{kernel}^2 > 0$ and lengthscale $\rho>0$ are hyperparameters. The followings are some commonly chosen $\nu$: $$\begin{aligned} \nu = 1/2, k(r) &= \sigma_{kernel}^2 exp \left(- \frac{r}{\rho}\right) \\ \nu = 3/2, k(r) &= \sigma_{kernel}^2 \left(1+\frac{\sqrt{3}r}{\rho}\right) exp \left(- \frac{\sqrt{3}r}{\rho}\right) \\ \nu = 5/2, k(r) &= \sigma_{kernel}^2 \left(1+\frac{\sqrt{5}r}{\rho} + \frac{5r^2}{3\rho^2}\right) exp \left(- \frac{\sqrt{5}r}{\rho}\right)\end{aligned}$$ #### Squared exponential kernel $$\begin{aligned} k(r)= \sigma_{kernel}^2 exp \left(- \frac{r^2}{\rho} \right),\end{aligned}$$ where scale $\sigma_{kernel}^2 > 0$ and lengthscale $\rho \geq 0$ are hyperparameters. #### Squared exponential kernel - multivariate $$\begin{aligned} k(\bm{x} - \bm{x}')= \sigma_{kernel}^2 exp \left(- \frac{1}{2} (\bm{x} - \bm{x}')^T \Lambda_{kernel} (\bm{x} - \bm{x}') \right),\end{aligned}$$ where scale $\sigma_{kernel}^2 > 0$ and positive definite matrix $\Lambda_{kernel}$ are hyperparameters. This kernel is not isotropic but is still stationary. Eigen-decomposition of SE kernels --------------------------------- We write a unit scale SE kernel in the form $$\begin{aligned} k(\bm{x}, \bm{x}')=exp(- \frac{1}{\rho} |\bm{x} - \bm{x}'|_2^2),\end{aligned}$$ for some positive lengthscale $\rho$. Here we additionally introduce eigen-decomposition of such kernel. We need this information to prove our theorems in later sections. Let $\mathcal{X} \subseteq \mathbb{R}^D$ be a compact set and $\mu(\bm{x} \in \mathcal{X})=\mathcal{N}(\bm{x} | \bm{0}, (4a)^{-1}I_D)$ be a measure. Here the variance is formed as $(4a)^{-1}$ to make the later constants clean, but it can essentially be any positive real number. If $D=1$ then SE kernel has $s$-th eigenvalues $\lambda_s$ ($\lambda_1 \geq \lambda_2 \geq ...$) and the corresponding eigenvector $\Psi_s(\cdot)$ given by eq. (43)-(45) of @Zhu1998GROFDLM as $$\begin{aligned} \label{supp_eqn-SE_eigenvalue} \lambda_s &= \sqrt{\frac{2a}{A}}B^s \\ \label{supp_eqn-SE_eigenfunction} \Psi_s(x) &= exp(-(c-a)x^2) H_s(\sqrt{2c}x),\end{aligned}$$ where $$\begin{aligned} H_s(x) = (-1)^s exp(x^2) \frac{d^s}{dx^s} exp(-x^2)\\ A = a + \frac{1}{\rho} + c, \ \ c = \sqrt{a^2 + 2a \frac{1}{\rho}}, \ \ B = \frac{1}{A \rho}.\end{aligned}$$ @4494702-appendix II further derived that if $D \geq 2$ $$\begin{aligned} \label{supp_eqn-SE_eigenvalue_D} \lambda_s \leq \mathcal{O}(B^{s^{1/D}}).\end{aligned}$$ We can see that $0<B<1$, which is an important property later. Notice that such eigen-decomposition means $$\begin{aligned} \label{supp_eqn-eigen_integral} \int &k(\bm{x},\bm{x}') \Psi_s(\bm{x}) d \mu(\bm{x}) = \lambda_s \Psi(\bm{x}'), \text{and} \\ \label{supp_eqn-eigen-decomposition} &k(\bm{x}, \bm{x}') = \sum_{s \geq 1} \lambda_s \Psi_s(\bm{x}) \Psi_s^*(\bm{x}').\end{aligned}$$ Please see section 4.3 of @3569, Mercer's theorem for more details. Additional Lemmas {#supp_section-additional_lemmas} ================= Inequalities ------------ Before going further, we would like to introduce few inequalities we use later. Notice that the proofs of all of the following statements are in section \[supp\_section-additional\_lemmas\_proof\]. \[supp\_ineq-determinant\] Given positive semidefinite matrices $Q_1$ and $Q_2$, $det(Q_1+Q_2) \geq det(Q_1) + det(Q_2)$. \[supp\_ineq-determinant\_posterior\_cov\] For any $N$ and any $\bm{x}_*$, the predictive covariance $\Sigma(\bm{x}_{*})$ shown in equation \[eqn-mo\_posterior\_cov\] satisfies $det(\Omega_{**}) \geq det \left( \Sigma(\bm{x}_{*}) \right) $ and, similarly, the variance $\Sigma(\bm{x}_{*}, p_*)$ in eqation \[eqn-mo\_posterior\_cov\_poo\] satisfies $\eta_{p_*, p_*}(\bm{x}_{*}, \bm{x}_{*}) \geq \Sigma(\bm{x}_{*}, p_*)$. \[supp\_lemma-omega\_bound\] Recall that kernel $\eta_{p, p'}(\cdot, \cdot)=\sum_{l=1}^{L}W_{pl}W_{p'l} k_l(\cdot, \cdot)$ for some kernels $k_l$. With finite $P$ and $L$, if $\hat{w} \geq |W_{pl}|$ and $\hat{v} \geq k_l(\cdot, \cdot) \geq 0$, for all $p, l$, then $det(\Omega_{**}) \leq (L \hat{w}^2 \hat{v})^P$ and $\eta_{p_*, p_*}(\cdot, \cdot) \leq L \hat{w}^2 \hat{v}$. The last lemma is adapted from Weyl's inequality for matrices. \[supp\_lemma-weyl\_ineq\] Let $\{ A_l \in \mathbb{R}^{M \times M} \}_{l=1}^{L}$ be Hermitian matrices, and let $B = \sum_{l=1}^{L}A_l$. Let $\alpha_{l,1} \geq \alpha_{l,2} \geq ... \geq \alpha_{l,M}$ be eigenvalues of $A_l$ and $\beta_1 \geq ... \geq \beta_M$ be eigenvalues of $B$. Then $\beta_s \leq \sum_{l=1}^{L} \alpha_{l, \left[\frac{s-1}{L}+1\right]}$, where $[r]$ is the largest integer such that $r \geq [r]$, for all $r \in \mathbb{R}$. Mutual information ------------------ In addition, we here provide the mutual information in terms of the GP posterior, which can be used to obtain lemma \[lemma-predictive\_cov\_bound\] and theorem \[thm-convergence\_guarantee\] in the main script. \[supp\_lemma-mutual\_information\] Given data points $\{(\bm{x}_{n_i}, p_i)\}_{i=1}^{k-1}$ or $\{\bm{x}_i\}_{i=1}^{k-1}$, let $\hat{\Sigma}_{k-1}(\bm{x}_{n_k}, p_k)$ be predictive variance of $f_{p_k}(\bm{x}_{n_k}) | \{y_{p_{i}n_{i}}\}_{i=1}^{k-1}$ for partially observed output, and $\hat{\Sigma}_{k-1}(\bm{x}_{k})$ predictive covariance of $\bm{f}(\bm{x}_{k}) | \{\bm{y}_{i}\}_{i=1}^{k-1}$ for fully observed output. In addition, let $\hat{\Sigma}_{0}(\bm{x}_{n_1}, p_1)=\eta_{p_{n_1}}(\bm{x}_{n_1}, \bm{x}_{n_1})$ and $\hat{\Sigma}_{0}(\bm{x}_{1})=\bm{\eta}(\bm{x}_{1}, \bm{x}_{1})$ for the two settings. The mutual information is then described as follows: $$\label{supp_eqn-mutual_information_poo} I \left( \bm{Y}_{\phi}, \{ f_{p_k}(\bm{x}_{n_k}) \}_{k=1}^{N_{sum}} \right) = \frac12 \sum_{k=1}^{N_{sum}} log \left( 1 + \frac{1}{\sigma_{p_k}^{2}}\hat{\Sigma}_{k-1}(\bm{x}_{n_k}, p_k) \right) ,$$ $$\label{supp_eqn-mutual_information} I \left( \bm{Y}, \{ \bm{f}(\bm{x}_{k}) \}_{k=1}^{N} \right) = \frac12 \sum_{k=1}^{N} log \left( \big| I_P + diag(\{\sigma_i^{2}\}_{i=1}^{P})^{-1} \hat{\Sigma}_{k-1}(\bm{x}_{k}) \big| \right) .$$ Notice that this lemma does not involve active learning query yet. It only correlates posterior (co)variance to mutual information. This is why I use the notation $\hat{\Sigma}_{k-1}$ which is different from $\Sigma_{k-1}$ in the main paper and in the next section of this supplementary content. Kernel on rotated data ---------------------- We also need a lemma about kernel eigenvalues for later analysis. \[supp\_lemma-kernel\_rotated\_data\] Let $\mathcal{X} \in \mathbb{R}^D$ be a compact set, $Q$ be an orthonormal matrix (rotation matrix), $\mathcal{U}=Q \mathcal{X}$. Given distribution $\mu(\cdot) = \mathcal{N}(\cdot | \bm{0}, (4a)^{-1}I)$ for some positive constant $a$, and given kernel $k(\bm{x}, \bm{x}')$ and $k_Q(\bm{u}, \bm{u}')$ s.t. $k_Q(\bm{u}, \bm{u}') = k(\bm{x}, \bm{x}')$ for $\bm{u}=Q\bm{x}, \bm{u}'=Q\bm{x}'$ and max $k(\cdot, \cdot)=1$. Then $k$ and $k_Q$ must have the same eigenvalues w.r.t. the same distribution $\mu$. Proof of Additional Lemmas {#supp_section-additional_lemmas_proof} ========================== Proof of lemma \[supp\_ineq-determinant\] ----------------------------------------- Let $\{q_{1,i}\}$ and $\{q_{2,i}\}$ be the eigenvalues of $Q_1$ and $Q_2$, respectively, then $q_{j,i} \geq 0$ implies $$\begin{aligned} det(Q_1+Q_2) &= \prod_{i}(q_{1,i} + q_{2,i}) \\& \geq \prod_{i}q_{1,i} + \prod_{i}q_{2,i} = det(Q_1) + det(Q_2).\end{aligned}$$ Proof of corollary \[supp\_ineq-determinant\_posterior\_cov\] ------------------------------------------------------------- Let $B=\Omega_{NN} + diag\{\sigma_i^2\} \otimes I_N$. As $B$ is a positive definite matrix, so is its inverse. This means $$\begin{aligned} \forall a \in \mathbb{R}^{PN} \setminus \bm{0}, &a^T B ^{-1} a > 0 \\ \Rightarrow \forall b \in \mathbb{R}^{P}, &b^T \Omega_{N*}^T B^{-1} \Omega_{N*} b \\=\ &(\Omega_{N*} b)^T B^{-1} (\Omega_{N*} b) \geq 0,\end{aligned}$$ which implies that $\Omega_{N*}^T B^{-1} \Omega_{N*}$ is semi-positive definite. Notice that $\Omega_{N*} b$ might be a zero vector. Apply lemma \[supp\_ineq-determinant\], let $$\begin{aligned} Q_1 &= \Sigma(\bm{x}_{*}),\\ Q_2 &= \Omega_{N*}^T B^{-1} \Omega_{N*},\\\end{aligned}$$ then $Q_1 + Q_2 = \Omega_{**}$ implies that $$\begin{aligned} det(\Omega_{**}) &\geq det(\Sigma(\bm{x}_{*})) + det(\Omega_{N*}^T B^{-1} \Omega_{N*}) \\ &\geq det(\Sigma(\bm{x}_{*})).\end{aligned}$$ To addapt similar result to eq. \[eqn-mo\_posterior\_cov\_poo\], it is actually not necessary to use lemma \[supp\_lemma-omega\_bound\], but it is easier for applying later if we put the statements together. Let $B = \Omega_{N_{sum}N_{sum}} + diag(\{\sigma_{p_k}^2\}_{k=1}^{N_{sum}})$, then $[\Omega_{N_{sum}*}]_{all,p*}^T B^{-1} [\Omega_{N_{sum}*}]_{all,p*}$ is a non-negative scalar. Therefore, from eq. \[eqn-mo\_posterior\_cov\_poo\], we directly see that $$\begin{aligned} \eta_{p_*, p_*}(\bm{x}_{*}, \bm{x}_{*}) &\geq \Sigma(\bm{x}_{*}, p_*).\end{aligned}$$ Proof of lemma \[supp\_lemma-omega\_bound\] ------------------------------------------- For any $N$ and any $\bm{x}_*$, since eigen values of semi-positive definite matrices are non-negative, inequality of arithmetic and geometric means gives us $$\begin{aligned} det(\Omega_{**}) &= \prod \{eigenvalues\} \\&\leq (\frac{\sum \{eigenvalues\}}{P})^P \\&= \left( \frac{1}{P} trace(\Omega_{**}) \right) ^P \\&= \left( \frac{1}{P} \sum_{p=1}^{P} \eta_{p,p}(*, *) \right)^P \\&= \left( \frac{1}{P} \sum_{p=1}^{P} \sum_{l=1}^{L} W_{pl}^2 k_l(*, *) \right) ^P \\&\leq \left( \frac{1}{P} \sum_{p=1}^P\sum_{l=1}^L \hat{w}^2 \hat{v} \right) ^P \\&= (L \hat{w}^2 \hat{v})^P.\end{aligned}$$ Meanwhile, from line 4 we have also obtained $\eta_{p_*, p_*}(\cdot, \cdot) \leq L \hat{w}^2 \hat{v}$. Proof of lemma \[supp\_lemma-weyl\_ineq\] ----------------------------------------- First notice that $\left[\frac{s-1}{L}+1\right] \in \mathbb{N}$ for all $s \in \mathbb{N}$. We use the induction. 1. When $L=2$, Weyl's inequality tells us that $$\begin{aligned} \beta_s &\leq \alpha_{1, i} + \alpha_{2, j},\\ if \ s &\geq i + j - 1. \end{aligned}$$ We clearly see that $$\begin{aligned} s = 2\left(\frac{s-1}{2}+1\right)-1 &\geq \left[\frac{s-1}{2}+1\right] + \left[\frac{s-1}{2}+1\right] - 1,\\ \text{so } \beta_s &\leq \alpha_{1, \left[\frac{s-1}{2}+1\right]} + \alpha_{2, \left[\frac{s-1}{2}+1\right]}. \end{aligned}$$ 2. For any integer $T \geq 2$, assume $\beta_s \leq \sum_{l=1}^{T} \alpha_{l, \left[\frac{s-1}{T}+1\right]}$ for $B = \sum_{l=1}^{T}A_l$. Let $\hat{B}= \sum_{l=1}^{T+1}A_l$, and let $\{\hat{\beta}_i\}_{i=1}^{M}$ be it's eigenvalues ranking in order. Notice that by definition, it is easy to see that $[r]+1 = [r+1], \forall r \in \mathbb{R}$. $$\begin{aligned} \hat{B} &= B + A_{T+1},\\ \text{Weyl's inequality and } s &\geq \left[\frac{Ts+1}{T+1}\right] + \left[\frac{s-1}{T+1}+1\right] - 1\\ \Rightarrow \hat{\beta}_s &\leq \beta_{\left[\frac{Ts+1}{T+1}\right]} + \alpha_{T+1, \left[\frac{s-1}{T+1}+1\right]},\\ \text{Induction hypothesis} \Rightarrow \hat{\beta}_s &\leq \left( \sum_{l=1}^{T} \alpha_{l, \left[ \frac{1}{T} \left[\frac{Ts+1}{T+1}-1\right] +1 \right]} \right) + \alpha_{T+1, \left[\frac{s-1}{T+1}+1\right]}. \end{aligned}$$ Now notice that $$\begin{aligned} \frac{Ts+1}{T+1}-2 \leq &\left[\frac{Ts+1}{T+1}-1\right]\leq \frac{Ts+1}{T+1}-1\\ \Rightarrow \frac{Ts-T}{T+1}-1 \leq &\left[\frac{Ts+1}{T+1}-1\right]\leq \frac{Ts-T}{T+1}\\ \Rightarrow \left[\frac{s-1}{T+1}-\frac{1}{T}\right] + 1 \leq &\left[ \frac{1}{T} \left[\frac{Ts+1}{T+1}-1\right] + 1 \right]\leq \left[\frac{s-1}{T+1}\right]+1. \end{aligned}$$ Denote integer $I = \left[\frac{s-1}{T+1}\right]$. The previous line tells us either $I = \left[\frac{s-1}{T+1}-\frac{1}{T}\right]$ or $I > \left[\frac{s-1}{T+1}-\frac{1}{T}\right]$. Let's now inspect what they imply to the index of our interest $\left[\frac{s-1}{T+1}+1\right]=I+1$. If $I = \left[\frac{s-1}{T+1}-\frac{1}{T}\right]$ $$\begin{aligned} I \leq \left[ \frac{1}{T} \left[\frac{Ts+1}{T+1}-1\right] \right]\leq I \Rightarrow \left[ \frac{1}{T}\left[\frac{Ts+1}{T+1}-1\right] \right] = I, \end{aligned}$$ if $I > \left[\frac{s-1}{T+1}-\frac{1}{T}\right]$ $$\begin{aligned} &\Rightarrow I+\frac{1}{T}> \frac{s-1}{T+1} \geq I\\ &\Rightarrow TI +1 > \frac{Ts-T}{T+1} \geq TI\\ TI \in \mathbb{Z}&\Rightarrow TI+1 > \left[ \frac{Ts+1}{T+1}-1 \right] \geq TI\\ &\Rightarrow \left[ \frac{Ts+1}{T+1}-1 \right] = TI\\ &\Rightarrow \left[ \frac{1}{T}\left[\frac{Ts+1}{T+1}-1\right] \right] = I. \end{aligned}$$ We therefore know that the index $\left[ \frac{1}{T}\left[ \frac{Ts+1}{T+1}-1 \right] + 1 \right]$ is exactly $\left[\frac{s-1}{T+1}+1\right]$, which means $$\begin{aligned} \hat{\beta}_s &\leq \left( \sum_{l=1}^{T} \alpha_{l, \left[ \frac{1}{T} \left[\frac{Ts+1}{T+1}-1\right] +1 \right]} \right) + \alpha_{T+1, \left[\frac{s-1}{T+1}+1\right]}\\ &=\left( \sum_{l=1}^{T} \alpha_{l, \left[\frac{s-1}{T+1}+1\right]} \right) + \alpha_{T+1, \left[\frac{s-1}{T+1}+1\right]}\\ &=\sum_{l=1}^{T+1} \alpha_{l, \left[\frac{s-1}{T+1}+1\right]} \end{aligned}$$ 3. Then by induction we have proved the lemma. Proof of lemma \[supp\_lemma-mutual\_information\] -------------------------------------------------- We follow the idea of lemma 1 in @NEURIPS2018_b197ffde but extend to multioutput kernel. We prove the 2 cases separately. #### Proof of lemma \[supp\_lemma-mutual\_information\] for partially observed outputs By definition, $I \left( \bm{Y}_{\phi}, \{ f_{p_k}(\bm{x}_{n_k}) \}_{k=1}^{N_{sum}} \right) = H \left( \bm{Y}_{\phi} \right) - H \left( \bm{Y}_{\phi} | \{ f_{p_k}(\bm{x}_{n_k}) \}_{k=1}^{N_{sum}} \right) $. As $\bm{Y}_{\phi} | \{ f_{p_k}(\bm{x}_{n_k}) \}_{k=1}^{N_{sum}}$ are i.i.d. Gaussian noises, i.e. covariance $=diag \left( \{\sigma_{p_k}^2\}_{k=1}^{N_{sum}} \right) $, we immediately have $$\begin{gathered} H \left( \bm{Y}_{\phi} | \{ f_{p_k}(\bm{x}_{n_k}) \}_{k=1}^{N_{sum}} \right) = \sum_{k=1}^{N_{sum}}\frac{1}{2} log
{'source': 'arXiv_out/2203'}