'In-Between' Uncertainty in Bayesian Neural Networks
Andrew Y. K. Foong Yingzhen Li José Miguel Hernández-Lobato Richard E. Turner


[
`In-Between' Uncertainty in Bayesian Neural Networks

















Andrew Y. K. Foongcam
Yingzhen Limicrosoft
José Miguel Hernández-Lobatocam,microsoft,turing
Richard E. Turnercam,microsoft


camDepartment of Engineering, University of Cambridge, Cambridge, United Kingdom
microsoftMicrosoft Research Cambridge, United Kingdom
turingAlan Turing Institute, London, United Kingdom

Andrew Foongykf21@cam.ac.uk




Machine Learning, ICML, Bayesian Neural Networks, Regression, Approximate Inference, Uncertainty

0.3in
]









  



We describe a limitation in the expressiveness of the predictive uncertainty estimate given by mean-field variational inference (MFVI), a popular approximate inference method for Bayesian neural networks. In particular, MFVI fails to give calibrated uncertainty estimates in between separated regions of observations. This can lead to catastrophically overconfident predictions when testing on out-of-distribution data. Avoiding such overconfidence is critical for active learning, Bayesian optimisation and out-of-distribution robustness. We instead find that a classical technique, the linearised Laplace approximation, can handle `in-between' uncertainty much better for small network architectures. 





§ INTRODUCTION


Neural networks have been shown to be extremely successful for supervised learning. However, they are known to underestimate their uncertainty when trained by maximum likelihood or maximum a posteriori (MAP) methods. A neural network that returns reliable uncertainty estimates whilst maintaining the computational and statistical efficiency of standard networks would have numerous applications in active learning, reinforcement learning and critical decision-making tasks <cit.>. 

A variety of techniques have been proposed to obtain uncertainty estimates for neural networks in computationally efficient ways <cit.>. Among these, mean-field variational inference (MFVI) is a widely used approximate inference method that gives state-of-the-art performance in non-linear regression. On the commonly used UCI regression benchmark, MFVI with the reparameterisation trick <cit.> often outperforms Stochastic Gradient Langevin Dynamics, Probabilisitic Back-Propagation and ensemble methods <cit.>, and is competitive with Monte Carlo Dropout.[See <cit.> for a recent strong baseline for Monte Carlo Dropout on the UCI regression datasets.]

Performance on the UCI datasets is usually measured by held out log-likelihood. This represents both accuracy and uncertainty quantification, since it penalises methods that are overconfident in addition to being inaccurate. It is therefore perhaps surprising that MFVI performs poorly on sequential decision making tasks that require good uncertainty quantification, such as contextual bandits <cit.>. To perform well, a method must `know what it knows, and what it doesn't know': it should have high confidence near clusters of observations, and low confidence elsewhere. More specifically, a well-calibrated network should predict with high uncertainty far from data, as well as in regions between separated clusters of observations. However, the current UCI benchmark is not suitable for evaluating `in-between' uncertainty, as the test set is obtained by uniformly subsampling the full dataset. We therefore design another UCI benchmark to test for in-between uncertainty, by taking the `middle region' of the full dataset as the test set. We find that although MFVI performs well on the standard UCI benchmark, it can fail catastrophically on the in-between version, showing the detrimental effect the mean-field approximation has on in-between uncertainty. In contrast, a classical technique, the linearised Laplace approximation <cit.>, performs well on both. 

it should have high confidence near regions with a lot of observed data, and low confidence far from the data region and between separated clusters of data. Testing uncertainty quality in these different scenarios requires different kinds of datasets. We therefore test on two versions of the UCI datasets: one with the test set uniformly subsampled from the dataset, and one with the test set taken from the `middle' of the dataset, to test for `in-between' uncertainty, by which we mean the ability to
express increased uncertainty in between well-separated regions of observations. We find that a classical technique, the linearised Laplace approximation <cit.>, performs well on both. However, we find that although MFVI performs well on the former, it can fail catastrophically on the latter, showing the detrimental effect the mean-field approximation has on in-between uncertainty.



A single number from a single kind of test set split cannot be indicative of performance in all of these scenarios. Here we show that the standard UCI dataset splits are not good tests of `in-between' uncertainty, by which we mean the ability to express increased uncertainty in between well-separated regions of low uncertainty. This is a desirable property for any well-calibrated uncertainty estimate. Furthermore, we show that whereas MFVI fails to provide in-between uncertainty, a classical technique, the linearised Laplace approximation <cit.>, can. We provide comparisons on a 1D toy dataset and also on two versions of the UCI  datasets: the standard train/test splits, and splits designed to test for in-between uncertainty.




§ METHODS


This paper focuses on two approximate inference techniques for Bayesian Neural Networks (BNNs): Variational Inference (VI) and the Laplace approximation. We consider networks whose output f_θ(𝐱) given input 𝐱 and parameters θ is interpreted as the mean of a Gaussian distribution with homoscedastic output noise variance σ_o^2. We place a diagonal Gaussian prior over θ, here written as a column vector of all weights and biases in the network.

Variational Inference. Let the posterior distribution over θ given a dataset 𝒟 = { (𝐱_n, y_n) }_n=1^N be p(θ| 𝒟). VI approximates this posterior with a simpler distribution q_ϕ(θ). The parameters ϕ are learned by optimising a simple Monte Carlo estimate of the Evidence Lower Bound (ELBO):

    ℒ(ϕ)    = ∑_n=1^N 𝔼_q_ϕ[ log p(y_n|θ, 𝐱_n) ] - KL(q_ϕ(θ)||p(θ))
       ≈1/M∑_m=1^M ∑_n=1^N log p(y_n|θ_m, 𝐱_n) - KL(q_ϕ(θ)||p(θ)),

where θ_m are sampled from q_ϕ(θ). Optimising the ELBO minimises the KL-divergence between q_ϕ(θ) and the true posterior. Once ϕ is learned, we can make predictions by Monte Carlo sampling from q_ϕ(θ):

    p(y^*|𝐱^*, 𝒟)    = 𝔼_p(θ|𝒟)[p(y^*|𝐱^*, θ)] 
       ≈𝔼_q_ϕ(θ)[p(y^*|𝐱^*, θ)]
       ≈1/M∑_m=1^M p(y^*|𝐱^*, θ_m).

A common and scalable choice for the form of q_ϕ(θ) is the mean-field Gaussian approximation (MFVI), which is a fully factorised Gaussian distribution. Another choice is to let q_ϕ(θ) be a full covariance Gaussian (FCVI). This is more flexible, but the number of variational parameters ϕ is now quadratic in the number of parameters in the network. 

Laplace Approximation. The Laplace approximation <cit.> finds a mode θ_MAP of the posterior, and sets the approximate posterior to q(θ) = 𝒩(θ; μ, Σ) with μ = θ_MAP. Σ is set such that the curvature of log p(θ|𝒟) matches the curvature of the logarithm of the Gaussian approximation at θ_MAP, that is:

    Σ = -[ ∇_θ∇_θlog p(θ|𝒟) |_θ = θ_MAP]^-1.

In words, Σ is the negative inverse Hessian evaluated at the MAP solution. In practice we use the Gauss-Newton matrix, which is guaranteed to be positive semi-definite, and can be evaluated using only first derivatives: 

    Σ = -[ 1/σ_o^2∑_n=1^N g(𝐱_n) g(𝐱_n)^𝖳 + diag(p) ]^-1.

Here g(𝐱_n) = ∇_θ f_θ(𝐱_n) |_θ = θ_MAP and p is a vector whose ith element is 1/σ^2_i, where σ^2_i is the prior variance of θ_i.

Once θ_MAP and Σ are obtained, there are two different ways to make predictions. The first is to Monte Carlo sample from the approximate posterior as in equation (<ref>). We refer to this method as Sampled Laplace (SL). Unfortunately, the Laplace approximation is known to cause severe underfitting <cit.>. An alternative procedure which empirically alleviates this is to linearise the output of the network about θ_MAP. This leads to a linear Gaussian model that can be solved exactly for the predictive distribution:

    p(y^*|𝐱^*, 𝒟) ≈𝒩(y^*; f_θ_MAP(𝐱^*), σ_o^2 + g(𝐱^*)^𝖳Σ  g(𝐱^*));

see Appendix <ref> for details. We refer to this method as Linearised Laplace (LL).

Finding θ_MAP is identical to standard neural network training. Once at a mode, calculating the Gauss-Newton matrix requires one backward pass for each element of the dataset, which has a cost that scales linearly in the number of observations. Lastly the Laplace approximation requires inverting this matrix which has cubic cost in the number of parameters. This is still tractable for the smaller networks typically considered for regression on UCI datasets. Recent work has applied Kronecker-Factored Approximate Curvature (K-FAC) to obtain a scalable method; however they used sampling instead of linearisation and had to take steps to mitigate the underfitting problem inherent in the Laplace approximation <cit.>.



§ EXPERIMENTS


To test for in-between uncertainty, we compare these methods on two tasks. The first is a synthetic 1D regression dataset formed by adding Gaussian noise to a sine wave and observing two separated clusters of input points. The second are the UCI regression datasets.



Synthetic 1D Dataset. We plot results for four inference methods on the 1D dataset: MFVI, FCVI, LL and Hamiltonian Monte Carlo (HMC), in Figure <ref>.[SL is not shown as the fit is so poor that the error bars completely fill the figure. See <cit.> pp. 88 - 91.] We use a single hidden layer network with tanh non-linearities[ReLUs caused problems with LL, see Appendix <ref>.] and 50 hidden units. Diagonal Gaussian priors are used for all networks, and the observation noise σ_o is fixed to 0.1, the true value. Details and additional results are in Appendix <ref>. We see that MFVI fails to represent in-between uncertainty: its error bars are of similar magnitude in the data region and the in-between region. FCVI has larger uncertainty in the middle, but is slightly underconfident in the data region. LL and HMC show high confidence in the data region and increased uncertainty in between, showing that MFVI's failure is rooted in approximate inference, not the model class. 

There are several reasons for MFVI's overconfidence. First, we show in Appendix <ref> that a single hidden layer BNN with ReLU activations and with deterministic input weights and mean-field (possibly non-Gaussian) output weights must have an output variance that is convex as a function of its input. Such a BNN is incapable of expressing increased uncertainty between regions of low uncertainty. This would not be the case if the output weights had an unrestricted distribution. Although this insight does not immediately apply to BNNs with tanh activations and mean-field input weights, it shows that the mean field assumption can in some cases severely restrict the complexity of uncertainty estimates a BNN can express in function space.

Second, MFVI fails to express increased in-between uncertainty because fitting data in the outer region whilst having increased uncertainty in-between requires strong dependencies in the approximate posterior. This is because in a mean-field distribution, any parameter uncertainty used to express increased in-between uncertainty leads to uncontrolled variations in the fit in the data region. The only way to have a good fit and increased in-between uncertainty is to have variations in one parameter compensated for by variations in others, such that the resulting function still passes through the data points. This is explained in detail in Appendix <ref> via a synthetic example.





UCI Regression Datasets. We now investigate the uncertainty quality of BNNs in real-world datasets. The UCI datasets are usually split into training and test sets by subsampling the dataset uniformly. In the first experiment, we use the standard splits also used in <cit.>. In our second experiment, we create custom splits to test for in-between uncertainty. For each of the D input dimensions of 𝐱_n ∈ℝ^D, we sort the datapoints in increasing order in that dimension. We then remove the middle 1/3 of these datapoints for use as a test set. The outer 2/3 are the training set. 10% of the training set is used as a validation set. Thus a dataset with D inputs has D splits. We refer to these as the `gap splits'. A satisfactory method would achieve good results on the standard splits (showing an ability to fit the data well) and avoid catastrophically poor results on the gap splits (showing increased in-between uncertainty). The results are shown in Figures <ref> and <ref>.[FCVI was only run on one hidden layer due to its long training times, and only tanh was used for Laplace as ReLUs caused problems with linearisation - see Figure <ref>. Full UCI results and experimental details in Appendix <ref>.]

For the standard splits MFVI and LL perform best. FCVI does poorly, likely due to optimisation difficulties. LL does much better than SL, which is surprising given linearisation adds another approximation. However, it appears to redeem the poor Gaussian approximation <cit.>. For the gap splits, MAP is competitive with Bayesian methods on power, protein and wine. However, on energy and naval MAP fails catastrophically, doing dozens or hundreds of nats worse than LL. The test sets of energy and naval thus show very different behaviour from their training sets, and good in-between uncertainty is required to prevent overconfident extrapolations. In this situation we would expect Bayesian methods to outperform MAP. However MAP and MFVI perform similarly poorly on energy and naval, showing that MFVI is overconfident. The only method that performs well on the standard splits and avoids any catastrophic results on the gap splits is LL.



§ CONCLUSIONS


We have shown that MFVI fails to provide calibrated in-between uncertainty, and that the standard UCI splits fail to adequately test for it. However, the decades-old LL approximation performs far better in this regard. Although recent advances in variational inference have allowed BNNs to scale to larger architectures than ever before, in terms of uncertainty quality the mean-field approximation loses crucial expressiveness compared to the less scalable LL approximation. It is therefore key for the field of approximate inference to consider how the approximation of posteriors in parameter space affects the expressiveness of uncertainties in function space. Future work will investigate the conditions an approximate posterior must satisfy to reliably capture in-between uncertainty. It would also be natural to see if combining K-FAC Laplace <cit.> with linearisation leads to improved results.




§ ACKNOWLEDGEMENTS




We thank David R. Burt, Sebastian W. Ober and Ross Clarke for helpful discussions. AYKF gratefully acknowledges the Trinity Hall Research
Studentship and the George and Lilian Schiff Foundation for funding his studies.





icml2019



§ LINEARISED LAPLACE APPROXIMATION


To obtain the linearised Laplace approximation, we linearise the output of the network about θ_MAP:

    f_θ(𝐱) ≈ f_θ_MAP(𝐱) + g(𝐱)^𝖳(θ - θ_MAP).

We now have the following approximating distributions:

    p(θ|𝒟)    ≈𝒩(θ; θ_MAP, Σ),
    
        p(y^*|θ, 𝐱^*)    ≈𝒩(y^*; f_θ_MAP(𝐱) + g(𝐱)^𝖳(θ - θ_MAP), σ_o^2).

Since this is now a linear-Gaussian model, we can use standard formulas to obtain:

    p(y^*|𝐱^*, 𝒟) ≈𝒩(y^*; f_θ_MAP(𝐱^*), σ_o^2 + g(𝐱^*)^𝖳Σ  g(𝐱^*)).






§ CONVEX VARIANCE RESULT


 Consider a single hidden layer BNN with input 𝐱∈ℝ^D and output 𝐲∈ℝ^K with a mean field distribution over the output weights and biases (𝐖, 𝐛) but a point estimate for the input weights and biases (𝐔, 𝐯). In detail:

    y_k (𝐱)    = ∑_i W_kiϕ(a_i) + b_k,
    
        a_i    = ∑_j U_ijx_j + v_i.

We assume a fully factorised approximating distribution for the output weights such that: 

    q(𝐖, 𝐛) = q(𝐛)∏_k,iq_ki(W_ki).

 We further assume that 𝐔 and 𝐯 are deterministic constants. Consider the variance of the output under this distribution:
 
    Var[y_k (𝐱)]    = ∑_i Var[W_ki] ϕ(a_i)^2 + Var[b_k].

 Equation (<ref>) is justified since each weight is independent under q. This variance is a measure of the uncertainty in the output at 𝐱 represented by the approximate posterior q. Consider the Hessian of this variance 𝐇, where H_nm = ∂_x_n∂_x_mVar[y_k(𝐱)]. Taking derivatives, we have:
 
    H_nm   =  ∑_i 2 Var[W_ki] (ϕ(a_i)ϕ”(a_i) + ϕ'(a_i)^2 ) U_inU_im
    𝐇   = ∑_i 2Var[W_ki] (ϕ(a_i)ϕ”(a_i) + ϕ'(a_i)^2 ) 𝐮_i 𝐮^𝖳_i,

 where 𝐮_i is the column vector whose elements are the ith row of 𝐔. Since 𝐇 is a sum of outer products, it will be positive semi-definite (PSD) if ϕ(a_i)ϕ”(a_i) + ϕ'(a_i)^2 ≥ 0 for all i. This is the case for ReLU nonlinearities. The first and second derivatives of the ReLU do not exist at a_i = 0. However if we consider ϕ” to be a bump function of arbitrarily small width and area 1, then all these derivatives exist and ϕ(a_i)ϕ”(a_i) is non-negative. Since the Hessian of Var[y_k(𝐱)] is PSD, it follows that Var[y_k(𝐱)] is a convex function of 𝐱.[This argument can be made rigorous by constructing a sequence of networks with non-linearities ϕ_n such that ϕ_n” is a triangular function at zero with area 1 and width ϵ/n. Each network will have a convex output variance, and the variance of these networks converges pointwise to the variance of a ReLU network. Since a pointwise limit of convex functions is convex, the result holds for ReLU networks.] Therefore it is impossible for this kind of posterior to exhibit greater uncertainty in between two regions of low uncertainty. 
 
 To investigate the relevance of this result to the standard case where the input parameters (𝐔, 𝐯) are not deterministic but are also mean-field, we train three ReLU BNNs on the 1D dataset in Figure <ref>: (i) mean-field VI on all parameters (MVFI), (ii) maximum-likelihood on the input parameters and mean-field on the output parameters (MFVI-output) and (iii) maximum-likelihood on the input parameters followed by Bayesian linear regression on the output parameters (BLR). Results are shown in Figure <ref>. MFVI-output has convex variance, as predicted. MFVI also has convex variance, even though its input parameters are mean-field. BLR with its full-covariance Gaussian posterior shows increased in-between uncertainty even though its input weights are deterministic, showing that it is the mean-field assumption on the output parameters that is responsible for severely restricting the expressiveness of the predictive uncertainty. Further work is required to characterise the expressiveness of mean field distributions with deeper networks.
 


In practice when MFVI is used, the input weights to the hidden layer are not deterministic, but are also mean field. However this result sheds light on the difficulty of obtaining expressive uncertainty estimates with mean field posteriors. It shows that in addition to non-convex variances being difficult to obtain experimentally, they are theoretically impossible to obtain with a mean field distribution over just output weights. Note that for an unconstrained q distribution over the output weights, Var[y_k(𝐱)] is generally non-convex even with deterministic input weights. Therefore the convexity restriction is a direct consequence of the mean-field restriction on q. Further work is required to characterise the expressiveness of mean field distributions when input weights are also mean-field, and with deeper networks.

We plot the output variance of MFVI (on all parameters) on the 1D dataset from Section <ref> in Figure <ref>. For tanh the variance is close to zero everywhere, and for ReLU it is convex even though the input weights are also mean-field. We observed similar behaviour with deeper networks.






§ ANALYSIS OF UNCERTAINTY IN TOY CASE


To gain intuition for why MFVI fails to provide in-between uncertainty, we consider a toy example involving a single hidden layer network with two ReLU hidden units mapping x ∈ℝ→ y ∈ℝ:

    y(x) = W_1 ϕ (U_1 x + v_1) + W_2 ϕ (U_2 x + v_2) + b.

Here W_1, W_2 and b are the output weights and bias, and U_1, U_2 and v_1, v_2 are the input weights and biases. Consider the case where W_1, W_2, U_1, U_2 are all deterministic and positive so that y(x) is non-decreasing. Then:

    y(x)=
        
        b    (I)
    
        W_1U_1x + W_1v_1 + b    (II)
    
        (W_1U_1 + W_2U_2)x + W_1v_1 + W_2v_2 + b    (III)

where x< -v_1/U_1 in region (I), -v_1/U_1≤ x < -v_2/U_2 in region (II) and x≥ -v_2/U_2 in region (III). Consider a simple observed dataset with many points at (x_1, y_1) and (x_2, y_2) where x_2 > x_1 and y_2 > y_1. A reasonable Bayesian posterior predictive would have low uncertainty around x_1 and x_2, but large uncertainty in between. To first fit this dataset with deterministic weights, we could place x_1 in region (I) and x_2 in region (III). Then to fit the y-values we must set

    b    ≈ y_1, 
    
        (W_1U_1 + W_2U_2)x_2    + W_1v_1 + W_2v_2 + b ≈ y_2.




There are many settings of W_1, U_1, W_2, U_2, v_1 and v_2 that satisfy Equation <ref>. Consider choosing one such setting as a point estimate. To obtain a Bayesian method, we would now like to increase our uncertainty in the parameters. In particular, we should have relatively large uncertainty in the position of the `kinks' -v_1/U_1 and -v_2/U_2 since they can take any values between x_1 and x_2 and fit the data equally well.[Here we assume a reasonably broad prior such that the prior probabilities of the kink locations are roughly uniform over the range [x_1, x_2].] This corresponds to having large uncertainty between two regions of low uncertainty (x_1 and x_2), as in Figure <ref>. To express this, we could relax the distribution over, say, v_2 from a delta function to a Gaussian with positive variance. However, injecting randomness in v_2 jeopardises the fit in region (III) since v_2 is involved in Equation <ref>. The only way to express predictive uncertainty between x_1 and x_2 and still fit the data is to have the values of W_1, U_1, W_2, U_2 and v_1 compensate for any change in v_2 such that Equation <ref> still holds. In other words, we need strong dependencies between the parameters to simultaneously fit the data regions and express predictive uncertainty in the in-between region.[The in-between uncertainty seen in Figure 3 in <cit.> is seemingly an exception. However in that case radial basis function non-linearities were used. Since these have only local effects, the argument here does not apply.] 

The mean-field approximation assumes that there are no dependencies. Therefore any parameter randomness used to express increased in-between uncertainty leads to uncontrolled variations of y(x) in the data region. Hence Equation <ref> is not satisfied. There are two possibilities: either the data fit will be poor or the variances will be minimised (leading to a large penalty in the KL term in the ELBO). In practice MFVI finds a solution that prunes out hidden units, allowing it to fit the data with the minimum number of variances set to zero <cit.>.



§ EXTRA RESULTS AND EXPERIMENTAL DETAILS




Synthetic 1D Dataset. We use single hidden layer BNNs with 50 hidden units. We include results for ReLU activations in Figure <ref>. To verify that MFVI's lack of in-between uncertainty is due to approximate inference and not the model class, we include a Gaussian Process (GP) using the kernel for a BNN with infinitely many ReLU hidden units <cit.>. Note LL shows strange discontinuous behaviour in its uncertainty. This is because the non-smooth ReLU function makes the gradient g(𝐱) in equation (<ref>) discontinuous. Similar behaviour is seen in <cit.>.

We use independent 𝒩(0,1) priors on the biases and 𝒩(0,ω^2/H) priors on the weights; H is the number of inputs to the weight matrix. This scaling is chosen so that the GP limit exists <cit.>. ω is set to 4. To optimise Laplace, MFVI and FCVI we use ADAM <cit.> with learning rate 0.001 and 20,000 epochs. We use the entire dataset for each batch. For MFVI, weight means were initialised from 𝒩(0,1/√(4n_outputs)) and all variances were initialised to 10^-5. Bias means were initialised to zero. The local reparameterisation trick <cit.> was used. For FCVI, the Cholesky decomposition of the covariance matrix was parameterised as a lower triangular matrix, with the diagonal entries made positive by exponentiating them. The diagonal entries were initialised to log(0.05) and the off-diagonals were initialised to 0. The mean vector was initialised from 𝒩(0, 0.1). For both MFVI and FCVI we approximate the ELBO during training with 32 samples. For HMC, the number of leapfrog steps was chosen uniformly between 5 and 10, and the step size uniformly sampled from [0.001, 0.0015]. The chain was burned in for 10,000 iterations and samples were collected during the next 20,000 iterations. For MFVI, FCVI and HMC, the error bars in Figures <ref> and <ref> were estimated with 100 samples.

UCI Datasets. All BNNs had 50 neurons per hidden layer. Inputs and outputs were normalised to zero mean and unit variance. Hyperparameters were optimised by grid search on a validation set that consisted of 10% of the training set. The best hyperparameters were used to train again on the training set  with  validation  set  combined. This was repeated for each split. Minibatches were randomly selected from the training set with replacement. For MAP and Laplace, all parameters had independent 𝒩(0,ω^2) priors. For Laplace, minibatch size was 100. The hyperparameters optimised were: ω: [1, 2], learning rate: [0.01, 0.001], number of epochs: [40, 100, 200, 400]. For MAP, the same ranges were searched, except the number of epochs was [20, 40, 100], since we expected MAP to favour early stopping. For both methods, log(σ_o^2) was initialised to -1 and learned by maximum likelihood. For FCVI and MFVI, we used independent 𝒩(0,1) priors on all parameters. The hyperparameters searched for MFVI were: minibatch size: [32, 100], learning rate: [0.01, 0.001], number of epochs: [500, 1000, 2000] for smaller datasets (boston, concrete, energy, wine, yacht) and [50, 100, 200] for larger ones (kin8nm, naval, power, protein). The same ranges were used for FCVI except the learning rate was fixed to 0.001. The ELBO was approximated with 32 samples. For MFVI, weight means were initialised from 𝒩(0,1/√(4n_out)) and all variances initialised to 10^-5. Bias means were initialised to zero. For FCVI, the mean vector and covariance matrix of all the parameters in the network were optimised to maximise the ELBO. The Cholesky decomposition of the covariance matrix was parameterised directly as a lower triangular matrix, with the diagonal entries constrained to be positive by exponentiating them. The diagonal entries were initialised to log(10^-5) and the off-diagonal entries were initialised to 0. The mean vector was initialised randomly from 𝒩(0, 0.1). For MFVI, FCVI and sampled Laplace, test log-likelihoods were computed by sampling 100 times from the approximate posterior. log(σ_o^2) was initialised to -1 and learned by optimising the ELBO. 

Full results are given in Tables <ref> and <ref>. We also provide pairwise comparisons of LL versus MFVI-ReLU on the standard splits and the gap splits in Figures <ref>, <ref> & <ref>. Each point corresponds to one test log-likelihood. Each colour represents a different dataset. The histogram shows the log likelihood of the method on the x-axis minus that of the method on the y-axis. The dotted blue line is the line y=x.


















