Question: Represent the scientific article to find a short TLDR

Question: Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods may have large bias. In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation. The proposed method can generate low-biased samples by increasing the length of MCMC simulation and optimising the MCMC hyper-parameters, which offers attractive balance between approximation bias and computational efficiency. We show that our method produces promising on popular benchmarks when compared to recent hybrid methods of MCMC and VI. Statistical inference methods in machine learning are dominated by two approaches: simulation and optimisation. Markov chain Monte Carlo (MCMC) is a well-known simulation-based method, which promises asymptotically unbiased samples from arbitrary distributions at the cost of expensive Markov simulations. Variational inference (VI) is a well-known method using optimisation, which fits a parametric approximation to the target distribution. VI is biased but offers a computationally efficient generation of approximate samples. There is a recent trend of hybrid methods of MCMC and VI to achieve a better balance between computational efficiency and bias. Hybrid methods often use MCMC or VI as an algorithmic component of the other. In particular, proposed a promising modified VI method that reduces approximation bias by using MCMC transition kernels. Another technique reduces the computational complexity of MCMC by initialising the Markov simulation from a pretrained variational approximation . proposed to improve MCMC using flexible non-linear transformations given by neural networks and gradientbased auto-tuning strategies. In this work, we propose a novel hybrid method, called ergodic inference (EI). EI improves over both MCMC and VI by tuning the hyper-parameters of a flexible finite-step MCMC chain so that its last state sampling distribution converges fast to a target distribution. EI optimises a tractable objective function which only requires to evaluate the logarithm of the unnormalized target density. Furthermore, unlike in traditional MCMC methods, the samples generated by EI from the last state of the MCMC chain are independent and have no correlations. EI offers an appealing option to balance computational complexity vs. bias on popular benchmarks in machine learning. Compared with previous hybrid methods, EI has following advantages: • EI's hyperparameter tuning produces sampling distributions with lower approximation bias. • The bias is guaranteed to decrease as the length of the MCMC chain increases. • By stopping gradient computations, EI has less computational cost than related baselines. We also state some disadvantages of our method: • The initial state distribution in EI's MCMC chain has to have higher entropy than the target. • The computational complexity per simulated sample of EI is in general higher than in VI. Monte Carlo (MC) statistical inference approximates expectations under a given distribution using simulated samples. Given a target distribution π, MC estimations of an expectation E π [f (x)] are defined as empirical average of the evaluation of f on samples from π. To generate samples from π, we assume that the unnormalized density function π * (x) can be easily computed. In a Bayesian setting we typically work with π * (x|y) given by the product of the prior p(x) and the likelihood p(y|x), where y denotes observed variables and x denotes the model parameters specifying p(y|x). Markov chain Monte Carlo (MCMC) casts inference as simulation of ergodic Markov chains that converge to the target π. The MCMC kernel M (x |x) is characterised by the detailed balance (DB) property: π(x)M (x |x) = π(x)M (x|x). Given an unnormalised target density π *, an MCMC kernel can be constructed in three steps: first, sample an auxiliary random variable r from an auxiliary distribution q φ1 with parameters φ 1; second, create a new candidate sample as (x, r) = f φ2 (x t−1, r), where f φ2 is a deterministic function with parameters φ 2; finally, accept the proposal as x t = x with probability p MH = min {0, π * (x)q φ1 (r)/[π * (x t−1)q φ1 (r)]}, otherwise duplicate the previous sample as x t = x t−1. The last step is well known in the literature as the Metropolis-Hastings (M-H) correction step and it in MCMC kernels that satisfy the DB condition. In the following, we denote the joint MCMC parameters (φ 1, φ 2) by φ. If f φ2 does not preserve volume, then it requires a Jacobian correction factor in the ratio in p M H. Hamiltonian Monte Carlo (HMC) is a successful MCMC method which has drawn great attention. A few recent works based on this method are;;. In HMC, the auxiliary distribution q φ1 is often chosen to be Gaussian with zero-mean and a constant diagonal covariance matrix specified by φ 1. The most common f φ2 in HMC is a numeric integrator called the leapfrog algorithm, which simulates Hamiltonian dynamics defined by log π * . The leapfrog integrator requires the gradient of log π * and a step size parameter given by φ 2. Given any initial state x 0, MCMC can generate asymptotically unbiased samples x 1:n. For this, MCMC iteratively simulates the next sample x t through the application of the MCMC transition kernel to the previous sample x t−1. It is well known in the literature that MCMC is computationally demanding . In particular, it is often necessary to run sufficiently long burn-in MCMC simulations to reduce simulation bias. Another drawback of MCMC is sample correlation, which increases the variance of the MC estimator . To avoid strong sample correlation, the common practice in MCMC to tune hyper-parameters manually using sample quality metrics like effective sample size, , which has been developed into automated gradient-based tuning strategies in recent work . Variational inference (VI) is a popular alternative to MCMC for generating approximate samples from π. Unlike MCMC reducing sample bias by long burn-in simulation, VI casts the sample bias reduction as an optimisation problem, where a parametric approximate sampling distribution P is fit to the target π. In particular, VI optimises the evidence lower bound (ELBO) given by where, also known as the entropy, must be tractable to compute. L ELBO (P T π *) is a lower bound on the log normalising constant log Z = log π * (x) dx. This bound is tight when P = π. therefore, the approximation bias in VI can be defined as the gap between L ELBO (P π *) and log Z, that is, where D KL (P π) denotes the Kullback-Leibler (KL) divergence. Variational approximations often belong to simple parametric families like the multivariate Gaussian distribution with diagonal covariance matrix. This in computationally efficient algorithms for bias reduction and sample generation, but may also produce highly biased samples in cases of over-simplified approximation that ignores correlation. Designing variational approximation to achieve low bias under the constraint of tractable entropy and efficient sampling procedures is possible using flexible distributions parameterised by neural networks (NNs) . However, how to design such NNs for VI is still a research challenge. The balance between computational efficiency and bias is a challenge at the heart of all inference methods. MCMC represents a family of simulation-based methods that guarantee low-bias samples at cost of expensive simulations; VI represents a family of optimisation-based methods that generate high-bias samples at a low computational cost. Many recent works seek a better balance between efficiency and bias by combining MCMC and VI. proposed to reduce variational bias by optimising an ELBO specified in terms of the tractable joint density of short MCMC chains. The idea seems initially promising, but the proposed ELBO becomes looser and looser as the chain grows longer. construct an alternative ELBO for HMC that still has problems since the auxiliary momentum variables are sampled only once at the beginning of the chain, which reduces the empirical performance of HMC. Inspired by contrastive divergence, proposed a novel variational objective function to optimise variational parameters by adding additional term that minimise the KL between a MCMC distribution and variational approximation to reduce variational bias. and proposed to replace expensive burn-in simulations in MCMC with samples from pre-trained variational approximations. This approach is effective at finding good initial proposal distributions. However, it does not offer a solution for tuning HMC parameters , which are critical for good empirical performance. Another line of research has focused on improving inference using flexible distributions, which are transformed from simple parametric distributions by non-linear non-volume preserving (NVP) functions. proposed to tune NVP parameterised MCMC w.r.t. a variant of the expected squared jumped distance (ESJD) loss proposed by. proposed a similar auto-tuning for NVP parameterised MCMC using an adversarial loss. Ergodic inference (EI) is motivated by the well-known convergence of MCMC chains : MCMC chains converge in terms of the total variation (TV) distance between the marginal distribution of the MCMC chain and the target π. Inspired by the convergence property of MCMC chains, we define an ergodic approximation P T to π with T MCMC steps as following. Given a parametric distribution P 0 with tractable density p 0 (x 0 ; φ 0) parameterlized by φ 0 and an MCMC kernel M (x |x; φ) constructed using the unnormalised target density π * and with MCMC hyperparameter φ, an ergodic approximation of π is the marginal distribution of the final state of an T -step MCMC chain initialized from P 0: We call φ 0 and φ the ergodic parameters of P T. Well known in MCMC literature like , the ergodic approximation p T converges to π after every MCMC transition and with sufficiently long chain p T is guaranteed to be arbitrarily close to π with arbitrary φ and φ 0. It is important to clarify that ergodic approximation is different from the modified variational methods like which only optimise the variational parameters φ 0, but the optimisation objective functions involve MCMC similation. In the following section, we show how EI can tune the ergodic parameters to minimise the bias of P T as an approximation to the target π with finite T. To reduce the KL divergence D KL (P T π), one could tune the burn-in parameter φ 0 and the MCMC parameter φ by minimizing equation 2. However, this is infeasible because we cannot analytically evaluate p T in equation 3. Instead, we exploit the convergence of ergodic Markov chains and propose to optimise an alternative objective as the following constrained optimisation problem: max where h is a hyperparameter that should be close to the entropy of the target, that is, h ≈ H(π). We call the objective in equation 4 the ergodic modified lower bound (EMLBO), denoted by L(φ 0, φ, π *). Note that the EMLBO is similar to L ELBO (P T π *), with the intractable entropy H(P T) replaced by the tractable L ELBO (P 0 π *). We now give some motivation for this constrained objective. First, we explain the inclusion of the term L ELBO (P 0 π *) in equation 4 and its connection to H(P T). If we maximised only the first term E p [log π * (x)] with respect to a fully flexible distribution P, the would be a point probability mass at the mode of the target π. This degenerate solution is avoided in VI by optimising the sum of E p [log π * (x)] and the entropy term H(P), which enforces P to move away from a point probability mass. However, H(P T) is intractable in ergodic approximation. Fotunately, we notice that maximising the term L ELBO (P 0 π *) = E p0 [log π * (x)] + H(P 0) has similar effect of maximising H(P T) for preventing P 0 from collapsing to the mode of π. It is easy to show that P T cannot be a delta unless H(P T) = −∞, which also implies L ELBO (P T π *) does not exist. Since the KL divergence D KL (P t π) never increases after each MCMC transition step , The constraint in equation 5 is necessary to eliminate the following pathology. If P 0 does not satisfy, we will favor P T to stay close to P 0 instead of making it converge to π faster. This is illustrated by the plot in the right part of Figure 2. To avoid this pathological case, note that It is interesting to compare the EMLBO with the objective function optimised by , that is, the ELBO given by where p(x 0:T −1 |x T) denotes the conditional density of the first T states of the MCMC chain given the last one x T and r(x 0:T −1 |x T) is an auxiliary variational distribution that approximates p(x 0:T −1 |x T). Note that the negative KL term in equation 6 will increase as T increases. This makes the ELBO in equation 6 become looser and looser as the chain length increases. In this case, the optimisation of equation 6 in an MCMC sampler that fits well the biased inverse model r(x 0:T −1 |x T) but whose marginal distribution for x T does not approximate π well. This limits the effectiveness of this method in chains with multiple MCMC transitions. By contrast, the EMLBO does not have this problem and its optimisation will produce a more and more accurate P T as T increases. EI combines the benefits of MCMC and VI and avoids their drawbacks, as shown in Table 1. In particular, the bias in EI is reduced by using longer chains, as in MCMC, and EI generates independent samples, as in VI. Futhermore, EI optimises an objective that directly quantifies the bias of the generated samples, as in VI. Methods for tuning MCMC do not satisfy the latter and optimise instead indirect proxies for mixing speed, e.g. expected squared jumped distance . Importantly, EI can use gradients to tune different MCMC parameters at each step of the chain, as suggested by. This gives EI an extra flexibility which existing MCMC methods do not have. Finally, EI is different from parallel-chain MCMC: while EI generates independent samples, parallel-chain MCMC draws correlated samples from several chains running in parallel. We now show how to maximise the ergodic objective using gradient-based optimisation. The gradient ∂ φ0,φ L(φ 0, φ, π *) is equal to the sum of two gradient terms. The first one ∂ φ0 L ELBO (P 0 π *) is affected by the constraint H(P 0) > h, while the second term is not. If we ignore the constraint, the first gradient term can be estimated by Monte Carlo using the reparameterization trick proposed in (D.P. ;): where f φ0 (·) is a deterministic function that maps the random variable i sampled from a simple distribution, e.g. a factorized standard Gaussian, into the random variable x i 0 sampled from p 0 (·; φ 0). To guarantee that our gradient-based optimiser yields a solution satisfying the constraint, we first initialize φ 0 so that H(P 0) > h and, afterwards, we force the gradient descent optimiser to leave φ 0 unchanged if H(P 0) is to get lower than h during the optimisation process. The Monte Carlo estimation of ∂ φ E p T [log π * (x)] can also be computed using the reparameterization trick. For this, the Metropolis-Hastings (M-H) correction step in the MCMC transitions, as described in Section 2.2, can be reformulated as applying the following transformation to x t−1: where is an indicator function that takes value one if p MH > u and zero otherwise. In Hamiltonian Monte Carlo (HMC), f φ is the leapfrog integrator of Hamiltonian dynamics with the leapfrog step size φ. We define the T -times composition of g φ, given in equation 8, as the transformation x T = g T φ (x 0, r 1:T ; u 1:T). Then, the second gradient term can be estimated by Monte Carlo as follows: T t=1 q(r t)Unif(u t ; 0, 1). Note that the gradient term equation 9 is correct under the assumption f φ is volume-preserving in the joint space of (x t−1, r t), otherwise additional gradient term of the Jacobian of f φ w.r.t. φ is required. However, it is not a concern for many popular MCMC kernels. For example, the leapfrog integrator in HMC f φ guarantees the preservation of volume as shown in . It is worth to mention that the indicator function in equation 8 is not continuous but differentiable almost everywhere. Therefore, the gradient in equation 9 can be computed conveniently using standard autodifferentiation tools. The gradient in equation 9 requires computing ∂ φ g T φ (x 0, r 1:T ; u 1:T), which can be done easily by using auto-differentiation and gradient backpropagation through the transfromations g φ (·, r t ; u t) with t = T,..., 1. However, backpropagation in deep compositions can be computationally demanding. We discovered a trick to accelerate the gradient computation by stopping the backpropagation of the gradient at the input x t−1 of g φ (x t−1, r t ; u t), for t = 1,..., T. Empirically this trick has almost no impact on the convergence speed of the ergodic approximation, as shown in Figure 2. 3.3 THE ENTROPY CONSTRAINT AND HYPERPARAMETER TUNING As mentioned previously, ignoring the constraint H(P 0) > H(π) may lead to pathological when optimising the ergodic objective. To illustrate this, we consider fitting an ergodic approximation given by a Hamilton Monte Carlo (HMC) transition kernel with T = 9. P 9 denotes the initial ergodic approximation before traing and P * 9 denotes the same approximation after training. The target distribution is a correlated bivariate Gaussian given by π = N (0, (2.0, 1.5; 1.5; 1.6)). Samples from this distribution are shown in plot (a) in Figure 1. We optimise different a separate HMC parameter φ t, as described in Section 2.2, for each HMC step t. We consider two initial distributions. The first one is P 0 = N (0, 3I) which satisfies the assumption H(P 0) > H(π). The second one is P 0 = N (0, I) with the entropy H(P 0) < H(π), which violates the assumption. In this latter case, we perform the unconstrained optimisation of equation 4. Plots (b) and (c) in Figure 1 show samples from P 9 and P * 9 for the valid P 0. In this first example, maximising the ergodic objective under equation 5 significantly accelerates the chain convergence as further shown by the left plot in Figure 2. Plots (d) and (e) in Figure 1 show samples from P 9 and P * 9 for the invalid initial distribution P 0. In the second example, E p0 [log π * (x)] is higher than E π [log π * (x)] and, consequently, maximising the unconstrained ergodic objective actually deteriorates the quality of the ing approximation. This is further illustrated by the right plot in Figure 2 which shows how the convergence of E pt [log π * (x)] to E π [log π * (x)] is significantly slowed down by the optimisation under the invalid P 0. Fortunately, it is straightforward to prevent this type of failure cases by appropriately tuning the scalar hyperparameter h in equation 5. A value of h that is too low may in higher bias of P T after optimisation as illustrated by the convergence of E pt [log π * (x)] in the blue and orange curves in Plot (b) in Figure 2. Furthermore, in many cases, estimating an upper bound on H(π) is feasible. For example, in Bayesian inference, the entropy of the prior distribution p(x) is often higher than the entropy of the posterior p(x|y). Therefore, the prior entropy can be used as a reference for tuning h. Figure 1: Histograms of samples from ergodic inference using HMC transition kernels. P 9 denotes the ergodic approximation before traing; P * 9 denotes the ergodic approximation after training. ] as a function of the length of the chain T using 10000 samples: Left: with the valid P 0 as H(P 0) > H(π); Right: with invalid P 0 as H(P 0) < H(π). SG training means the stop gradient is applied to the x from previous HMC step in equation 9. We first describe the general configuration of the ergodic inference method used in our experiments. Our ergodic approximation is constructed using HMC, one of the most successful MCMC methods in machine learning literature. We use T HMC transitions, each one involving 5 steps of the vanilla leapfrog integrator which was implemented following. The leapfrog pseudocode can be found in the appendix. In each HMC transition, the auxiliary variables are sampled from a zero-mean Gaussian distribution with diagonal covariance matrix. We tune the following HMC parameters: the variance of the auxiliary variables and the leapfrog step size, as mentioned in Section 2.2. We use and optimise a different value of the HMC parameters for each of the T HMC transitions considered. We call our ergodic inference method Hamiltonian ergodic inference (HEI). The burn-in model P 0 is factorized Gaussian. The initial entropy of P 0 is chosen to be the same as the entropy of the prior. The stocastic optimisation algorithm is Adam with TensorFlow implemtation and the optimiser hyperparameter setting is (β 1 = 0.9, β 2 = 0.999, = 10 −8). The initial HMC leapfrog step sizes are sampled uniformly between 0.01 and 0.025. Additional experiment on Bayesian neural networks is included in Appendix 6.3. We first compare Hamiltonian ergodic inference (HEI) with previous related methods on 6 synthetic bivariate benchmark distributions. Histograms of ground truth samples from each target distribution using rejection sampling are shown in Figure 3. The baselines considered include: 1) Hamiltonian variational inference (HVI) ; 2) generalized Hamiltonian Monte Carlo (GHMC) using an NVP parameterized HMC kernel and gradient-based auto-tuning of MCMC parameters w.r.t. sample correlation loss ; 3) Hamiltonian annealed importance sampling (HAIS) . It is worth to mention that we do not consider other hybrid inference methods like in our experiment, because these methods only combines MCMC simulation with VI but not optimise the parameters of MCMC kernel using the gradient-based approach like EI. HVI is the most similar method to HEI among all three baselines, because both HEI and HVI methods generate samples from the last state of MCMC chains and use gradient-based MCMC hyperparameter tuning to reduce bias. For a fair comparison between HVI and HEI, we consider the HMC chains with exactly the same setting in both methods: the initial state follows a standard Gaussian distribution and the length of HMC chain is T = 10. The key difference between HVI and HEI is the hyperparameter tuning objective, as mentioned in Section 3.1. We trained HVI for 1000 iterations and verified the ELBO converges to a (local) minimum (plots of the training ELBO values are included in Appendix 6.2). We trained HEI for 50 iterations. Following the setting of HAIS by, we used 1,000 intermediate distributions with 5 leapfrog steps per HMC transition and manually tuned the HMC parameters to have acceptance rate around 70%. GHMC 1 was run using 100 parallel chains with 5 leapfrog steps per GHMC transition, 100 burn-in steps and 1000 auto-tuned training iterations. The verification of the convergence of E p T [log π * (x)] to E π [log π * (x)] for HEI is shown in plot (a) of Figure 5. We generate 100,000 samples with each method and evaluate sample quality using two metrics: 1) the histogram of simulated samples for visual inspection; 2) the MC estimation of E π [log π * (x)]. Effective sample size (ESS) is a popular sample correlation based evaluation metric in recent MCMC literature . However, we do not consider ESS in this experiment, because GHMC is the only method among all methods generating correlated samples. Therefore, the ESS of GHMC is guaranteed to be lower than HVI and HEI. To generate ground truth samples from benchmark distributions, we use. The ing sample histograms of the ground truth using rejection sampling are shown in figures 3 and considered approximated sampling methods are shown in 4. Table 2 shows the ing estimates of −E π [log π * (x)] together with the wall-clock simulation time for generating 100,000 samples. The left part of Table 3 shows the training time of the MCMC parameter optimisation for all methods except HAIS, which does not support gradient-based HMC hyperparameter tuning. HEI is faster than HVI and GHMC. Note, however, that the acceleration of HEI over HSVI is due to the stopping gradient trick described in Section 3.2. The histograms and the estimates of −E π [log π * (x)] generated by HEI are consistent with the of the more expensive unbiased samplers GHMC and HAIS, which are close to the ground truth. By contrast, HVI exhibits a clear bias in all benchmarks. Regarding the sampling time, HVI and HEI simulate HMC chains with the same length and, consequently, perform similarly in this case while sample simulation from HAIS and GHMC is much more expensive. 6.00 3000 -83.57 50 HVI(T =1, 16LF, n h =800, ConvNet encoder) 6.00 360 -83.68 48 HVAE(T =1, 16LF, n h =500, ConvNet encoder) 6.00 360 -84.22 48 HEI(T =30, 5LF, n h =500, no neural net encoder) 1.65 54 -83.17 48 HEI(T =30, 5LF, n h =500, no neural net encoder) 3.00 100 -82.76 46 HEI(T =30, 5LF, n h =500, no neural net encoder) 6.00 200 -82.65 45 HEI(T =30, 5LF, n h =500, no neural net encoder) 12.00 400 -81.43 38 HEI(T =15, 5LF, n h =500, no neural net encoder) 8.00 540 -83.30 48 Table 4: Comparisons in terms of compuational efficiency and test log-likelihood in the training of deep generative models on the MNIST dataset. We implemented the deconvolutional decoder network in to test HVI. , the test likelihood is estimated using importence-weighted samples from the encoder network. In our experiment, we use Hamiltonian annealled importance sampling and report the effective sample size (ESS). We now evaluate HEI in the task of training deep generative models. MNIST is a standard benchmark problem in this case with 60,000 grey level 28 × 28 images of handwritten digits. For fair comparison with previous works, we use the 10,000 prebinarised MNIST test images 2 used by. The architecture of the generative model considered follows the deconvolutional network from. In particular, the unnormalised target p θ (x, y) consists of 32 dimensional latent variables x with Gaussian prior p(x) = N (0, I) and a deconvolutional network p θ (y|x) from top to bottom including a single fully-connected layer with 500 RELU hidden units, then three deconvolutional layers with 5 × 5 filters, feature maps, RELU activations and a logistic output layer. We consider a baseline given by a standard VAE with a factorised Gaussian approximate 4 Table 3: Left. The training time of MCMC parameter optimisation in seconds for 100 iterations for all candidate methods to produce the in Figure 4. The training time of HEI is lower than HVI because of the stop gradient trick mentioned in Section 3.2. We do not report the training time for HAIS, because HAIS requires manual tuning of MCMC hyperparameters which is not directly comparable to the gradient-based autotuning used by the other methods. Right. The training time in seconds per epoch for the experiments with deep generative models (DGM). posterior generated by an encoder network q(x|y) which mirrors the architecture of the decoder . The code for is not publicly available. Nevertheless, we reimplemented their convolutional VAE and were able to reproduce the marginal likelihood reported by , as shown in Table 4. This verifies that our implementation of the generation network is correct. We implemented HVI in using an auxiliary reverse model in the ELBO parameterized by a single hidden layer network with 640 hidden units and RELU activations. We also implemented the Hamiltonian variational encoder (HVAE) method , which is similar to HVI but without the reverse model. Unlike in the original HVAE, our implementation does not use tempering but still produces similar to those from. For the HEI encoder, we use T = 30 HMC steps, each with 5 leapfrog steps. The initial approximation P 0 is kept fixed to be the prior p(x). We optimise the decoder and the HEI encoder jointly using Adam. Table 4 shows the marginal test log-likelihood for HEI and the other methods, as estimated with 1,000 HAIS samples . , we also include the effective sample size (ESS) of HAIS samples for the purpose of verifying the reliability of the reported test log-likelihoods. Overall, HEI outperforms HVI, HVAE and the standard VAE in test log-likelihood when the training time of all methods is fixed to be 6 hours. HEI still produces significant gains when the training time is extended to 12 hours and, with only 1.6 hours of training, HEI can already outperform the convolutional VAE of with 6 hours of training. To verify the convergence of HEI, we show in plot (b) of Figure 5 estimates of.., 10 on five randomly chosen test images, where the ground truth E π [log π * (x)] is estimated by HAIS, after HMC hyper-parameter tuning in HEI (blue) and without hyper-parameter tuning in HEI (green), i.e. just using the initial hyper-parameter values. Plot (c) in Figure 5 shows similar , but using the maximum mean discrepancy (MMD) score to quantify the similarity of samples from p T to samples from π, where the latter ground truth samples are generated by HAIS. These plots suggests that shortening the HEI chain to T = 10 HMC steps will have a negligible effect on final simulation accuracy. Finally, the right part of Table 3 shows the training time of HEI with and without the stopping gradient trick. These resuls show that the former method is up to 5 times faster.: a: the targets are 2D benchmarks with the ground truth of E π [log π * (x)]; b: the target π is the VAE posterior p(x|y) each curve represents one random chosen test MNIST image y with the ground truth of E π [log π * (x)] estimated by HAIS using 100 samples; c: MMD score between HEI samples and HAIS samples. We have proposed Ergodic Inference (EI), a novel hybrid inference method that bridges MCMC and VI. EI a) reduces the approximation bias by increasing the number of MCMC steps, b) generates independent samples and c) tunes MCMC hyperparameters by optimising an objective function that directly quantifies the bias of the ing samples. The effectiveness of EI was verified on synthetic examples and on popular benchmarks for deep generative models. We have shown that we can generate samples much closer to a gold standard sampling method than similar hybrid inference methods and at a low computational cost. However, one disadvantage of EI is that it requires the entropy of the first MCMC step to be larger than the entropy of the target distribution. Here is the code for the vanilla leapfrog algorithm we used in HVI, HEI and HAIS. Algorithm 1: Leapfrog Input: x: state, r: momenta, φ 1: r variance, φ 2: step size, m: number of steps Result: x: new state, r: new momentum x = x; r = r; for t ← 1 to m dō r =r − 0.5φ 2 ∂ x U (x); x =x + φ 2 /φ 1r; r =r − 0.5φ 2 ∂ x U (x); end x =x; r =r; return x and r; The plots in Figure 6 show training loss (negative ELBO) of HVI and the training expected log likelihood E p T [log π * (x)] with T = 10 HMC steps with Adam with hyperparameter setting described in Section 4. It is clear that HVI is well trained but the approximation is biased, because E p T [log π * (x)] does not converge to the true loss (the red line on the right plots). In comparison, in Figure 6 (Left) in our paper, E p T [log π * (x)] of HEI converges to the ground true by optimising our ergodic loss. In this additional experiment we approximate the posterior distribution of Bayesian neural networks with standard Gaussian priors. We consider four UCI datasets and compare HEI with the stochastic gradient Hamilton Monte Carlo (SGHMC) method from. The networks used in this experiment have 50 hidden layers and 1 real valued output unit, as stated in. The HEI chain contains 50 HMC transformation with 3 Leapfrog steps each. The initial proposal distribution P 0 is a factorised Gaussian distribution with mean values obtained by running standard mean-field VI using Adam for 200 iterations. We do not use in P 0 the variance values returned by VI because these are unlikely to in higher entropy than the exact posterior since VI tends to understimate uncertainty. Instead, we choose the marginal variances to be n −0.5 where n is the number of inputs to the neural network layer for the weight. To reduce computational cost, we use in this case stochastic gradients in the leapfrog integrator. For this, we split the training data into 19 mini-batches and only use one random sampled mini-batch for computing the gradient in each leapfrog iteration. We train our HEI for 10 epochs and the stationary distribution is chosen as approximate posterior on a random sampled mini-batch. The ing test log-likelihoods are shown in Table 5. Overall, HEI produce significantly better than SGHMC. We also show in the right plot of Figure 7 estimates of E pt [log p(x, y)] for t = 1,..., 50 after HMC hyper-parameter tuning and without hyper-parameter tuning.

Represent the short summary for finding the corresponding scientific article (long)

In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: represent the scientific article to find a short tldr

Question: We show that information about whether a neural network's output will be correct or incorrect is present in the outputs of the network's intermediate layers. To demonstrate this effect, we train a new "meta" network to predict from either the final output of the underlying "base" network or the output of one of the base network's intermediate layers whether the base network will be correct or incorrect for a particular input. We find that, over a wide range of tasks and base networks, the meta network can achieve accuracies ranging from 65% - 85% in making this determination. What do neural networks know and where do they know it? At what stage of a network's processing does a "decision" get made and are there reliable markers of a correct or incorrect decision either in the output or during a network's operation at one of its intermediate layers? To begin this investigation, we ask where in a neural network's operation it becomes possible to determine whether the network might be correct or incorrect in its output for a particular input. We feed a second, "meta" network the outputs of either an intermediate or final layer of the first, "base", network and train the meta network to predict whether the base network will be correct for an individual input. We call the second network a meta or metacognitive network because humans and other animals are known to make so-called metacognitive judgments to assess their confidence in the correctness of their beliefs or actions BID4.We find that the meta network is able to predict whether a base network will be correct or incorrect on previously unseen inputs with up to 69% accuracy for base networks 1 Computer Science Department, Columbia University, New York, New York, USA 2 Mechanical Engineering Department and Data Science Institute, Columbia University, New York, New York, USA. Correspondence to: Chad DeChant <chad.dechant@columbia.edu>.Identifying and Understanding Deep Learning Phenomena Workshop at the International Conference on Machine Learning 2019 FIG0. Meta network pipeline: the Meta network receives as input the output of one of the base network's layers for a particular input and predicts whether the base network will be correct.classifying ImageNet images and 85% accuracy for a base network classifying CIFAR 10 images. As these two examples suggest, the accuracy of the meta network is higher for simpler underlying tasks in our experiments. The usefulness of the layers' outputs for predicting the accuracy of the network is lowest at the earliest layers in the network and increases to be highest either at the last hidden layer or, in most cases, the final output. Meta networks trained on different layers' outputs have significant but not complete overlap in which examples they are able to correctly predict will go on to be accurately or inaccurately classified, suggesting that there is slightly different information at each level which can be used to make assessments of accuracy. Our approach has two main stages. First, we run example images or text passages through a pretrained base network and save the final and intermediate outputs of that base network. We use PyTorch and save intermediate layer outputs using its hook feature BID5. For each example for which we save the output of intermediate stages, the We therefore train the meta network using examples from the training set of the base network. Given the relatively high accuracy of the base models in our experiments, it would easy for the meta classifier to "cheat" by predicting that the base network is always correct. To prevent this, we balance the classes at training time and choose our models based on the best and most balanced accuracy on the validation set. During training we define this combination of highest accuracy and balance to be the geometric mean of the meta network's accuracy on "base correct" (C) and "base incorrect" (I) classes minus the absolute value of their difference: DISPLAYFORM0 All numbers reported here are from a held out test set of inputs previously unseen by both the base and meta networks. To determine how general and widely occurring is the phenomenon we are investigating, we train and test meta networks on a variety of tasks and base networks. Most of our testing is done on base networks which are trained for image classification tasks; we use six networks available in the PyTorch library. To assess the accuracy of networks trained on ImageNet BID7, we use AlexNet, Resnet 18, VGG 16, DenseNet 161, and ResNet 152 networks. For these networks we save and use for training the network's final outputs, the output of the last hidden layer (referred to as "last" in the tables"), and in some cases the output of the penultimate hidden layer ("penultimate" in the tables).For CIFAR 100 BID3 we use and train a VGG 16 network; for CIFAR 10 we train and use a VGG 19 network. For these models we train meta networks on the final output and the output of the last hidden layer, the penultimate hidden layer, the last convolutional layer, a middle convolutional layer (the fifth in VGG16, the eighth in VGG19), and the first convolutional layer. Our base networks for CIFAR 100 and CIFAR 10 had an accuracy of 71.5% and 91.1% on their respective test sets. To test whether intermediate layers can be predictive of accuracy on a non-vision task, we use a Bi-Directional Attention Flow (BiDAF) model BID8 pretrained on the Stanford Question Answering Dataset (SQuAD) version 1.1 BID6. The SQuAD task gives a base network a context passage and a question, and requires the network to output where in the passage the answer to the question starts and ends. We run each example passage and question pair in the SQuAD 1.1 dataset through a pretrained model available in the AllenNLP library BID1 ). This base model has an exact match accuracy (where both the start and end locations of the answer predicted by the model exactly match the ground truth) of 68.03%. Further details of the BiDAF model can be found in FIG2 in the Appendix. 3.1. Images: ImageNet, CIFAR 100, and CIFAR 10Accuracy numbers for the meta networks trained on various models classifying ImageNet images are found in TAB0.For any particular layer, the meta networks display balanced accuracies when the base network as correct and incorrect, ranging from 63% to 70%.Results for meta networks for a VGG16 network trained on CIFAR 100 can be found in TAB1; for meta networks for a VGG19 model trained on CIFAR 10 are in Table 3.There is a clear pattern: when a meta network is trained on the outputs of the first and middle convolutional layers, its accuracy is at best only somewhat better than chance (for CIFAR 100) and at worst no better than chance on average and very unbalanced (for CIFAR 10). Trained on the final outputs, a meta network reaches 77% accuracy averaged between the two classes for the CIFAR 100 base network and 85% for the CIFAR 10 network. Between these two extremes is a gradual increase in accuracy as meta networks are trained on later stages of the base network. Accuracy numbers for the meta networks trained on intermediate and final outputs of a BiDAF model for the SQuAD 1.1 dataset are found in TAB2. The output layer is the concatenation of the output prediction of the start and end locations of the answer. The pattern seen in meta network accuracies for networks trained on vision tasks is not evident here: the highest accuracy is not reached when the meta network is trained on the final outputs. Instead, the best meta network accuracy is found when classifying the output of the Modeling layer composed of Long Short Term Memory (LSTM) units just before the final output layer of the BiDAF model. We have seen that meta networks trained on the last few layers of network achieve similar accuracies. A natural question to ask about the use of many layers for a meta network, then, is whether the meta networks are getting nearly all of the same examples right even when looking at different layers' outputs. We found that while there was considerable overlap, a significant percentage (approximately 20%, depending on which layers we compare) of examples were correctly classified by a meta network looking at one layer of a VGG16 network, but not by a different meta network looking at another layer. In other words, it was not the case that the meta networks' verdicts for each example were the same no matter which layer was considered, suggesting that there might be different information about the accuracy of the base network present at different layers. TAB3 shows the overlaps of a meta network's verdicts for the outputs of the VGG16 network trained on CIFAR 100.A similar was evident in the meta classification when trained on the BiDAF model for the SQuAD dataset. The overlap between correct meta network predictions of accuracy was 73.2% on examples for which the base network was correct and 75.7% for those examples which were originally incorrect coming out of the base network. It is clear that the meta networks are able to learn something about the intermediate and final outputs which are indicative of the networks' accuracy. Just what that is and whether it can be useful in improving or interpreting the networks is as yet unclear. It is difficult to estimate the accuracy of a neural network at runtime. On tasks that involve a choice between discrete options, the value of the highest output after it is put through a softmax is often considered to represent the network's confidence or estimate of the probability of the corresponding class's being correct. However, it is not clear that this interpretation is warranted. Recent work has shown that these outputs are not reliable BID2. It is interesting, then, to consider whether when a meta network is trained on the final outputs it learns to simply classify those outputs in which the predicted class has very high values as correct and those with relatively low values as incorrect. This would correspond to the general intuition that high values for predicted classes indicate meaningfully high confidence. Figure 2 graphically illustrates the outputs of a ResNet18 network trained on ImageNet, with sample outputs of the highest confidence class arrayed along the x axis (a similar chart for outputs of the BiDAF model is found in the Appendix). It shows that while there is certainly a correlation between a base network's accuracy and the value of the output corresponding to the highest predicted class, it is not a simple or completely reliable one. On average, the base network indeed tends to be more confident in its correct answers than its wrong answers, and the set of examples the meta network is correct on shows this pattern clearly while the examples the meta network gets wrong show less distinct base "confidence" numbers. However, it is apparent that the base network is often very "confident" of a wrong answer and not confident of a correct answer. From inspecting the plots it is clear that the meta network is not judging the net- FIG1. Examples of maximum values (arrayed along the x axis) output by a Resnet18 network on ImageNet after the softmax function. The meta network is correct in both cases in the top row and incorrect in the bottom row; the Resnet base classifier is correct on the left and incorrect on the right in both rows. The mean value in each category is given. This shows that the meta network does not learn to simply classify the output based on the value of the class prediction, which is often interpreted as the network's'confidence'.work's output simply by learning a threshold "confidence" level above which it predicts it will be correct and below which it predicts it will be incorrect. This is evident by the large number of incorrect high "confidence" outputs of the base network which the meta network accurately marks as incorrect, as well as the correct low "confidence" outputs which the meta networks finds correct. Further study will be required to better understand what features the meta network has learned to look for to measure accuracy. Neural networks designed for a classification-type task are generally trained to give an answer, not to also indicate whether they are likely to be right or wrong. While there has has certainly been work to address this, notably that involving Bayesian networks BID0, the present work and its future extensions may point in other fruitful directions for characterizing a network's likely accuracy at runtime. There may also be interesting connections to work studying neural networks from an information theoretic perspective BID9. We train meta networks to judge whether a base network is correct or incorrect on particular inputs by feeding the meta network outputs, final or intermediate, from the base network. The blue arrows show which outputs of the base Bi-Directional Attention Flow model the meta network examines when classifying the base network's output as accurate or inaccurate. Image adapted from BID8

Represent the short summary for finding the corresponding scientific article (long)!

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: We develop a new algorithm for imitation learning from a single expert demonstration. In contrast to many previous one-shot imitation learning approaches, our algorithm does not assume access to more than one expert demonstration during the training phase. Instead, we leverage an exploration policy to acquire unsupervised trajectories, which are then used to train both an encoder and a context-aware imitation policy. The optimization procedures for the encoder, imitation learner, and exploration policy are all tightly linked. This linking creates a feedback loop wherein the exploration policy collects new demonstrations that challenge the imitation learner, while the encoder attempts to help the imitation policy to the best of its abilities. We evaluate our algorithm on 6 MujoCo robotics tasks.

Unsupervised self-imitation algorithm capable of inference from a single expert demonstration.

We propose Tendency RL to efficiently solve goal-oriented tasks with large state space using automated curriculum learning and discriminative shaping reward, which has the potential to tackle robot manipulation tasks with perception.


Question: Represent the scientific article to find a short TLDR

Question: Significant work has been dedicated to developing methods for communicating reasons for decision-making within au- tomated scheduling systems to human users. However, much less focus has been placed on communicating reasons for why scheduling systems are unable to arrive at a feasible solution when over-constrained. We investigate this problem in the context of task scheduling. We introduce the agent resource-constrained project scheduling problem (ARCPSP), an ex- tension of the resource-constrained project scheduling problem which includes a conception of agents that execute tasks in parallel. We outline a generic framework, based on efficiently enumerating minimal unsatisfiable sets (MUS) and maximal satisfiable sets (MSS), to produce small descriptions of the source of infeasibility. These descriptions are supple- mented with potential relaxations that would fix the infeasibility found within the problem instance. We illustrate how this method may be applied to the ARCPSP and demonstrate how to generate different types of explanations for an over- constrained instance of the ARCPSP. In many real-world applications, human users in charge of developing plans and making decisions are aided by automated planning and scheduling systems. For example, NASA mission planning makes use of a large team of human planners that use various automated scheduling systems in order to construct day-to-day as well as long-term plans for crew members. A primary function of these automated systems is generating different types of plans and schedules while ensuring that various constraints do not conflict. When plans are ultimately constructed by human planners for a human crew, it is essential for both the planners, and the crew executing the plans, to understand how and why certain scheduling decisions were made by automated tools. In general, when the primary function of such constraint satisfaction and optimization tools is to support human decision-making, it is necessary for the automated systems to be transparent in how they arrive at certain outputs. Significant work has been dedicated to generating humanunderstandable explanations for why certain automated planning decisions were made BID10 ).However, little work has been done in generating reasons for why plans or schedules cannot be generated under certain specifications. Human users interacting with such constraint satisfaction or optimization tools are bound to run into configurations for which no feasible solution exists. Fixing infeasible configurations is a challenging task for the human user if they are unable to understand why the solver arrives at an unsatisfiable . While various partial constraint satisfaction tools exist for solving such over-constrained problems BID4, solutions employing these tools have significant limitations that make them less applicable in certain real-life scenarios. Most of these methods employ constraint hierarchies to determine which constraints should be violated in order to satisfy more important ones. However, in complicated planning or scheduling applications involving multiple human agents, constructing such a hierarchy is often impractical. Instead, if reasons for infeasibility can be properly conveyed back to the human user, they can make high-level decisions to solve infeasibility in any way they see fit. In this paper, we provide a framework for iteratively generating human-understandable explanations of infeasibility for a specific class of scheduling problems. These explanations manifest themselves as minimal sets of specifications (or constraints) that are responsible for causing infeasibility, coupled with suggestions for relaxations through which feasibility could be achieved. The method proposed in this paper allows users to enumerate over a series of explanations for infeasible instances of problems at varying levels of abstraction. For example, raw explanations of relevant low-level constraints may be directly output or a causal link may be established back to higher level descriptions of the problem to understand what specifications were responsible for the feasibility issue. This system also allows directed questions about feasibility to be asked, such as "why can task A not be scheduled after task B?"A strategy for iteratively generating minimal unsatisfiable sets (MUS) and maximal satisfiable sets (MSS) forms the basis for interpreting the infeasibility of the problem. Existing methods such as QuickXplain BID5 ) focus on generating a single most preferable explanation of infeasibility. Likewise, BID1 aims to generate a single explanation in the context of optimization without attempting to achieve minimality. However, overconstrained problems may contain several infeasibility issues which cannot be solved by changing only a single part of the problem. So, because a single MUS only provides indication of a single feasibility issue, we aim to enumerate several sets of MUS to highlight multiple feasibility issues found within the problem instance. Therefore, the proposed enumeration strategy is based on MARCO BID8 ), a flexible algorithm for generating MUSes and MSSes in succession. Motivated by the domain of space mission scheduling, we introduce and investigate the agent resource-constrained project scheduling problem (ARCPSP), an extension of the resource-constrained project scheduling problem (RCPSP) that incorporates the delegation of tasks to differing agents. This problem cannot be framed as an instance of the RCPSP because it deals with the case of asymmetric agents in which certain tasks may only be executed by a subset of the agents. This problem is meant to model applications in which efficient scheduling for teams of differing agents is critical. While we only explicitly investigate this problem, the generality of the approach outlined in this paper would allow the methodology to be adapted for different types of constraint satisfaction and optimization tools as well as different types of planning and scheduling problems. The main contributions of this paper are the following: firstly, we provide a formal definition of the agent resourceconstrained project scheduling problem (ARCPSP) in Section 3. Then in Section 4 we outline a difference logic encoding of the ARCPSP which is used to check feasibility of problem instances. The framework for generating humanunderstandable explanations of infeasibility for instances of the ARCPSP is described in Section 5. Finally, we provide an overview of the trade-off between interpretability and expressibility of different types of explanations and conclude by discussing how these ideas can be extended. In this section, we introduce relevant information and definitions used throughout the paper. These concepts will set the stage for formulating the ARCPSP in terms of satisfiability modulo theory and using minimal unsatisfiable sets and maximal satisfiable sets to generate explanations of infeasibility. Let X be a set of variables and clauses C 1,..., C n be formulas representing constraints over X. Consider a formula of the form DISPLAYFORM0 We say the formula ϕ is satisfiable if there exists some assignment to the variables in X which makes ϕ evaluate to TRUE. Otherwise, it is unsatisfiable. Note that if ϕ takes the form of equation FORMULA0, as it does throughout this paper, every clause C i must be TRUE in order for ϕ to evaluate to TRUE. To implement the temporal constraints within a schedule, the clauses C i are taken from the theory of difference logic (DL), which makes deciding ϕ a satisfiability modulo theory (SMT) problem. To check satisfiability of problem instances, we use the Microsoft Z3 SMT solver (De Moura and Bjørner 2008). As will be discussed in Section 4, the agent resourceconstrained project scheduling problem (ARCPSP) can be encoded in difference logic (DL), a fragment of linear real arithmetic (LRA). The numerical components of DL are solvable in polynomial time BID2 using graph-based procedures based on an incremental BellmanFord algorithm. In general, decidability for DL using these methods is more efficient than the simplex-based methods used to decide LRA. Under DL, atoms are restricted to the form DISPLAYFORM0 However, we can rewrite the following atoms in difference form: DISPLAYFORM1 Bounds x ≤ k can also be incorporated by writing them as x − x 0 ≤ k where x 0 is a special variable that is later set to zero. Definition 1. A minimal unsatisfiable set (MUS) of a set C of constraints is a subset M ⊆ C such that M is unsatisfiable and every proper subset M ⊂ M is satisfiable. A maximal satisfiable set (MSS) of a set C of constraints is a subset M ⊆ C such that M is satisfiable and every proper superset M, with C ⊇ M ⊃ M, is unsatisfiable. A minimal correction set (MCS) of a set C of constraints is the complement of some maximal satisfiable set of C, and can be understood as a minimal set of constraints which need to be removed from C in order to make it satisfiable. It is important to note that MUSes, MSSes, and MCSes are only locally maximal (or minimal), and are different from concepts of globally optimal subsets. MUSes can be understood as isolated, infeasible subsets of the constraints. Their primary characteristic is that removing any single constraint would make the set satisfiable. However, this does not necessarily guarantee the feasibility of the entire set of constraints because there might be many disjoint MUSes within the set. In order to make the entire set feasible (satisfiable), a hitting set of the MUSes must be removed. Every MCS is precisely one combination of such a hitting set. Definition 2. A of a set C of constraints is a subset B ⊆ C of hard constraints, which must be necessarily satisfied. In the context of scheduling problems, s typically include constraints that ensure that the outcome of the schedule is logical, including conditions such as tasks not overlapping and resource constraints not being exceeded. We denote everything outside of the M \ B as the foreground. Hence, the and foreground partition the set C of constraints. A minimal conflict of an over-constrained set C of constraints with respect to a B is then a subset of the foreground M ⊂ C \ B such that M ∪ B is unsatisfiable and, for any superset M ⊃ M, M ∪B is satisfiable. A minimal relaxation of an over-constrained set C of constraints with respect to a B is a subset of the foreground M ⊂ C \ B such that (C \ M) ∪ B is satisfiable and, for any superset M ⊃ M, (C \ M) ∪ B is unsatisfiable. Then an explanation is a sequence of minimal conflicts and minimal relaxations for a problem instance. The definitions of minimal conflicts and minimal relaxations mirror the concepts of MUSes and MCSes, respectively, while incorporating a of constraints which cannot be modified. A is necessary for specifying hard constraints which cannot be relaxed or modified. This way we can prevent certain constraint from consideration for conflicts or relaxations. A also allows the generation of explanations concerning different aspects of a scheduling problem instance, a concept which will be explored later in the paper. The problem that we formulate is an extension of the resource-constrained project scheduling problem (RCPSP). Loosely, the RCPSP considers nonpreemptive, precedenceconstrained tasks of known durations that are constrained by reusable resource requirements (i.e. resources that are returned after a task stops using them). The agent resourceconstrained project scheduling problem extends the RCPSP to include a set number of agents that execute the tasks in parallel, subject to certain compatibility constraints. Additionally, while the RCPSP generally cares about optimizing the total makespan of the schedule, we instead introduce a set start and end time for each scheduling instance and only focus on its feasibility (i.e. whether or not all tasks can be completed within this specified time frame). An instance of an agent resource-constrained project scheduling problem (ARCPSP) is defined by a tuple (M, J, s, p, U, E, R, B, b), where the components are defined as follows. DISPLAYFORM0 are the allowable time ranges in which the tasks should be executed, where DISPLAYFORM1 -p = [p 1, . . ., p n] is a vector of the durations of tasks J, where p i is the duration of task J i.-U = {U 1, U 2, · · ·, U n} is the compatibility set for the tasks. Each task J i can be completed by a subset U i ⊆ M of agent.-E ⊆ J × J is a set of precedence relations. (J i, J j) ∈ E if and only if task J i must terminate before task J j begins. Precedence relations must be defined in a consistent way (by respecting the transitive property).-R = {R 1, R 2, · · ·, R q} is a set of reusable resources.-B ∈ N q represents the total availability of the resources R. The tasks that share resource B i are mutually exclusive if B i = 1.-b ∈ N n×q represents the resource demands of tasks where task J i requires b i,j units of resource R j during its execution. The total demand of resource R j at anytime cannot exceed its total availability B j. DISPLAYFORM2 is a solution to an instance of an ARCPSP, where S i and A i are the start time and the assigned agent of task J i, respectively. A schedule is feasible if it satisfies the following constraints:• No agent has overlapping tasks, DISPLAYFORM3 • Every task falls within its allowable time frame DISPLAYFORM4 • The activities are assigned to compatible agents DISPLAYFORM5 • The precedence relations are met DISPLAYFORM6 • The resource constraints are satisfied, let J t = {J i ∈ J | S i ≤ t < S i + p i} represent the set of tasks being executed at time t, then DISPLAYFORM7 The constraints of the ARCPSP can be formulated in terms difference logic in the following way. Constraint can be rewritten as DISPLAYFORM0 Constraint can be rewritten as DISPLAYFORM1 and constraint can be rewritten as DISPLAYFORM2 By representing the agents as integers M i ∈ N, constraint can be rewritten as Encoding the resource constraints is slightly more challenging. For mutually exclusive constraints (B i = 1), the tasks that share a resource can simply be encoded as not being allowed to be executed at the same time. That is, constraint FORMULA10 can be rewritten as DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 We can generalize this idea to non-mutually exclusive constraints through the concept of minimal forbidden sets. First introduced by BID9 ), forbidden sets are unsatisfiable sets with respect to resource constraints only. They represent the sets of tasks that cannot be simultaneously scheduled because they would otherwise exceed the availability of some resource constraint. The essential feature of a minimal forbidden set is that a single task can be rescheduled to another time to make the set respect the resource constraint. Therefore, given a minimal forbidden set J *, we would like to encode a constraint requiring that they cannot all be executing at the same time DISPLAYFORM6 where J t = {J i ∈ J | S i ≤ t < S i + p i} represents the set of tasks being executed at time t. This encoding is similar to methods which encode the RCPSP in terms of linear arithmetic BID0, but this requires discretizing time and incurs a cumbersome number of constraints if there are a large number of time-points. Moreover, equation FORMULA0 cannot easily be formulated in terms of difference logic. Instead, we can reformulate the constraint as there being at least two tasks in J * that do not overlap DISPLAYFORM7 for every minimal forbidden set J *. This constraint is logically equivalent to requiring that at any time-point, there be at least one task in each minimal forbidden set that is not being executed. Constraining all of the minimal forbidden sets, a subset of all of the forbidden sets, is sufficient to prevent resource conflicts because every forbidden set is a superset of some minimal forbidden set. Algorithms exist for computing all minimal forbidden sets (Stork and Uetz 2005) so we will not discuss such a computation here. Encoding resource constraints as forbidden sets is efficient in the context of the ARCPSP in comparison to other methods, such as equation FORMULA0. This is primarily because of the computational advantage achieved by difference logic over other theories such as linear real arithmetic and the encoding not requiring a discretization of time. Representing resource constraints as minimal forbidden sets also provides an explicit representation of resource constraints in terms of MUSes. If a resource constraint appears in an explanation, we can represent it as the minimal forbidden set which is being violated. For example, if a resource constraint constitutes a component of some MUS, it will be represented as some subset {A, B, C} of tasks, meaning that tasks A, B, and C cannot be scheduled at the same time because they would violate a resource constraint. Example 1. Scheduling astronauts aboard the ISS We model the problem of scheduling astronauts aboard the International Space Station (ISS) as an instance of the AR-CPSP for which the elements of M = {M 1, M 2, · · ·, M m} represent the crew members. We consider the case of m = 6 astronauts. The bounds on task execution are from minute 120 to 840; the sleeping related tasks outside of this bound are fixed so are not a part of the problem instance. The availability of the power resource, is 1000 units. The tasks are divided into different categories: -There are 6 laboratory tasks, each of duration 120 with allowable time ranges of, the entire work day. However, they have precedence constraints {(J Li, J Li+1) | 1 ≤ i < 6}, each laboratory task must be completed before the next one begins. Each laboratory task can be completed by any astronaut, so the compatibility set is all of the astronauts. Each laboratory task also has a power requirement of 400 units. -There are m weights and m treadmill tasks, one for each agent, each of duration 75. They have allowable time ranges of. Each weight and treadmill task must be completed by a unique astronaut so their compatibility sets can be specified by letting the ith task only be completed by astronaut M i. However, there is only one set of weights and treadmill equipment, so we can define reusable resources R W and R T both with availability B W, B T = 1, respectively. Each treadmill task also has a power requirement of 200 units. -There are m meal tasks. They have allowable time ranges of. Similar to the exercise tasks, each one must be completed by a unique astronaut so their compatibility sets can be specified by letting the ith meal task only be completed by astronaut M i. -Several miscellaneous tasks, deploy cubesat, collect sample, hardware gather, and eva suit test with durations 60, 60, 120, and 120 respectively, do not fall into any particular group. These tasks have an allowable time range of and can all be completed by any astronaut. They require 400, 500, 400, 400 units of power, respectively. A feasible schedule for this instance of the ARCPSP is visualized in FIG0. We modify the previous example slightly to produce an unsatisfiable problem instance. If we change the duration of each laboratory task from 120 to 121, we get an overconstrained system of constraints for which no feasible schedule exists. We'll use this running example to produce explanations in the next several sections. The proposed strategy for enumerating subsets is based on MARCO BID8 ) over other systems such as CAMUS BID7 because outputting at least some MUSes quickly is more important than explicitly generating every MUS. MARCO relies on four main functions to explore the feasibility of a power set lattice of constraints which we outline here in the language of conflicts and relaxations. -BlockUp -Called whenever a conflict is found. Marks every supersets of the current set, preventing it from being explored later on. -BlockDown -Called whenever a relaxation is found. Marks every subsets of the current set, preventing it from being explored later on. -Grow -If the current set is satisfiable, adds constraints to the current set until adding any other constraint would make it unsatisfiable. -Shrink -If the current set is unsatisfiable, removes constraints from the current set until removing any other constraint would make it satisfiable. A power set lattice of Boolean variables representing each constraint in the foreground is maintained throughout the execution of the algorithm. First, a random subset of constraints is constructed by choosing a point in the Boolean lattice. Then the SAT solver checks whether the set is SAT or UNSAT. If the ing assignment is SAT (feasible), then constraints are iteratively added to the current set until a minimal relaxation is found. If the initial set is instead DISPLAYFORM0 The power set lattice of {C 1, C 3, C 4} with C 2 and corresponding relaxation (green) and conflict (red).UNSAT (infeasible), constraints are iteratively removed until a minimal conflict is found. After a minimal conflict is found, BlockUp is called, removing any supersets of the minimal conflict from consideration in the lattice. We can do this because any superset of a conflict must be unsatisfiable because it contains the conflict. The opposite direction also applies, any subset, after a minimal relaxation has been removed, must be satisfiable so we can rule them out of consideration. Hence, after a minimal relaxation is found, we BlockDown, removing any subsets from consideration in the lattice. Then a new satisfying assignment is pulled from the remaining sections of the boolean lattice and new conflicts and relaxations are generated until the entire lattice is blocked or the algorithm times out. Consider the unsatisfiable conjunction of the following set of clauses, DISPLAYFORM0 with B = {C 2}. We'll use this example to step through an execution of the subset enumeration algorithm. A visualization of the associated Boolean lattice is shown in Figure 2. A random initial seed has us start with clause {C 1, C 3} and the SAT solver says it's UNSAT. We then Shrink and remove C 3 from the set and the SAT solver says {C 1} is still UNSAT and minimal. We then output this minimal conflict {C 1}. Because this set is now minimal, we can BlockUp, removing supersets {C 1}, {C 1, C 3}, {C 1, C 4}, {C 1, C 3, C 4} from consideration. We then choose a new seed, let's say {C 3}. The ing set is SAT so we Grow to the set {C 3, C 4} which is then SAT and maximal so we can BlockDown subsets {C 4}, {C 3}, {∅} and output {C 1}, the complement of {C 3, C 4}, as a relaxation. The lattice is then entirely blocked off so we terminate with the single conflict and relaxation. Figure 2 shows the power set lattice of the foreground along with the corresponding relaxation (green) and conflict (red). The standard for the ARCPSP involves constraints to ensure that the ing schedule is logical. This way, the foreground only involves constraints which can be altered by parameters that are controlled by the user.-New variables S 0 and S m+1 are introduced that mark the beginning and end of the schedule bounds. These variables prevents the subset enumeration from relaxing the temporal constraints of tasks outside of the feasible bounds of the schedule DISPLAYFORM0 -Each task is assigned to some existing agent. Without this constraint, the solver could assign tasks to a nonexistent agent to solve conflicts DISPLAYFORM1 -No agent has overlapping activities. Disallowing the solver from consider cases in which tasks can overlap prevents it from generating meaningless . This condition is precisely constraint.We will refer to this set of constraints throughout the following section. The method of generating minimal conflicts and relaxations can be applied to both sets of individual constraints and, by modifying the , sets of tasks. In this section, we investigate the first case, which we call constraint explanations. Following the strategy in the beginning of Section 5, we enumerate only over the constraints that are in the foreground, as specified in Section 5.1. That is, we consider constraints timeframe, compatibility, precedence, and resource referring to equations,,, and, respectively, for each task in the schedule. The rest of the constraints are implied as a part of the because they only correspond to imposing a logical structure on the solution, not constraining the parameters of the schedule. Hence, the Boolean lattice which is enumerated over contains only these four types of constraints. The outputs for constraint explanations are formatted as a tuple of the relevant tasks followed by a constraint type. For example, (LAB 0, LAB 1) precedence refers to the precedence constraint between the first and second laboratory tasks. When a constraint is only relevant to a single task, we write the task followed by the constraint type (e.g., MEAL 0 compatible refers to the agent compatibility constraint for the first meal task).The full constraint explanation for Example 2 includes 14 minimal relaxations and 3 minimal conflicts. FIG2 shows a representative part of this full constraint explanation. The omitted conflicts and relaxations are identical in structure to the ones shown in FIG2 and give practically redundant information. Computing the set of minimal forbidden sets took 2.11 seconds and calculating the full explanation took 1.57 seconds. In this example, relaxations provide the user with minimal ways in which constraints could be changed to fix the schedule. Meanwhile, conflicts give insight into why infeasibility is occurring. For example, Relex 1 indicates that removing the precedence between laboratory task 3 and 4 would make the schedule feasible. However, Confl 1 indicates that the precedence between all of the laboratory tasks does not fit in the schedule. A user could use this latter information to alter the original parameters of the schedule rather than having to void an entire task or constraint. One possible solution could be extending the length of the schedule or shortening the length of some of the laboratory tasks, an option which is not revealed by relaxations alone. This formulation of explainability in terms of conflicts and relaxations also allows a user to ask pointed questions concerning the feasibility of an instance of an ARCPSP problem. Given a feasible instance of a problem, such as Example 1, specific questions may be asked about infeasible modifications of the problem. The modification in Example 2 is gotten by extending the lengths of the laboratory tasks. Hence, the explanation in FIG2 may be interpreted as an answer to the question: "why can the laboratory tasks not have a duration longer than 120 minutes?" In the following sections we explore two variations of the subset enumeration algorithm to generate higher-level descriptions of infeasibility. This is accomplished by pushing every constraint to the and populating the foreground with a fresh set of Boolean variables. Then, a set L of constraints can be constructed that encodes a logical relationship between the new symbolic variables in the foreground and the actual constraints in the . The set L of logical relations linking symbolic variables to the constraints also becomes part of the . Then, only the set of Boolean variables remains to be enumerated over in the foreground. In practice, this can be accomplished by replacing the Boolean lattice outlined in Section 5 by the symbolic lattice composed of the new variables. This enables the generation of explanations concerning these symbolic variables, which is dependent on the relationship L.Depending on what kinds of constraints populate the foreground, the size of the Boolean lattice which needs to be enumerated over can be greatly reduced. This reduces the number of calls that need to be made to the SMT solver before arriving upon conflicts and relaxation. Additionally, these type of explanations can reduce redundancies and produce more compact descriptions of infeasibility. The following section outlines how this concept can be applied to generate minimal conflicts and relaxations of sets of tasks. Task explanations can be generated by replacing the foreground (and hence, the power set lattice) with a set of variables representing individual tasks. We introduce a Boolean variable for each task and separate the constraints of the ARCPSP into two classifications: individual and relational. Constraints FORMULA7 and FORMULA8 as well as constraints FORMULA0 and FORMULA0 are individual constraints, involving only a single task. Constraints, FORMULA10, and are relational constraints, involving multiple tasks. We then encode the constraint that the truth of each task's Boolean variable J j implies the truth of every one of its individual constraints J j =⇒ I j where I j represents a conjunction of the task's timeframe, compatibility, and feasibility Hence, individual constraints need only be satisfied if their associated task's Boolean variable is true and relational constraints need only be satisfied if all of their associated tasks' Boolean variables are true. Through this logical relationship, we can now enumerate over these Boolean variables, each of which conceptually represents a task. As the Boolean variables are toggled on and off, the associated constraint lattice becomes constrained as if the schedule had been constructed with only that subset of tasks. Executing the enumeration algorithm over this modified foreground for the same overconstrained problem (Example 2) produces a similar set of conflicts and relaxations, part of which is shown in FIG3. The full explanation includes 3 minimal relaxations and 17 minimal conflicts in total. It took 4.03 seconds to compute the minimal forbidden sets and 0.55 seconds to compute the full explanation. Here, the task and constraint explanations are quite similar, but this is not always the case. The task explanations can often be much more compact than the constraint explanations because each variable represents many constraints. For similar reasons, the number of total conflicts and relaxations is often greatly reduced. Because of this, task explanations can give more straight forward explanations for the over-constrained problem, but they lack the granularity of the constraint explanations. For example, with the constraint explanations we were able to diagnose that the precedence between the lab tasks was creating an issue rather than a resource or other constraint. The task explanations leave out this detail, sacrificing expressibility for interpretability. In order to draw explanations back to a high-level interpretation of the problem, the foreground can be replaced by a set of human-written specifications. This further reduces the size of the power set lattice that is constructed out the foreground and reduces redundancy in the generated conflicts and relaxations. Tasks are often formed in groups which share certain scheduling specifications. For example, the meal tasks in Example 1 all share the same parameters except that they are assigned to unique astronauts. When Example 1 was described, such similar tasks were naturally formulated in different categories (e.g., meal, weights, etc.). Hence, specifications for tasks may be more succinctly expressed by making use of these similarities. An informal, human written list of constraints specifying Specifications Confl 1 {Lab tasks must happen in sequence, the scheduling horizon is 6am to 6pm} Relax 1 {Lab tasks must happen in sequence} Relax 2 {The scheduling horizon is 6am to 6pm} the parameters of Example 1 could be as follows:-The scheduling horizon is 6am to 6pm.-Meal tasks must be scheduled between 1pm and 2pm.-Lab tasks must happen in sequence.-Each meal/treadmill/weights task must be assigned to a different astronaut.-Weights/treadmill tasks cannot happen 60 minutes before pre-sleep.-No more than 1000 units of power may be drawn at once.-The treadmill tasks require 200 units of power.-Tasks EVA SUIT TEST, HW GATHER, C-SAT, and SAMP require 400, 400, 500, and 400 units of power.-There is only one set of weights and treadmill equipment. Then a relevant logical relationship may be drawn back to the actual set of constraints for each such specification. For example, the precedence relations between the lab tasks could be related through:"Lab tasks must happen in sequence."(LAB 0, LAB 1) precedence (LAB 1, LAB 2) precedence (LAB 2, LAB 3) precedence (LAB 3, LAB 4) precedence (LAB 4, LAB 5) precedence (LAB 5, LAB 6) precedence Similar constraints may be encoded for the rest of the specifications, which compose the set L linking the human written specifications to the actual constraints of the problem. This construction allows the generation of specification explanations. The specification explanation for Example 1 is displayed in FIG4. Notice the greatly reduced size of the specification explanation. Unlike the constraint and task explanation, the specification explanation does not suffer from producing many redundant conflicts and relaxations. A fundamental trade-off exists between the expressibility and interpretability of different kinds of explanations. Low-level explanations involving constraints provide detailed reasons for infeasibility but may be difficult for a human user to parse or understand. In contrast, because the high-level specification explanations correlate directly with the types of constraints which a human planner may think in, they potentially provide more direct and concise information to the user. However, they lack the fine tuned granularity of information that constraint and task explanations provide. For example, if only a single precedence constraint between the laboratory tasks was causing an issue, the specification explanation would obscure which of the constraints is responsible. We introduced the agent resource-constrained project scheduling problem (ARCPSP) along with an associated difference logic encoding. We proposed a general framework for generating minimal conflicts and minimal relaxations based on the MARCO algorithm and demonstrated how it could be used to generate varying types of descriptions for why infeasibility is occurring in instances of the ARCPSP. The framework outlined in this paper is general enough to be applied to constraint satisfaction formulations for various other scheduling and planning problems. These ideas may potentially be further extended to different kinds of formal languages, such as linear temporal logic, that are used to describe planning problems. In an interactive system, such as a scheduling software, when a user attempts to make an infeasible modification, it may be useful to generate a reason for the infeasibility in real time. Similarly, a user could query whether a modification to a feasible schedule would preserve feasibility and, if not, why not? Explanations similar to the ones constructed throughout this paper may likely be used to such an effect. Investigating methods for synthesizing natural language sentences out of the explanations is also subject to future research. Following the goal of QuickXplain BID5, given a partial ordering of constraint or task importance, preferred conflicts and relaxations may be explored earlier and full explanations may list conflicts and relaxations in preferential order. Such functionality would be especially useful in cases for which generating the full explanation is intractable. A preferential ordering of explanations may be achieved by adding and removing constraints during the Grow and Shrink steps based on the constraint preference ordering. Similarly, methods for enumerating disjoint (or otherwise distinct) conflicts may also be useful for producing a representative set of conflicts as concisely as possible. Currently, the most limiting bottleneck for scaling to larger problem instances comes from the number of minimal forbidden sets which can grow exponentially with the number of tasks. Certain lazy clause generation algorithms (Laborie 2003) may be used to represent resource constraints in a more efficient manner. Such representations may also be adapted to implement consumable resources in an explainability setting.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We develop a framework for generating human-understandable explanations for why infeasibility is occurring in over-constrained instances of a class of resource-constrained scheduling problems.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems