Question: Represent the article for finding a single-sentence TLDR

Question: In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way. For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image gen- eration is split into 2 stages. In the first stage a generator network outputs 3D ob- jects. In the second, a differentiable renderer produces an image of the 3D object from a random viewpoint. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to over- come them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset. Generative Adversarial Nets (GAN) (see) have become the gold standard generative model over the years. Their capability is demonstrated in many data sets of natural images. These generative models can create sample images that are nearly indistinguishable from real ones. GANs do not need to make assumptions on the data formation other than applying a neural network on a latent noise input. They can also operate in a fully unsupervised way. GANs have strong theoretical foundations since the beginning. demonstrate that, under suitable conditions, GANs learn to generate samples with the same probability distribution as samples in the training data set have. However, there is one notable drawback of GANs compared to classical generative models like Gaussian Mixture Models by or Naive Bayes classifiers by. Classical models are interpretable, but GANs are not. Interpretability is achieved in classical models by making strong assumptions on the data formation process and by using them on interpretable engineered features. Our work combines the best of both worlds for 3D shape learning. We keep the advantages of GANs: unsupervised training, applicability on real datasets without feature engineering, the theoretical guarantees and simplicity. We also make our representation interpretable, as our generator network provides a 3D mesh as an output. To make the learning possible, we make the assumption that natural images are formed by a differentiable renderer. This renderer produces fake images by taking the 3D mesh, its texture, a image and a viewpoint as its input. During training a discriminator network is trained against the generator in an adversarial fashion. It tries to classify whether its input is fake or comes from the training dataset. The key observation is that a valid 3D object should look realistic from multiple viewpoints. This image realism is thus enforced by the GAN training. This idea was first applied for generating 3D shapes in. Their pioneering work had several limitations though. It was only applicable to synthetic images, where the mask is available and produced only black and white images. Our method works on natural images and we do not need silhouettes as supervision signal. From a theoretical point of view, image realism means that the images lie inside the support of the probability distribution of natural images (see , where image realism was used for super-resolution). AmbientGAN of proves that one can recover an underlying latent data distribution, when a known image formation function is applied to the data, thus achieving data realism by enforcing image realism. Our work and that of are special cases of AmbientGAN for 3D shape learning. However this task in general does not satisfy all assumptions in , which in ambiguities in the training. We resolve these issues in our paper by using suitable priors. We summarize our contributions below: • For the first time, to the best of our knowledge, we provide a procedure to build a generative model that learns explicit 3D representations in an unsupervised way from natural images. We achieve that using a generator network and a renderer trained against a discriminator in a GAN setting. Samples from our model are shown in Figure 1. • We introduce a novel differentiable renderer, which is a fundamental component to obtain a high-quality generative model. Notably, it is differentiable with respect to the 3D vertex coordinates. The gradients are not approximated, they can be computed exactly even at the object boundaries and in the presence of self-occlusions. • We analyze our learning setup in terms of the ambiguities in the learning task. These ambiguities might derail the training to undesirable . We show that these problems can only be solved when one uses labels or prior knowledge on the data distribution. Finally, we provide practical solutions to overcome the problems that originate from the ambiguities. Before we discuss prior work, we would like to state clearly what we mean by supervised, unsupervised and weakly supervised learning in this paper. Usually, supervised learning is understood as using annotated data for training, where the annotation was provided by human experts. In some scenarios the annotation comes from the image acquisition setup. Often these approaches are considered unsupervised, because the annotation is not produced by humans. Throughout our paper we consider supervision based on the objective function and its optimization, and not based on the origin of the annotation. To that purpose, we use the notion of the target and training objective. The target objective is defined as the function that measures the performance of the trained model f, i.e., where l t is the loss function, x, y are the data and labels respectively. The training objective is the function that is optimized during the training, They are defined for the supervised, weakly supervised and unsupervised case as where y w denotes a subset of labels and the loss functions l s, l w and l u may be different from l t. In the case of most supervised tasks (e.g. in classification) the target is the same as the training objective (cross-entropy). Another example is monocular depth estimation. In this case, the inputs are monocular images and the model can be trained using stereo images. This makes it a weakly supervised method under the definitions above. The GAN training is unsupervised, it has the same target and training objectives (the Jensen-Shannon divergence), and thus even the target objective does not use labels. In Table 1, we show the most relevant prior work with a detailed list of used supervision signals. There, we consider the full training scenario from beginning to end. For example if a method uses a pre-trained network from another previous work in its setup, we consider it supervised if the pre-trained network used additional annotation during its training. A very successful 3D generative model is the Basel face model introduced by , which models the 3D shape of faces as a linear combination of base shapes. To create it, classical 3D reconstruction techniques (see) and laser scans were used. This model is used in several methods (e.g., and autoencoders (e.g.) that learn 3D representations by directly using 3D as the supervision signal. The most relevant papers similar to our work are the ones that use differentiable rendering and using randomly sampled viewpoints to enforce image realism. There are GAN based methods by;; and Variational autoencoder based methods by;;. However, Figure 2: Illustration of the training setup. G and D are the generator and discriminator neural networks. R is the differentiable renderer and it has no trainable parameters. The random variables z, m and v are the latent vector, 3D object and the viewpoint parameters. The fake images are x f and the real images are x r. these methods are only applicable on synthetic data or use weak supervision for training as shown in Table 1. Our method can also be interpreted as a way to disentangle the 3D and the viewpoint factors from images in an unsupervised manner. used image triplets for the task. They utilized an autoencoder to reconstruct an image from the mixed latent encodings of other two images. and only use image pairs that share the viewpoint attribute, thus reducing part of the supervision in the GAN training. and use mixing latent variables for unsupervised disentangling. HoloGAN is a method for disentangling objects and viewpoints using latent shape representations. In contrast we learn an explicite 3D mesh representation, which can be used in traditional rendering pipelines. By using this rendering in the training, we demonstrate the disentangling of the 3D shape from the viewpoint without any labels and also guarantee interpretability and consistency across viewpoints. An important component of our model is the renderer. Differentiable renderers like Neural mesh renderer Kato et al. or OpenDR Loper & Black (2014 . have been used along with neural networks for shape learning tasks. Differentiability is essential to make use of the gradient descent algorithms commonly employed in the training of neural networks. We introduce a novel renderer, where the gradients are exactly computed and not approximated at the object boundaries. We are interested in building a mapping from a random vector to a 3D object (texture and vertex coordinates), and a image. We call these three components the scene representation. To generate a view of this scene we also need to specify a viewpoint, which we call the camera representation. The combination of the scene and camera representations is also referred to as simply the representation, and it is used by a differential renderer R to construct an image. We train a generator G in an adversarial fashion against a discriminator D by feeding zero-mean Gaussian samples z as input (see Fig. 2). The objective of the generator is to map Gaussian samples to scene representations m that in realistic renderings x f for the viewpoint v used during training. The discriminator then receives the fake x f and real x r images as inputs. The GAN training solves the following optimization problem, where x f = R(G(z), v) are the generated fake images, m = G(z) are the 3D shape representations and x r are the real data samples. The renderer R is a fixed function, i.e., without trainable parameters, but differentiable. The viewpoints v are randomly sampled from a known viewpoint distribution. In practice, G and D are neural networks and the optimization is done using a variant of stochastic gradient descent (SGD). In this section we give a theoretical analysis of our method and describe assumptions that are needed to make the training of the generator succeed. We build on the theory of and examine its assumptions in the 3D shape learning task. The images in the dataset x r = R(m r, v r) are formed by the differentiable rendering function R given the 3D representation m r ∼ p m and viewpoint v r ∼ p v. Here p m and p v are the "true" probability density functions of 3D scenes and viewpoints. This assumption is needed to make sure that an optimal generator exists. If some real images cannot be generated by the renderer then the generator can never learn the corresponding model. We can safely assume we have a powerful enough renderer for the task. Note that this does not mean the real data has to be synthetically rendered with the specific renderer R. Assumption 2 The ground truth viewpoint v r ∼ p v and the 3D scenes m r ∼ p m are independent random variables. The distribution of the 3D representations p m is not known (it will be learned). The viewpoint distribution p v is known, and we can sample from it, but the viewpoint v r is not known for any specific data sample x r. This assumption is satisfied unless images in the dataset are subject to some capture bias. For example, this would be the case for celebrities that have a "preferred" side of their face when posing for pictures. More technically, this assumption allows us to randomly sample v viewpoints independently from the generated m models. Assumption 3 Given the image formation model This assumption is necessary for learning the ground truth p m. In the unsupervised learning task we can only measure the success of the learning algorithm on the output images during the training. If multiple distributions can induce the same data x ∼ p x, there is no way for any learning algorithm to choose between them, unless prior knowledge on p m (in practice it is an added constraint on m in the optimization) is available. The 3D shape learning task in general is ambiguous. For example in the case of the hollow-mask illusion (see) people are fooled by an inverted depth image. Many of these ambiguities depend on the parametrization of the 3D representation. For example different meshes can reproduce the same depth image with different triangle configurations, thus a triangle mesh is more ambiguous than a depth map. However, if the aim is to reconstruct the depth of an object, the ambiguities arising from the mesh representation do not cause an ambiguity in the depth. We call these acceptable ambiguities. For natural images one has to model the whole 3D scene, otherwise there is a trivial ambiguity when a static is used. The generator can simply move the object out of the camera field of view and generate the images on the like in the GAN training to generate natural images. Modelling the whole scene with a triangle mesh however is problematic in practice because of the large size of the (multiple) meshes that would need. We propose a compromise, where we only model the object with the mesh and we generate a large and crop a portion of it randomly during training. In this way, even if the generator outputs images with a view of the object, there is no guarantee that the crop will still contain the object. Thus, the generator will not be able to match the statistics of real data unless it produces a realistic 3D object. The generator G and discriminator D have large enough capacity, and the training reaches the global optimum of the GAN training 5. In practice, neural networks have finite capacity and we train them with a local iterative minimization solver (stochastic gradient descent). Hence, the global optimum might not be achieved. Nonetheless we show that the procedure works in practice. Now we show that under these conditions the generator can learn the 3D geometry of the scene faithfully. Theorem 1 When the above assumptions are satisfied, the generated scene representation distribution is identical to the real one, thus G(z) ∼ p m, with z ∼ N (0, I). The proof can be readily adapted from. Differentiability is essential for functions used in neural network training. Unfortunately traditional polygon renderers are only differentiable with respect to the texture values but not for the 3D vertex coordinates. When the mesh is shifted by a small amount, the rendered pixel can shift to the or to another triangle in the mesh. This can cause problems during training. Thus, we propose a novel renderer that is differentiable with respect to the 3D vertex coordinates. We make the renderer differentiable by extending the triangles at their boundaries with a fixed amount in pixel space. Then, we blend the extension against the or against an occluded triangle. The rendering is done in two stages. First, we render an image with a traditional renderer, which we call the crisp image. Second, we render the triangle extensions and its alpha map, which we call the soft image. Finally we blend the crisp and the soft images. We render the crisp image using the barycentric coordinates. Let us define the distance of a 2D pixel coordinate p to a triangle T as d(p, T) = min pt∈T d(p, p t) and its closest point in the triangle as p * (T) = arg min pt∈T d(p, p t), where d is the Euclidean distance. For each pixel and triangle the rendered depth, attribute and alpha map can be computed as where z c indicates the depth of the crisp layer, b i are the barycentric coordinates and z i (T) are the depth values of the triangle vertices and z f ar is a large number representing infinity. The vertex attributes of a triangle T are a i (T). The closest triangle index is computed as T * c (p) = arg min T z c (p, T), and it determines which triangle is rendered for the attribute a c (p) in the crisp image. The soft image is computed as where B is the width of the extension around the triangle and λ slope > 0. The final image a(p) is computed as where bcg is the crop. UV mapping is supported as well: first the UV coordinates are rendered for the crisp and soft image. Then the colors are sampled from the texture map for the soft and the crisp image separately. Finally, the soft and the crisp images are blended. Figure 3 shows an illustration of the blended rendering as well as its effect on the training. The 3D representation m = [s, t, b] consists of three parts, where s denotes the 3D shape of the object, t are the texture and b are the colour values. The shape s is a 3 dimensional array with the size of 3×N ×N. We call this the shape image, where each pixel corresponds to a vertex in our 3D mesh and the pixel value determines the 3D coordinates of the vertex. The triangles are defined as the subset of the regular triangular mesh on the N × N grid; we only keep the triangles of the middle circular region. The texture (image) is an array of the size 3 × N t × N t. The renderer uses the UV mapping technique, so the size of the texture image can have higher resolution than the shape image. In practice we choose N t = 2N, so the triangles are roughly 1 or 2 pixels wide and the texture can match the image resolution when rendering a N × N image. The is a color image of size 3 × 2N × 2N. The renderer uses a perspective camera model, where the camera is pointing at the origin and placed along the Z axis such that the field of view is set so the unit ball fits tightly in the rendered image. The viewpoint change is interpreted as rotating the object in 3D space, while the camera stays still. Finally a random N × N section of the renderer is cropped and put behind the object. Notice that the 3D representation (3 dimensional arrays) are a perfect match for convolutional neural network generators. We designed it this way, so we can use StyleGAN as generator (see). We use the StyleGAN generator of with almost vanilla settings to generate the shape image, texture image and the . StyleGAN consist of two networks: an 8 layer fully connected mapping network that produces a style vector from the latent inputs, and a synthesis network that consist of convolutional and upsampling layers that produces the output image. The input of the synthesis network is constant and the activations at each layer are perturbed by the style vector using adaptive instance normalization. For each layer activation also noise is added. It is also possible to mix styles from different latent vectors, by perturbing the convolutional layer activations with different styles at each layer. In our work we used the default setting for style mixing and for most parameters. We modified the number of output channels and the resolution and learning rates. The training was done in a progressively growing fashion, starting at the an image resolution of N = 16. The final resolution was set to N = 128. One StyleGAN instance (G o) generates the shape image and texture and another (G b) generates the . The inputs to both generators are 512 dimensional latent vectors z o and z b. We sampled them independently, assuming the and the object are independent. We set both G o and G b to produce images at 2N × 2N resolution where N is the rendered image size. The output of the object generator is then sliced into the shape and texture image, then the shape image is downsampled by a factor of 2. We multiplied the shape image by 0.002, which effectively sets a relative learning rate, then added s 0 to the output as an initial shape. For faces we set s 0 to a sphere with a radius r = 0.5 and centered at the origin. For buses and cars s 0 was a flat sheet. We noticed that during the training the generation of shapes would not easily recover from bad local minima, which ed in high frequency artifacts and the hollow-mask ambiguity. Thus we use a shape image pyramid to tackle this problem. The generator is set to produce K = 4 shape images, then these images are blurred with varying amounts and summed: where blur(·) is the Gaussian blur and σ is interpreted in pixels on the shape image. We also noticed that the 3D models of the object tended to grow large and tried to model the . This is the of an acceptable ambiguity in the parametrization. In terms of the GAN objective it does not matter if the is modelled by b or s and t. As we are interested in where the object fits in the image, we added a size constraint on the object. The output coordinates are computed as where we set s max = 1.3 and the L 2 norm and tanh functions are interpreted pixel-wise on the shape image s. The effect of both the shape image pyramid and the size constraint can be seen in Figure 4. The discriminator architecture was also taken from StyleGAN and we trained it with default settings. SOFT SIZE FULL VIEW CAR BUS Figure 4: Samples from our methods. CRISP used the crisp renderer, while SOFT uses our proposed renderer. SIZE adds a size constraint in order to prevent the mesh modelling the . FULL is our final model that adds the shape pyramid parametrization. CAR and BUS uses the same settings as FULL except it is initialized with a flat sheet instead of a sphere and we did not add size contraint. We trained our model on the FFHQ faces and on LSUN cars and buses. FFHQ contains 70k high resolution (1024 × 1024) colour images. We resized the images to 128 × 128 and trained our generative model on all images in the dataset. For the viewpoint we found that randomly rotating in the range of ±45 degrees along the vertical and ±15 degrees along the horizontal axis yielded the best . This is not surprising, as most faces in the dataset are close to frontal. For our full model we used pyramid shapes of 4 levels and size constraint of 1.3 and we trained our model for 5M iterations. The are shown on Figure 1 and more samples can be found in the Appendix. We also trained our model on 100k images from each of the LSUN categories. We used the same settings as for FFHQ except for the initialization of the 3D shape and without a size constraint. In Figure 4 we show ablations of the choices of the renderer and network architecture settings. We can see that our soft renderer has a large impact on the training. The crisp renderer cannot learn the shape. Furthermore we can see that the size constraint prevents the mesh to model the , and the shape pyramid reduces the artifacts and folds on the face. We can also see that it is important that we set the viewpoint distribution accurately. With the a large viewpoint range of ±120 degrees our method struggles to learn. It produces self-intersecting meshes and puts multiple faces on the object. We can see that our method can also learn the 3D of other categories, such as cars and buses. Table 2 explains the options used in detail and shows quantitative . Figure 5 shows on interpolated latent vectors. We can see the viewpoint and the identity is disentangled and the transition is smooth. Here, we would like to acknowledge the limitations of our work: • Currently, we use a triangle mesh with fixed topology. In general, this is not sufficient for modeling challenging objects and scenes of arbitrary topology. • The is currently not modeled as part of the triangle mesh, which limits our method for datasets where the object is found at the center of the image. Note that this limitation is the of the specific parametrization and the architecture of the generator and not of the expressive power of our method in general. • The imaging model is currently a Lambertian surface illuminated with ambient light. However, specularity and directional light can be added to the renderer model. This is a relatively simple extension, but the representation of lights as random variables during training needs extensive experimental evaluation. One might claim that our work does use supervision, as the faces FFHQ dataset were carefully aligned to the center of the images. We argue that our method still does not use any explicit supervision signal other than the images themselves. Moreover, as discussed in the limitation section, the centering of the object will be irrelevant when the and the object share the same 3D mesh and texture. In contrast, methods that use annotation cannot be extended to deal with more challenging datasets, where that annotation is not available. Another point is the motivation to generate faces in an unsupervised manner, since there already exist several data sets with lots of annotation. First, we choose FFHQ because it is a very clean dataset and our intention is to demonstrate that unsupervised 3D shape learning is possible. Second, we believe that unsupervised learning is the right thing to do even if annotation is available. Unsupervised methods can be extended to other datasets where that annotation is not available. In , we provide a solution to the challenging and fundamental problem of building a generative model of 3D shapes in an unsupervised way. We explore the ambiguities present in this task and provide remedies for them. Our analysis highlights the limitations of our approach and sets the direction for future work. A SAMPLES Figure 6: Samples from our generator trained on the FFHQ dataset at 128 × 128 resolution. The first column shows random rendered samples. The other columns show the 3D normal map, texture, and textured 3D shapes for 5 canonical viewpoints in the range of ±90 degrees. The images were picked to illustrate the range of quality achieved by our method. We can see that most of the samples have anatomically correct shapes. In some cases there are exaggerated features like a large chin, that is only apparent from the profile view. Faces from those viewpoints are not present in the dataset, which might explain the shortcomings of our method. There are failure cases as well (the bottom row), that look realistic from the frontal view, but do not look like a face from the side.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We train a generative 3D model of shapes from natural images in an fully unsupervised way.

We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We focus on the problem of black-box adversarial attacks, where the aim is to generate adversarial examples using information limited to loss function evaluations of input-output pairs. We use Bayesian optimization (BO) to specifically cater to scenarios involving low query budgets to develop query efficient adversarial attacks. We alleviate the issues surrounding BO in regards to optimizing high dimensional deep learning models by effective dimension upsampling techniques. Our proposed approach achieves performance comparable to the state of the art black-box adversarial attacks albeit with a much lower average query count. In particular, in low query budget regimes, our proposed method reduces the query count up to 80% with respect to the state of the art methods. Neural networks are now well-known to be vulnerable to adversarial examples: additive perturbations that, when applied to the input, change the network's output classification . Work investigating this lack of robustness to adversarial examples often takes the form of a back-and-forth between newly proposed adversarial attacks, methods for quickly and efficiently crafting adversarial examples, and corresponding defenses that modify the classifier at either training or test time to improve robustness. The most successful adversarial attacks use gradient-based optimization methods , which require complete knowledge of the architecture and parameters of the target network; this assumption is referred to as the white-box attack setting. Conversely, the more realistic black-box setting requires an attacker to find an adversarial perturbation without such knowledge: information about the network can be obtained only through querying the target network, i.e., supplying an input to the network and receiving the corresponding output. In real-world scenarios, it is extremely improbable for an attacker to have unlimited bandwidth to query a target classifier. In evaluation of black box attacks, this constraint is usually formalized via the introduction of a query budget: a maximum number of queries allowed to the model per input, after which an attack is considered to be unsuccessful. Several recent papers have proposed attacks specifically to operate in this query-limited context (; 2018; ; ;); nevertheless, these papers typically consider query budgets on the order of 10,000 or 100,000. This leaves open questions as to whether black-box attacks can successfully attack a deep network based classifier in severely query limited settings, e.g., with a query budget of 100-200. In such a query limited regime, it is natural for an attacker to use the entire query budget, so we ask the pertinent question: In a constrained query limited setting, can one design query efficient yet successful black box adversarial attacks? This work proposes a black-box attack method grounded in Bayesian optimization , which has recently emerged as a state of the art black-box optimization technique in settings where minimizing the number of queries is of paramount importance. Straightforward application of Bayesian optimization to the problem of finding adversarial examples is not feasible: the input dimension of even a small neural network-based image classifier is orders of magnitude larger than the standard use case for Bayesian optimization. Rather, we show that we can bridge this gap by performing Bayesian optimization in a reduced-dimension setting and upsampling to obtain our final perturbation. We explore several upsampling techniques and find that a relatively simple nearest-neighbor upsampling method allows us to sufficiently reduce the optimization prob-lem dimension such that Bayesian optimization can find adversarial perturbations with more success than existing black-box attacks in query-constrained settings. We compare the efficacy of our adversarial attack with a set of experiments attacking three of the most commonly used pretrained ImageNet classifiers: ResNet50 , Inception-v3 , and VGG16-bn . Results from these experiments show that with very small query budgets (under 200 queries), the proposed method BAYES-ATTACK achieves success rates comparable to or exceeding existing methods, and does so with far smaller average and median query counts. Further experiments are performed on the MNIST dataset to compare how various upsampling techniques affect the attack accuracy of our method. Given these we argue that, despite being a simple approach (indeed, largely because it is such a simple and standard approach for black-box optimization), Bayesian Optimization should be a standard baseline for any black-box adversarial attack task in the future, especially in the small query budget regime. Within the black-box setting, adversarial attacks can be further categorized by the exact nature of the information received from a query. The most closely related work to our approach are score-based attacks, where queries to the network return the entire output layer of the network, either as logits or probabilities. Within this category, existing approaches draw from a variety of optimization fields and techniques. One popular approach in this area is to attack with zeroth-order methods via some method of derivative-free gradient estimation, as in methods proposed in , which uses time-dependent and data-dependent priors to improve the estimate, as well as , which replaces the gradient direction found using natural evolution strategies (NES). Other methods search for the best perturbation outside of this paradigm; cast the problem of finding an adversarial perturbation as a discrete optimization problem and use local search methods to solve. These works all search for adversarial perturbations within a search space with a hard constraint on perturbation size; other work incorporates a soft version of this constraint and performs coordinate descent to decrease the perturbation size while keeping the perturbed image misclassified. The latter of these methods incorporates an autoencoderbased upsampling method with which we compare in Section 5.3.1. One may instead assume that only part of the information from the network's output layer is received as the of a query. This can take the form of only receiving the output of the top k predicted classes , but more often the restrictive decision-based setting is considered. Here, queries yield only the predicted class, with no probability information. The most successful work in this area is in , which reformulates the problem as a search for the direction of the nearest decision boundary and solves using a random gradient-free method, and in and, both of which use random walks along the decision boundary to perform an attack. The latter work significantly improves over the former with respect to query efficiency, but the number of queries required to produce adversarial examples with small perturbations in this setting remains in the tens of thousands. A separate class of transfer-based attacks train a second, fully-observable substitute network, attack this network with white-box methods, and transfer these attacks to the original target network. These may fall into one of the preceding categories or exist outside of the distinction: in , the substitute model is built with score-based queries to the target network, whereas trains an ensemble of models without directly querying the network at all. These methods come with their own drawbacks: they require training a substitute model, which may be costly or time-consuming, and overall attack success tends to be lower than that of gradient-based methods. Finally, there has been some recent interest in leveraging Bayesian optimization for constructing adversarial perturbations. Bayesian optimization (BO) has played a supporting role in several methods. For example, use BO to solve the δ-step of an alternating direction of method multipliers (ADMM) approach, search within a set of procedural noise perturbations using use BO to find maximal distortion error by optimizing perturbations defined using 3 parameters. On the other hand, prior work in which Bayesian optimization plays a central role performs experiments only in relatively low-dimensional problems, highlighting the main challenge of its application: examines an attack on a spam email classifier with 57 input features, and in image classifiers are attacked but notably do not scale beyond MNIST classifiers. In contrast to these past works, the main contribution of this paper is to show that Bayesian Optimization presents a scalable, query-efficient approach for large-scale black-box adversarial attacks, when combined with upsampling procedures. The following notation and definitions will be used throughout the remainder of the paper. Let F be the target neural network. We assume that F: K is a K-class image classifier that takes normalized inputs: each dimension of an input x ∈ R d represents a single pixel and is bounded between 0 and 1, y ∈ {1, · · · K} denotes the original label, and the corresponding output F (x) is a K-dimensional vector representing a probability distribution over classes. Rigorous evaluation of an adversarial attack requires careful definition of a threat model: a set of formal assumptions about the goals, knowledge, and capabilities of an attacker . We assume that, given a correctly classified input image x, the goal of the attacker is to find a perturbation δ such that x + δ is misclassified, i.e., arg max k F (x + δ) = arg max k F (x). We operate in the score-based black-box setting, where we have no knowledge of the internal workings of the network, and a query to the network F yields the entire corresponding K-dimensional output. To enforce the notion that the adversarial perturbation should be small, we take the common approach of requiring that δ p be smaller than a given threshold in some p norm, where varies depending on the classifier. This work considers the ∞ norm, but our attack can easily be adapted to other norms. Finally, we denote the query budget with t; if an adversarial example is not found after t queries to the target network, the attack fails. As in most work, we pose the attack as a constrained optimization problem. We use an objective function suggested by and used in;: where Most importantly, the input x + δ to f is an adversarial example for F if and only if f (x, y, δ) > 0. We briefly note that the above threat model and objective function were chosen for simplicity and for ease of directly comparing with other black box attacks, but the attack method we propose is compatible with many other threat models. For example, we may change the goals of the attacker or measure δ in 1 or 2 norms instead of ∞ with appropriate modifications to the objective function and constraints in equation 1. In this section, we present the proposed black-box attack method. We begin with a brief description of Bayesian optimization followed by its application to generate black-box adversarial examples. Finally, we describe our method for attacking a classifier trained with highdimensional inputs (e.g. ImageNet) in a query-efficient manner. Bayesian Optimization (BO) is a method for black box optimization particularly suited to problems with low dimension and expensive queries. Bayesian Optimization consists of two main components: a Bayesian statistical model and an acquisition function. The Bayesian statistical model, also referred to as the surrogate model, is used for approximating the objective function: it provides a Bayesian posterior probability distribution that describes potential values for the objective function at any candidate point. This posterior distribution is updated each time we query the objective function at a new point. The most common surrogate model for Bayesian optimization are Gaussian processes (GPs) , which define a prior over functions that are cheap to evaluate and are updated as and when new information from queries becomes available. We model the objective function h using a GP with prior distribution N (µ 0, Σ 0) with constant mean function µ 0 and Matern kernel as the covariance function Σ 0, which is defined as: where d is the dimension of input and {θ i} d i=0 and µ 0 are hyperparameters. We select hyperparameters that maximize the posterior of the observations under a prior . The second component, the acquisition function A, assigns a value to each point that represents the utility of querying the model at this point given the surrogate model. We sample the objective function h at x n = arg max x A(x|D 1:n−1) where D 1:n−1 comprises of n − 1 samples drawn from h so far. Although this itself may be a hard (non-convex) optimization problem to solve, in practice we use a standard approach and approximately optimize this objective using the LBFGS algorithm. There are several popular choices of acquisition function; we use expected improvement (EI) , which is defined as where E n [·] = E[·|D 1:n−1] denotes the expectation taken over the posterior distribution given evaluations of h at x 1, · · ·, x n−1, and h * n is the best value observed so far. Bayesian optimization framework as shown in Algorithm 2 runs these two steps iteratively for the given budget of function evaluations. It updates the posterior probability distribution on the objective function using all the available data. Then, it finds the next sampling point by optimizing the acquisition function over the current posterior distribution of GP. The objective function h is evaluated at this chosen point and the whole process repeats. In theory, we may apply Bayesian optimization directly to the optimization problem in equation 1 to obtain an adversarial example, stopping once we find a point where the the objective function rises above 0. In practice, Bayesian optimization's speed and overall performance fall dramatically as the input dimension of the problem increases. This makes running Bayesian optimization over high dimensional inputs such as ImageNet (input dimension 3 × 299 × 299 = 268203) practically infeasible; we therefore require a method for reducing the dimension of this optimization problem. Images tend to exhibit spatial local similarity i.e. pixels that are close to each other tend to be similar. showed that this similarity also extends to gradients and used this to reduce query complexity. Our method uses this data dependent prior to reduce the search dimension of the perturbation. We show that the adversarial perturbations also exhibit spatial local similarity and we do not need to learn the adversarial perturbation conforming to the actual dimensions of the image. Instead, we learn the perturbation in a much lower dimension. We obtain our final adversarial perturbation by interpolating the learned, low-dimension perturbation to the original input dimension. We define the objective function for running the Bayesian optimization in low dimension in Algorithm 1. We let Π p B(0,) be the projection onto the ∞ ball of radius centered at origin. Our method finds a low dimension perturbation and upsamples to obtain the adversarial perturbation. Since this upsampled image may not lie inside the ball of radius centered at the origin, we project back to ensure δ ∞ remains bounded by. With the perturbation δ in hand, we compute the objective function of the original optimization problem defined in equation 1. We describe the complete algorithm our complete framework in Algorithm 2 where x 0 ∈ R d and y 0 ∈ {1, . . ., K} denote the original input image and label respectively. The goal is to learn an adversarial perturbation δ ∈ R d in much lower dimension, i.e., d < < d. We begin with a small dataset D = {(δ 1, v 1), · · ·, (δ n0, v n0)} where each δ n is a d dimensional vector sampled from a Quering randomly chosen n 0 points. Update the GP on D Updating posterior distribution using available points 4: t ← n 0 Updating number of queries till now 5: while t ≤ T do 6: Optimizing the acquisition function over the GP 7: Querying the model 8: given distribution and v n is the function evaluation at δ n i.e v n = OBJ-FUNC(x 0, y 0, δ n). We iteratively update the posterior distribution of the GP using all available data and query new perturbations obtained by maximizing the acquisition function over the current posterior distribution of GP until we find an adversarial perturbation or run out of query budget. The Bayesian optimization iterations run in low dimension d but for querying the model we upsample, project and then add the perturbation to the original image as shown in Algorithm 1 to get the perturbed image to conform to the input space of the model. To generate a successful adversarial perturbation, it is necessary and sufficient to have v t > 0, as described in Section 3. We call our attack successful with t queries to the model if the Bayesian optimization loop exits after t iterations (line 12 in Algorithm 2), otherwise it is unsuccessful. Finally, we note that the final adversarial image can be obtained by upsampling the learned perturbation and adding to the original image as shown in Figure 1. In this work, we focus on ∞ -norm perturbations, where projection is defined as: where is the given perturbation bound. The upsampling method can be linear or non-linear. In this work, we conduct experiments using nearest neighbor upsampling. A variational autoencoder or vanilla autoencoder could also be trained to map the low dimension perturbation to the original input space. We compare these different upsampling schemes in Section 5.3.1. The initial choice of the dataset D to form a prior can be done using standard normal distribution, uniform distribution or even in a deterministic manner (e.g. with Sobol sequences). Our experiments focus on the untargeted attack setting where the goal is to perturb the original image originally classified correctly by the classification model to cause misclassification. We primarily consider performance of BAYES-ATTACK on ImageNet classifiers and compare its performance to other black-box attacks in terms of success rate over a given query budget. We also perform ablation studies on the MNIST dataset by examining different upsampling techniques and varying the latent dimension d of the optimization problem. We define success rate as the ratio of the number of images successfully perturbed for a given query budget to the total number of input images. In all experiments, images that are already misclassified by the target network are excluded from the test set; only images that are initially classified with the correct label are attacked. For each method of attack and each target network, we compute the average and median number of queries used to attack among images that were successfully perturbed. We treat the latent dimension d used for running the Bayesian optimization loop as a hyperparameter. For MNIST, we tune the latent dimension d over {16, 64, 100, 256, 784}. Note that 784 is the original input dimension for MNIST. While for ImageNet, we search for latent dimension d and shape over the range {48(3 × 4 × 4), 49(1 × 7 × 7), 100(1 × 10 × 10), 108(3 × 6 × 6), 400(1 × 20 × 20), 432(3 × 12 × 12), 576(1 × 24 × 24), 588(3 × 14 × 14), 961(1 × 31 × 31), 972(3 × 18 × 18)}. For ImageNet, the latent shapes with first dimension as 1 indicate that the same perturbation is added to all three channels while the ones with 3 indicate that the perturbation across channels are different. In case of ImageNet, we found that for ResNet50 and VGG16-bn different perturbation across channels work much better than adding the same perturbation across channels. While for Inception-v3, both seem to work equally well. We initialize the GP with n 0 = 5 samples sampled from a standard normal distribution. For all the experiments in next section, we use expected improvement as the acquisition function. We also examined other acquisition functions (posterior mean, probability of improvement, upper confidence bound) and observed that our method works equally well with other acquisition functions. We independently tune the hyper-parameters on a small validation set and exclude it from our final test set. We used BoTorch 1 packages for implementation. We compare the performance of the proposed method BAYES-ATTACK against NES , BANDITS-TD and PARSIMONIOUS , which is the current state of the art among score-based black-box attacks within the ∞ threat model. On ImageNet, we attack the pretrained 2 ResNet50 , Inception-v3 and VGG16-bn . We use 10,000 randomly selected images (normalized to ) from the ImageNet validation set that were initially correctly classified. We set the ∞ perturbation bound to 0.05 and evaluate the performance of all the methods for low query budgets. We use the implementation NES and BANDITS-TD. Similarly for PARSIMONIOUS, we use the implementation 4 and hyperparameters given by. Figure 2 compares the performance of the proposed method BAYES-ATTACK against the set of baseline methods in terms of success rate at different query budgets. We can see that BAYES-ATTACK consistently performs better than baseline methods for query budgets < 200. Even for query budgets > 200, BAYES-ATTACK achieves better success rates than BANDITS-TD and NES on ResNet50 and VGG16-bn. Finally, we note that for higher query budgets (> 1000), both PARSIMONIOUS and BANDITS-TD method perform better than BAYES-ATTACK. To compare the success rate and average/median query, we select a point on the plots shown in Figure 2. Table 1 compares the performance of all the methods in terms of success rate, average and median query for a query budget of 200. We can see that BAYES-ATTACK achieves higher success rate with 80% less average queries as compared to the next best PARSIMONIOUS method. Thus, we argue that although the Bayesian Optimization adversarial attack approach is to some extent a "standard" application of traditional Bayesian Optimization methods, the performance over the existing state of the art makes it a compelling approach particularly for the very low query setting. We also compare the average 2 distortion of the generated adversarial perturbations in Appendix B. Success rate (a) Performance comparison with different upsampling schemes. For MNIST, we use the pretrained network (used in) with 4 convolutional layers, 2 max-pooling layers and 2 fully-connected layers which achieves 99.5% accuracy on MNIST test set. We conduct ∞ untargeted adversarial attacks with perturbation bound = 0.2 on a randomly sampled 1000 images from the test set. All the experiments performed on MNIST follow the same protocols. The proposed method requires an upsampling technique for mapping the perturbation learnt in the latent dimension to the original input dimension. In this section, we examine different linear and non-linear upsampling schemes and compare their performance on MNIST. The approaches we consider here can be divided into two broad groups: Encoder-Decoder based methods and Interpolation methods. For interpolation-based methods, we consider nearest-neighbor, bilinear and bicubic interpolation. For encoder-decoder based approaches, we train a variational autoencoder by maximizing a variational lower bound on the log marginal likelihood. We also consider a simple autoencoder trained by minimizing the mean squared loss between the generated image and the original image. For both the approaches, we run the Bayesian optimization loop in latent space and use the pretrained decoder (or generator) for mapping the latent vector into image space. For these approaches, rather than searching for adversarial perturbation δ in the latent space, we learn the adversarial image x + δ directly using the Bayesian optimization. Figure 3a compares the performance of different upsampling methods. We can see that Nearest Neighbor (NN) interpolation and VAE-based decoder perform better than rest of the upsampling schemes. However, the NN interpolation achieves similar performance to the VAE-based method but without the need of a large training dataset which is required for accurately training a VAEbased decoder. We perform a sensitivity analysis on the latent dimension hyperparameter d used for running the Bayesian optimization. We vary the latent dimension over the range d ∈ {9, 16, 64, 100, 256, 784}. Figure 3b shows the performance of nearest neighbor interpolation method for different latent dimension. We observe that lower latent dimensions achieve better success rates than the original input dimension d = 784 for MNIST. This could be because with increase in search dimension, Bayesian optimization needs more queries to find successful perturbation. We also note that for the case of latent dimension d = 9, BAYES-ATTACK achieves lower success rates which could mean that it is hard to find adversarial perturbations in such low dimension. We show the convergence plots of BAYES-ATTACK on ImageNet and MNIST in Appendix A. We considered the problem of black-box adversarial attacks in settings involving constrained query budgets. We employed Bayesian optimization based method to construct a query efficient attack strategy. The proposed method searches for an adversarial perturbation in low dimensional latent space using Bayesian optimization and then maps the perturbation to the original input space using the nearest neighbor upsampling scheme. We successfully demonstrated the efficacy of our method in attacking multiple deep learning architectures for high dimensional inputs. Our work opens avenues regarding applying BO for adversarial attacks in high dimensional settings. In this section, we show the convergence of Bayesian Optimization (BO) in terms of objective value versus the number of queries. Note that we have framed our objective of finding adversarial perturbation to be a maximization problem and we stop the iteration loop of BO once the objective value reaches a positive value. A positive objective value corresponds to a successful adversarial perturbation as described in Section 3. Figure 4 shows the convergence of objective function in the BAYES-ATTACK on RESNET50 trained on ImageNet as described in Section 5.2. We run the BO in 972 dimensions (3 × 18 × 18) and upsample the perturbation to the original input dimension of 150, 528(3×224×224). The plot shows ten randomly chosen images from the validation set, with different colors representing different images. We also show the convergence of the BO on MNIST by varying the latent dimension. Specifically, we compare the convergence with latent dimension 16(4 × 4) and the original input dimension 784(28 × 28). The plot is shown in Figure 5. Each color represents a test image, while dashed lines and solid lines represent runs of BO using 16 and 784 dimensions, respectively. As we can see from the graph, BO in 784 dimensions does not converge to a successful attack (i.e., objective value > 0) in 500 iterations on either of the images, while BO with 16 dimensions on the same images finds the adversarial perturbation in less than 200 iterations. This aligns with our observation that with increase in latent dimension, it becomes harder for BO to find successful perturbation and indeed it would require much more queries than running BO in lower dimensions. We compare the average 2 distortion per image of the proposed method BAYES-ATTACK with the current state-of-the-art methods including gradient-based approaches on RESNET50 trained on ImageNet. We fix the query budget at 200 similar to the experiments described in Section 5.2 and compute the distortion using only the successful adversarial perturbations. As we can see from Table 2, the 2 distortion of adversarial examples generated using BAYES-ATTACK is almost similar to the current state-of-the-art methods but achieves better attack success rate in low query budget regimes. Having said that, as in BANDITS-TD and PARSIMONIOUS , our approach focuses on finding successful adversarial perturbations subject to a pre-defined maximum distortion specified in terms of ∞ distance.

represent the tldr for finding the corresponding scientific article

E.g. We introduce online explanation to consider the cognitive requirement of the human for understanding the generated explanation by the agent. == As Artificial Intelligence (AI) becomes an integral part of our life, the development of explainable AI, embodied in the decision-making process of an AI or robotic agent, becomes imperative. For a robotic teammate, the ability to generate explanations to explain its behavior is one of the key requirements of an explainable agency. Prior work on explanation generation focuses on supporting the reasoning behind the robot's behavior. These approaches, however, fail to consider the mental workload needed to understand the received explanation. In other words, the human teammate is expected to understand any explanation provided, often before the task execution, no matter how much information is presented in the explanation. In this work, we argue that an explanation, especially complex ones, should be made in an online fashion during the execution, which helps spread out the information to be explained and thus reducing the mental workload of humans. However, a challenge here is that the different parts of an explanation are dependent on each other, which must be taken into account when generating online explanations. To this end, a general formulation of online explanation generation is presented along with three different implementations satisfying different online properties. We base our explanation generation method on a model reconciliation setting introduced in our prior work. Our approaches are evaluated both with human subjects in a standard planning competition (IPC) domain, using NASA Task Load Index (TLX), as well as in simulation with ten different problems across two IPC domains. As intelligent robots become more prevalent in our lives, the interaction of these AI agents with humans becomes more frequent and essential. One of the most important aspects of human-AI interaction is for the AI agent to provide explanations to convey the reasoning behind the robot's decision-making BID0. An explanation provides justifications for the agent's intent, which helps the human maintain trust of the robotic peer as well as a shared situation awareness BID1, BID2. Prior work on explanation generation often focuses on supporting the motivation for the agent's decision while ignoring the underlying requirements of the recipient to understand the explanation BID3, BID4, BID5. However, a good explanation should be generated in a lucid fashion from the recipient's perspective BID6.To address this challenge, the agent should consider the discrepancies between the human and its own model while generating explanations. In our prior work BID6, we encapsulate such inconsistencies as model differences. An explanation then becomes a request to the human to adjust The model reconciliation setting BID6. M R represents the robot's model and M H represents the human's model of expectation. Using M H, the human generates π M H, which captures the human's expectation of the robot. Whenever the two plans are different, the robot should explain by generating an explanation to reconcile the two models. the model differences in his mind so that the robot's behavior would make sense in the updated model, which is used to produce the human's expectation of the robot. The general decision-making process of an agent in the presence of such model differences is termed model reconciliation BID6, BID7.One remaining issue, however, is the ignorance of the mental workload required of the human for understanding an explanation. In most earlier work on explanation generation, the human is expected to understand any explanation provided regardless of how much information is present and no discussion has been provided on the process for presenting the information. In this work, we argue that explanations, especially complex ones, should be provided in an online fashion, which intertwines the communication of explanations with plan execution. In such a manner, an online explanation requires less mental workload at any specific point of time. One of the main challenges here, however, is that the different parts of an explanation could be dependent on each other, which must be taken into account when generating online explanations. The online explanation generation process spreads out the information to be communicated while ensuring that they do not introduce cognitive dissonance so that the different parts of the information are perceived in a smooth fashion. Let us illustrate the concept of online explanations through a familiar situation between two friends. Mark and Emma want to meet up to study together for an upcoming exam. Mark is a take-it-easy person so he plans to break the review session into two 60 minutes parts, grab lunch in between the sub-sessions and go for a walk after lunch. On the other hand, Mark knows that Emma is of a more focused type who would rather keep the review in one session and get lunch afterwards. Mark would like to keep his plan. However, had he explained to Emma at the beginning of his plan, he knew that Emma would have proposed to order takeouts for lunch on the way before the review session. Instead, without revealing his plan, he goes with Emma to the library. After studying for 60 minutes, he then explains to Emma that he cannot continue without energy, which makes going to lunch the best option for both. At the same time, Mark refrained from telling Emma (until after lunch) that he also needed a walk since otherwise Emma would have proposed for him to take a walk alone while she stays a bit longer for review, and then to meet up at the lunch place. The above example demonstrates the importance of providing an explanation in an online fashion. Mark gradually reveals the reasoning to maintain his plan as the execution unfolds so that it also becomes both acceptable and understandable to Emma, even though being subject to different values due to model differences (e.g., Mark values lunch break more than Emma thinks he does). The key point here is to explain minimally and only when necessary. In this way, the information to be conveyed is spread out throughout the plan execution, potentially with even a reduced amount of information, so that there is less mental workload requirement at the current step-from Emma's perspective, the interaction with Mark is more straightforward. In this paper, we develop a new method for explanation generation that intertwines explanation with plan execution. The new form of explanation is referred to as online explanation, which considers the mental workload of the receiver of an explanation by breaking it into multiple parts that are to be communicated at different times during the plan execution. We implemented three different approaches for online explanation generation, each focusing on different "online" properties. In the first approach, our focus is on matching the plan prefix. In the second approach, the focus is on making the very next action understandable to the human teammate. In the third approach, the focus is on matching the prefix of the robot's plan with any possible optimal human plan. We use a model search method that ensures that the earlier information communicated would not affect the later parts of the explanation. This creates a desirable experience for the recipient by significantly reducing the mental workload. Our approaches are evaluated both with human subjects and in simulation. II. RELATED WORK AI and its numerous applications have provided astounding benefits in areas such as transportation, medicine, finance and military in recent years, but AI agents are so far limited in their ability to operate as a teammate. To be considered a teammate, the agent must not only achieve a given task, but also provide a level of transparency to other members of the team BID2. One of the ways to achieve this is to enable AI agents to be self-explanatory in their behaviors. Recently, explainable AI paradigm BID8 rises as one essential constituent of human-AI collaboration. Explainable AI helps improve human trust of the AI agent and maintain a shared situation awareness by contributing to the human's understanding of the underlying decision-making process of the agent. The explainable agency's effectiveness BID9 is assessed based on its capability to model the human's perception of the AI agent accurately. This means that an explainable AI agent must not only model the world, but also the other agents' perception of itself BID10. This model of the other agents allows the agent to infer about their expectation of itself. Using this model, an agent can generate legible motions BID11, explicable plans BID7, BID12, BID13, or assistive actions BID14. In these approaches, an agent often substitutes cost optimality with a new metric that simultaneously considers cost and explicability. Another way of using the model is for an AI agent to signal its intention before execution BID15. The motivation here is to use the model to search for additional context information that would help improve human understanding. A third way of using this model is for the agent to explain its behavior by generating explanations BID3, BID4, BID5. Similar to intention signaling, this method has the benefit that the agent can maintain its optimal behavior. Research along this direction has focused on generating the "right" explanations based on the recipient's perception model of an explanation BID6, BID16. This is useful, however, only with the assumption that the explanation can be understood, regardless of how much information is provided or whether sufficient time is given-the mental workload that is required for understanding an explanation is largely ignored. In our prior work, we have studied how the ordering of the information of an explanation may influence the perception of an explanation BID17. In this work, we further argue that an explanation must sometimes be made in an online fashion. This is especially true for complex explanations that require a large amount of information to be conveyed. The idea behind online explanation generation is to provide a minimal amount of information that is sufficient to explain part of the plan that is of interest currently (e.g., the next action), and in such a way intertwine explanation generation with plan execution. Our problem definition is based on the model reconciliation setting defined in our prior work BID6. We provide a brief review of the relevant concepts before defining our problem in this work. Our problem is closely associated with planning problems so we first provide the here. A planning problem is defined as a tuple (F, A, I, G) using PDDL BID18, similar to STRIPS BID19. F is the set of predicates used to specify the state of the world and A is the set of actions used to change the state of the world. Actions are defined with a set of preconditions, add and delete effects. I, G are the initial and goal state. DISPLAYFORM0 and π * I,G is the robot's plan to be explained. Where cost(π * I,G, M R) is the cost of the plan generated using M R and cost * M R (I, G) is the cost of the optimal plan based on the initial and goal state pair under M R. In other words, the robot plan to be explained is required to be optimal according to M R, assuming rational agents. The model reconciliation setting also takes the human's model M H into account, which captures the human's expectation of the robot's behavior. When the robot's behavior to be explained (i.e., π * I,G) matches with the human's expected behavior, the models are said to be reconciled for the plan. A figure that illustrates the model reconciliation setting is presented in FIG0. Explanation generation in a model reconciliation setting means bringing two models, M H and M R, "close enough" by updating M H such that π * I,G, the robot's plan, becomes fully explainable (optimal) in the human's model. A mapping function was defined in BID6 to convert a planning problem into a set of features that specifies the problem as Γ: M −→ S is a mapping function, which transfers any planning problem (F, A, I, G) to a state s in the feature space as follows: DISPLAYFORM1 In other words, the mapping function converts a planning problem into a set of features that specifies the problem. Definition 2 (Explanation Generation BID6): The explanation generation problem is a tuple (π * I,G, M R, M H), and an explanation is a set of unit feature changes to DISPLAYFORM2 where M H is the model after the changes. An explanation hence reconciles two models by making the cost difference between the human's expected plan and the robot's plan smaller after the model updates. Definition 3 (Complete Explanation BID6): Given an explanation generation problem, a complete explanation is an explanation that satisfies cost(π * DISPLAYFORM3 . The robot's plan must be optimal in the human's model after a complete explanation ( M H). A minimal complete explanation (MCE) BID6 is defined as a complete explanation that contains the minimum number of unit feature changes. While the previous explanation generation approach provides a framework to generate explanations considering both the robot's model and the human's model, it largely ignores the mental workload requirement of the human for understanding the explanation. We introduce online explanation generation to address this issue. The key here is to only provide a minimal amount of information during the plan execution to explain the part of the plan that is of interest and not explainable. Definition 4 (Online Explanation Generation): Given a model reconciliation problem, an online explanation is a set of sub-explanations (e k, t k), where e k represents the kth set of unit features to be made (as a sub-explanation) at step t k in the plan. Basically, an online explanation requires only that any actions in the robot's plan before the kth sub-explanations will match with that of the human's expectation. In such a way, the robot can split an explanation into multiple parts, which are made in an online fashion as the plan is being executed. We provide three different approaches of online explanation generation based on the definition provided, while each of these approaches focus on one aspect of explanation generation intertwined with plan execution. Section IV-A discusses OEG with Plan Prefix matching, Section IV-B describes OEG with Next Action matching and Section IV-C explains OEG with any prefix matching. To generate the sub-explanations (i.e., {e k}) for an online explanation, the planning process must consider how the sequence of model changes would in the changes of the human's expectations after each sub-explanation. Similar to the search process for complete explanations BID6, we convert the problem of explanation generation to the problem of model search in the space of possible models. The challenge here is that the model changes may not be independent, i.e., future changes may render a mismatch in the previously reconciled plan prefixes. To address this issue, it must be ensured that the model changes after e k, i.e., e k+1:m where m denotes the size of the set of sub-explanations, would not change the plan prefixes in M H. This can be achieved by searching from M R to M H to find the largest set of model changes which ensure that the plan prefix would not change afterwards after further sub-explanations. This search process is illustrated in FIG1. An OEG-PP is a set of subexplanations (e k, t k) such that: DISPLAYFORM0 where Prefix(π, t) returns the prefix of a plan π up to step t k−1. E k represents e 1:k and π H E k is the optimal plan created from M H E k (M H after providing sub-explanations e 1 to e k). More specifically, the following process will be performed recursively for each sub-explanation. First, we continue moving along π * I,G = (a 1, a 2, ..., a n) as long as the plan prefix matches with the prefix of the plan using the human model M H. Let t = t 1 be the first plan step where they differ. Our search for the sub-explanation starts BID6, the difference is that in our approach the search starts from the robot model and stops where the plan prefixes for the updated human model and the robot model match, while in the previous approach the search process starts from the human model (M H). In this aspect, our research process is more akin to MME BID6. However, since we are focusing on matching the prefixes rather than the whole plan in one shot, our approach must run this process multiple times compared to only once in MME. While seemingly more computationally expensive, this characteristic actually allows us to beat both MCE and MME in terms of computation since our approaches at any time consider only a small set of changes (see ). The dotted line represents the border of the maximum state space model modification in robot model which reconciles the two models up to where the plan execution currently is. Maximum updates to the robot model is equivalent to minimum updates to the human model. with M R. It finds the largest set of model changes to M R such that the prefix of a plan using the corresponding model (i.e., M R minus the set of changes) matches with that of π * I,G up to step t 2 − 1. The complement set of changes (i.e., the difference between M H and M R, minus this set of changes) will be e 1. For the next recursive step, we will start from action t 1 and the human model will be M H E1. To ensure that the prefix (up to t 2 − 1) will be maintained for future steps, we directly force the later plans to be compatible with the prefix. Since we know that an optimal plan exists that satisfies this requirement following the search process, this would not affect our solution for online explanation. The recursive search algorithm for model space OEG is presented in Algorithm 1 for finding e k given E k−1. To search for e k, we use a recursive model reconciliation procedure on the model space. Given M DISPLAYFORM1 and M R, we start off with finding the difference between these two models, and modify M R with respect to M H to find the largest set of model changes that can satisfy constraints introduced in Eq.. This algorithm continues until the human's plan matches with that of the robot's plan. B. OEG for matching Next Action (OEG-NA)Throughout OEG-PP, we assume that generating explanations would modify M H, and the goal of explanation generation is to ensure that the robot and human plan have the same prefix at any step of plan execution. However, this is not always required since the human may not be interested in actions that occurred. Hence, we relax earlier than the current action the plan prefix condition, such that the robot needs only to reconcile between M R and M H to match the very next action in π * I,G and π DISPLAYFORM2 at step t k, regardless of the earlier actions in the plan prefix. This approach is also motivated by the fact that the human is with prefix set up to DISPLAYFORM3 DISPLAYFORM4, t k, ∆); return λ max as e k; known to have limited cognitive memory span BID20. In the most limited case, the agent focuses on explaining the very next action that is different between the most recent human plan π DISPLAYFORM5 and π * I,G. Similar to Algorithm 1, We perform a recursive model reconciliation procedure on the model space. Compared to other two approaches, first, we perform the search from M H \M H rather than M R (see FIG1) since it is computationally faster due to the fact that the plan prefixes do not need to be identical and since the search procedure is monotonic, the search would be equivalent as if the procedure started from M R. The other difference here is that we do not compare the entire plan prefix. Instead, the agent explains only the immediate next action that does not match in the human and robot plans that, without requiring the explanation also maintains the match between the prefixes. In this aspect, the search process of OEG is similar to that of minimally monotonically explanation (MME) in BID6, except that the process must be executed multiple times for OEG due to its online fashion. In the implementations, however, our algorithms actually combine search from M H and M R for a better performance, given the fact that latter model updates do not often affect the previous sub-explanations: DISPLAYFORM6 One assumption in the OEG-PP approach is that the robot has only right plan. Subsequently, the robot's goal is to reconcile the human's plan with respect to its own plan using model space search. We relax this assumption by assuming that there is a set of optimal plans. In such a setting, the robot does not need to explain as long as there exists a human plan that has the same prefix as the robot's plan earlier than the current action. The goal of OEG here is thus to satisfy the following: DISPLAYFORM0 is the human optimal plan generated from the original human model (M H). A straightforward solution to OEG-AP is to generate all human optimal plans and check if any one of them matches with the robot's plan (prefix). This approach however is computationally expensive. Instead, we implemented a compilation approach. To check that a plan prefix Prefix(π * I,G, t k − 1) in the robot's plan is also a prefix in the human's model, we first compile the problem in the human's model into a new problem such that the robot's plan prefix would always be a prefix of the human's plan. If the cost of the human's optimal plan in this new domain model is equal to the cost of the human's optimal plan before the compilation, then clearly there exists an optimal plan in the human's model that matches the prefix. Otherwise, we know that an explanation must be made. Hence, the key here is to ensure that a plan prefix is always satisfied in the compiled model. This is not difficult to achieve. For all i ≥ 1, such that a i, a i+1 ∈ Prefix(π * I,G, t k − 1), where a i, a i+1 are two consecutive actions in π * I,G, the compilation can be achieved by adding a predicate p i to a i as an effect, which is a prerequisite for a i+1. a i+1, in its turn deletes p i and adds p i+1 which is a prerequisite for a i+2, etc. To search for e k, we again use a recursive model reconciliation process on the model space, similar to Algorihm 1. Similar to IV-A, we start off with finding the difference between these two models. The main difference in this approach is that after each model update after a subexplanation, the agent checks if there exists a human optimal plan that has the same plan prefix as the robot's plan up until the next action using the compilation approach described above. This check stops when such a plan does not exist and a new sub-explanations must be identified by model space search. This process continues until an optimal human plan exists that matches the robot's plan. Note however that this does not mean that an optimal planner would necessary return the same plan using the human's model. We evaluated our approach for online explanation generation both with human subjects and in simulation for the different approaches introduced above and compared the with Minimally Complete Explanation (MCE) BID6 approach. For simulation, the goal is to see that how online explanation is in general different from MCE in terms of the information needed and computation time. We evaluated our approach on ten different problems across the rover domain and barman domain-two standard IPC domain described below. For both human and simulation evaluations, the differences between M H and M R are made by randomly removing preconditions from an arbitrarily chosen set of model features. For human subject study, the aim is to confirm the benefits of online explanation generation. Our hypothesis is as follows:• Online explanation generation will reduce mental workload and improve task performance. We evaluated our approach with human subjects on a modified rover domain (see Sec. V-D). In this domain, the rover is supposedly on Mars and the goal is to explore the space to take rock and soil samples as well as taking images and communicate the after analysis to the base station via the lander. In order to take any image, the rover must first calibrate its camera with respect to the target. To sample rock or soil, the robot must have an empty space in its storage. At any point of time, the rover only has enough space to store one sample. In order to take and π R for the Rover domain problems. The y-axis represents the distances while x-axis represents the number of E k (sub-explanations). multiple samples, it must drop the current sample before taking another sample BID22. In this domain, the robot assumes the role of a barman whose goal is to serve a desired set of drinks using drink dispensers, glasses and a shaker. The constraints are that the robot can grab one object if its hand is empty, the robot can grab one object with one hand, and before filling it with a drink, a glass should be empty and clean BID22. TAB1 shows the simulation comparing minimally complete explanations (MCE) withx OEG-PP, OEG-NA and OEG-AP approaches for 5 problems in the rover domain and 5 problems in the barman domain. While the average number of model features of OEG (in a sub-explanation) being shared at each instance of time is considerably lower that MCE (every feature in the explanation is presented at once), the total number of model features in an explanation are the same for MCE and OEG-PP across most of the problems. We can see that in some cases (for instance, P3 from the Rover domain), the total number of model features in the explanation for OEG-PP and OEG-NA is more than that of MCE, which is expected since OEG is focused on generating the minimal amount of information at each time step, instead of the amount overall. The reason for sharing more information in total in OEG-PP and OEG-NA, when compared to MCE, lies in the dependence between the features and the behavior of the planner (i.e., which optimal is returned). While OEG-AP seems to have improved over the amount of information in an explanation, it actually only shows the advantage of considering all optimal plans instead of the one returned by the planner. Comparing both the OEG-NA and OEG-AP approaches with MCE and OEG-PP, there is a remaining distance between the robot's plan and the human's plan in terms of plan action distance (also returned by an optimal planner). The distance of OEG-NA is due to the fact that only the immediate next action is considered. For OEG-AP, as we explained, there is no guarantee that the plan returned using the human's model will be the same as the robot's plan since it considers all optimal human plans and only requires one of them to match the robot's. This is also illustrated more clearly in FIG3. Furthermore, ion OEG approaches, since the execution and explanation is intertwined, the plan distance BID21 between π DISPLAYFORM0 and π R in our approaches gradually moves towards 0 as shown, which suggests a "smoother" adjustment for M H during the execution. This is expected to have a positive effect on the human's mental workload, which we evaluated next.. In our implementation, the possible model updates are sorted ascending based on their feature size and our algorithms start checking the ones with the smallest changes from the robot's side. The consistency check is left as we proceed to the next sub-explanation and backtracking is performed when it fails. This search process takes advantage of the fact that latter information often does not affect the previous sub-explanations. To test our hypothesis, we designed a human study to compare our three approaches for online explanation generation with minimally complete explanation (MCE) BID6. Furthermore, to ensure that the performance difference is not solely due to simply breaking information into multiple pieces, we also implement another approach that randomly breaks MCE during plan execution (referred to as MCE-R). We conducted our experiment using Amazon Mechanical Turk (MTurk) with 3D simulation. The subjects were given an introduction to the rover domain and the task they were supposed to help with. Each subject was given a 30-minute limit to finish the task. Explanations were provided using plain English language and rover actions were depicted using GIF images from a 3D simulated scenario as the rover executes the plan. Figure 4 shows the 3D simulated scenario presented to the subjects. In this experiment, the human subject acts as the rover's commander, where the robot is on Mars and supposed to perform a mission autonomously. The human subject observes the rover's plan sequentially and is asked to determine whether the rover's current action is questionable or not, with explanations provided by OEG approaches or MCEs. Each subject can only perform the task for one setting to reduce the influence between different runs. To observe the effect on mental workload more clearly, we have also added a few spatial puzzles to the experiment as a secondary task to create additional cognitive demand. In the scenario, we deliberately remove certain information from the domain so that the subject would create an incorrect plan, when no explanation is given. In particular, we did not inform them that the storage is limited, the memory is limited, the camera must be calibrated, and the camera must be calibrated with respect to the objective. This hidden information introduces differences between M H and M R in the model reconciliation setting, and hence ing in scenarios where explanations must be provided. In this scenario, for example, the subject may question the action for calibrating the camera if they were not specifically told to consider that. In MCE setting, the robot shares all the information at the beginning of the task BID6, while the information is randomly broken to be communicated at different steps in MCE-R. In each of the OEG setting, the robot uses different approaches of online explanation generation, which intertwines the communication of explanation with the plan execution. In particular, the four pieces of missing information are provided to the subjects at different steps. In all settings, the subjects were asked to determine whether the robot's action makes sense or not at a time. The minimally complete explanations are generated based on BID6 and online explanations are generated using approaches introduced above. At the end of the study, the subjects were provided the NASA Task Load standard questionnaire to evaluate the efficiency of different explanation approaches by NASA Task Load Index (TLX) BID23. The NASA TLX is a subjective workload assessment tool to evaluate human-machine interface systems. Mental workload is a multidimensional variable which can be captured by different variables and NASA TLX is one of the most frequently used subjective measurements for capturing different aspect of mental workload BID24. It calculates an overall mental workload score using a weighted average on sub-scales: mental demand, physical demand, temporal demand, performance, effort and frustration. Since our experiment does not involve physical demand, we did not include the corresponding question. The description of questions used for each category is presented as follows:• Mental Demand: How mentally demanding was the task? We created the academic survey using Qualtrics and recruited 150 human subjects on MTurk, 30 subjects for each setting. To improve the quality of the responses, we set the criteria that the worker's HIT acceptance rate must be greater than 98%. After sifting out invalid responses (i.e., failing to identify the two purposely inserted random actions), we had 94 valid responses in total: 19 for each of MCE-R and MCE, 20 for OEG-PP, and 18 for each of OEG-NA and OEG-AP. The age range of subjects was between 18 and 70, and 29.8% of the subjects were female. We examined how well the human subjects understand the robot's plan given the different explanations, and compared the distances across the five different settings. We compute the distance between the robot's plan and the human's expected plan by the ratio between the number of questionable actions and the total number of actions in a plan. The lower the distance value, the closer the human's plan is to the robot's plan. This metric intuitively captures how much the human subject understands the robot's plan. We calculated the averaged of each settings over all of the subjects participated in that setting, using subjective questions from The overall show that OEG approaches are able to better reduce the human's mental workload than MCE approaches. This is backed up by the fact that OEG approaches ed in better performance in almost all NASA TLX measures. Due to intertwining the explanation process with the plan execution, the OEG approaches create more temporal demand according to the experiment, which is expected. FIG6 presents both objective performance measures, and subjective of the human study amongst the 5 TLX categories. First, the number of questionable actions are significantly lower among OEG approaches when comparing to the MCEs. This indicates that the subjects had more trust towards robots in the OEG cases. Moreover, the accuracy of identifying the correct actions (questionable vs. non-questionable) among OEG approaches are higher. Between the three approaches, OEG-AP has the least number of questionable actions and the most accuracy. We have also presented the p-value for the mental load based on the subjective measures in TAB1 for all measures ranging from 0 to 100). The indicate a statistical significant difference between OEG approaches and MCEs for the mental workload in a pairwise comparison. The overall p-value across five categories is 0.0068 between OEGs (as a group) and MCEs (as a group).We also did some time analysis. The average overall time taken to accomplish the task for each of the categories is as follows: OEG-NA (567.44s) < OEG-AP (629.56s) < MCE-R (678.98s) < MCE (763.47s) < OEG-PP (775.65s), although we did not see a statistically significant difference due to large variances. The accuracy of the secondary task is also not significantly different between the various approaches. In this paper, we introduced a novel approach for explanation generation to reduce the mental workload needed for the human to interpret the explanations, throughout a humanrobot interaction scheme. The key idea here is to break down a complex explanation into smaller parts and convey them in an online fashion, while intertwined with the plan execution. We take a step further from our prior work by considering not only providing the correct explanations, but also the explanations that are easily understandable. We provided three different approaches each of which focuses on one aspect of explanation generation weaved in plan execution. This is an important step toward achieving explainable AI. We evaluated our approaches using both simulation and human subjects. Results showed that our approaches achieved better task performance while reducing the mental workload.

We show that a relatively simple black-box adversarial attack scheme using Bayesian optimization and dimension upsampling  is preferable to existing methods when the number of available queries is very low.

Represent the TLDR for finding the corresponding scientific article!

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.