Question: represent the article for finding a single-sentence tldr

Question: Significant strides have been made toward designing better generative models in recent years. Despite this progress, however, state-of-the-art approaches are still largely unable to capture complex global structure in data. For example, images of buildings typically contain spatial patterns such as windows repeating at regular intervals; state-of-the-art generative methods can’t easily reproduce these structures. We propose to address this problem by incorporating programs representing global structure into the generative model—e.g., a 2D for-loop may represent a configuration of windows. Furthermore, we propose a framework for learning these models by leveraging program synthesis to generate training data. On both synthetic and real-world data, we demonstrate that our approach is substantially better than the state-of-the-art at both generating and completing images that contain global structure. There has been much interest recently in generative models, following the introduction of both variational autoencoders (VAEs) BID13 and generative adversarial networks (GANs) BID6. These models have successfully been applied to a range of tasks, including image generation BID16, image completion BID10, texture synthesis BID12; BID22, sketch generation BID7, and music generation BID3.Despite their successes, generative models still have difficulty capturing global structure. For example, consider the image completion task in Figure 1. The original image (left) is of a building, for which the global structure is a 2D repeating pattern of windows. Given a partial image (middle left), the goal is to predict the completion of the image. As can be seen, a state-of-the-art image completion algorithm has trouble reconstructing the original image (right) BID10.1 Real-world data often contains such global structure, including repetitions, reflectional or rotational symmetry, or even more complex patterns. In the past few years, program synthesis Solar- BID17 has emerged as a promising approach to capturing patterns in data BID4; BID19. The idea is that simple programs can capture global structure that evades state-of-the-art deep neural networks. A key benefit of using program synthesis is that we can design the space of programs to capture different kinds of structure-e.g., repeating patterns BID5, symmetries, or spatial structure BID2 -depending on the application domain. The challenge is that for the most part, existing approaches have synthesized programs that operate directly over raw data. Since programs have difficulty operating over perceptual data, existing approaches have largely been limited to very simple data-e.g., detecting 2D repeating patterns of simple shapes BID5.We propose to address these shortcomings by synthesizing programs that represent the underlying structure of high-dimensional data. In particular, we decompose programs into two parts: (i) a sketch s ∈ S that represents the skeletal structure of the program BID17, with holes that are left unimplemented, and (ii) components c ∈ C that can be used to fill these holes. We consider perceptual components-i.e., holes in the sketch are filled with raw perceptual data. For example, the original image x * partial image x part completionx (ours) completionx (baseline) Figure 1: The task is to complete the partial image x part (middle left) into an image that is close to the original image x * (left). By incorporating programmatic structure into generative models, the completion (middle right) is able to substantially outperform a state-of-the-art baseline BID10 (right). Note that not all non-zero pixels in the sketch rendering retain the same value in the completed picture due to the nature of the following completion process program represents the structure in the original image x * in Figure 1 (left). The black text is the sketch, and the component is a sub-image taken from the given partial image. Then, the draw function renders the given sub-image at the given position. We call a sketch whose holes are filled with perceptual components a neurosymbolic program. Building on these ideas, we propose an approach called program-synthesis (guided) generative models (PS-GM) that combines neurosymbolic programs representing global structure with state-of-the-art deep generative models. By incorporating programmatic structure, PS-GM substantially improves the quality of these state-of-the-art models. As can be seen, the completion produced using PS-GM (middle right of Figure 1) substantially outperforms the baseline. We show that PS-GM can be used for both generation from scratch and for image completion. The generation pipeline is shown in FIG0. At a high level, PS-GM for generation operates in two phases:• First, it generates a program that represents the global structure in the image to be generated. In particular, it generates a program P = (s, c) representing the latent global structure in the image (left in FIG0, where s is a sketch (in the domain considered here, a list of 12 for-loop structures) and c is a perceptual component (in the domain considered here, a list of 12 sub-images).• Second, our algorithm executes P to obtain a structure rendering x struct representing the program as an image (middle of FIG0). Then, our algorithm uses a deep generative model to complete x struct into a full image (right of FIG0). The structure in x struct helps guide the deep generative model towards images that preserve the global structure. The image-completion pipeline (see Figure 3) is similar. Training these models end-to-end is challenging, since a priori, ground truth global structure is unavailable. Furthermore, representative global structure is very sparse, so approaches such as reinforcement learning do not scale. Instead, we leverage domain-specific program synthesis algorithms to produce examples of programs that represent global structure of the training data. In particular, we propose a synthesis algorithm tailored to the image domain, which extracts programs with nested for-loops that can represent multiple 2D repeating patterns in images. Then, we use these example programs as supervised training data. Our programs can capture rich spatial structure in the training data. For example, in FIG0, the program structure encodes a repeating structure of 0's and 2's on the whole image, and a separate repeating structure of 3's on the right-hand side of the image. Furthermore, in Figure 1, the generated image captures the idea that the repeating pattern of windows does not extend to the bottom portion of the image.for loop from sampled program P structure rendering x struct completed image x (ii) Our model executes P to obtain a rendering of the program structure x struct (middle). (iii) Our model samples a completion x ∼ p θ (x | s, c) of x struct into a full image (right).Contributions. We propose an architecture of generative models that incorporates programmatic structure, as well as an algorithm for training these models (Section 2). Our learning algorithm depends on a domain-specific program synthesis algorithm for extracting global structure from the training data; we propose such an algorithm for the image domain (Section 3). Finally, we evaluate our approach on synthetic data and on a real-world dataset of building facades Tyleček &Šára, both on the task of generation from scratch and on generation from a partial image. We show that our approach substantially outperforms several state-of-the-art deep generative models (Section 4).Related work. There has been growing interest in applying program synthesis to machine learning, for purposes of interpretability BID21; BID20, safety BID1, and lifelong learning BID19. Most relevantly, there has been interest in using programs to capture structure that deep learning models have difficulty representing; BID4;. For instance, BID4 proposes an unsupervised learning algorithm for capturing repeating patterns in simple line drawings; however, not only are their domains simple, but they can only handle a very small amount of noise. Similarly, BID5 captures 2D repeating patterns of simple circles and polygons; however, rather than synthesizing programs with perceptual components, they learn a simple mapping from images to symbols as a preprocessing step. The closest work we are aware of is BID19, which synthesizes programs with neural components (i.e., components implemented as neural networks); however, their application is to lifelong learning, not generation, and to learning with supervision (labels) rather than to unsupervised learning of structure. Additionally, there has been work extending neural module networks BID0 to generative models BID2. These algorithms essentially learn a collection of neural components that can be composed together based on hierarchical structure. However, they require that the structure be available (albeit in natural language form) both for training the model and for generating new images. Finally, there has been work incorporating spatial structure into generative models for generating textures BID12; however, their work only handles a single infinite repeating 2D pattern. In contrast, we can capture a rich variety of spatial patterns parameterized by a space of programs. For example, the image in Figure 1 generated by our technique contains different repeating patterns in different parts of the image. We describe our proposed architecture for generative models that incorporate programmatic structure. For most of this section, we focus on generation; we discuss how we adapt these techniques to image completion at the end. We illustrate our generation pipeline in FIG0.Let p θ,φ (x) be a distribution over a space X with unknown parameters θ, φ that we want to estimate. We study the setting where x is generated based on some latent structure, which consists of a program sketch s ∈ S and a perceptual component c ∈ C, and where the structure is in turn generated partial image x part synthesized program P part extrapolated programP structure renderingx struct+part completionx (ours) original image x * Figure 3: Our image completion pipeline consists of the following steps: (i) Given a partial image x part (top left), our program synthesis algorithm (Section 3) synthesizes a program P part representing the structure in the partial image (top middle).(ii) Our model f extrapolates P part to a program P = f (P part) representing the structure of the whole image. (iii) Our model executesP to obtain a rendering of the program structurex struct+part (bottom left). (iv) Our model completesx struct+part into an imagex (bottom middle), which resembles the original image x * (bottom right).conditioned on a latent vector z ∈ Z-i.e., To sample a completion, our model executes P to obtain a structure rendering x struct = eval(P) (middle), and then samples a completion based on x struct -i.e., DISPLAYFORM0 DISPLAYFORM1 We now describe our algorithm for learning the parameters θ, φ of p θ,φ, followed by a description of our choices of architecture for p φ (s, c | z) and p θ (x | s, c). DISPLAYFORM2 Since log p θ,φ (x) is intractable to optimize, we use an approach based on the variational autoencoder (VAE). In particular, we use a variational distribution qφ(s, c, z DISPLAYFORM3 which has parametersφ. Then, we optimizeφ while simultaneously optimizing θ, φ. Using qφ(s, c, z | x), the evidence lower bound on the log-likelihood is DISPLAYFORM4 where D KL is the KL divergence and H is information entropy. Thus, we can approximate θ *, φ * by optimizing the lower bound instead of log p θ,φ (x). However, remains intractable since we are integrating over all program sketches s ∈ S and perceptual components c ∈ C. Using sampling to estimate these integrals would be very computationally expensive. Instead, we propose an approach that uses a single point estimate of s x ∈ S and c x ∈ C for each x ∈ X, which we describe below. Synthesizing structure. For a given x ∈ X, we use program synthesis to infer a single likely choice s x ∈ S and c x ∈ C of the latent structure. The program synthesis algorithm must be tailored to a specific domain; we propose an algorithm for inferring for-loop structure in images in Section 3. Then, we use these point estimates in place of the integrals over S and C-i.e., we assume that DISPLAYFORM5 where δ is the Dirac delta function. Plugging into gives DISPLAYFORM6 where we have dropped the degenerate terms log δ(s − s x) and log δ(c − c x) (which are constant with respect to the parameters θ, φ,φ). As a consequence, decomposes into two parts that can be straightforwardly optimized-i.e., DISPLAYFORM7 where we can optimize θ and φ,φ independently: DISPLAYFORM8 Latent structure VAE. Note that L(φ,φ; x) is exactly equal to the objective of a VAE, where qφ(z | s, c) is the encoder and p φ (s, c | z) is the decoder-i.e., learning the distribution over latent structure is equivalent to learning the parameters of a VAE. The architecture of this VAE depends on the representation of s and c. In the case of for-loop structure in images, we use a sequence-to-sequence VAE.Generating data with structure. The term L(θ; x) corresponds to learning a probability distribution (conditioned on the latent structure s and c)-e.g., we can estimate this distribution using another VAE. As before, the architecture of this VAE depends on the representation of s and c. Rather than directly predicting x based on s and c, we can leverage the program structure more directly by first executing the program P = (s, c) to obtain its output x struct = eval(P), which we call a structure rendering. In particular, x struct is a more direct representation of the global structure represented by P, so it is often more suitable to use as input to a neural network. The middle of FIG0 shows an example of a structure rendering for the program on the left. Then, we can train a model DISPLAYFORM9 In the case of images, we use a VAE with convolutional layers for the encoder q φ and transpose convolutional layers for the decoder p θ. Furthermore, instead of estimating the entire distribution p θ (x | s, c), we also consider two non-probabilistic approaches that directly predict x from x struct, which is an image completion problem. We can solve this problem using GLCIC, a state-of-the-art image completion model BID10. We can also use , which solves the more general problem of mapping a training set of structured renderings {x struct} to a training set of completed images {x}. Image completion. In image completion, we are given a set of training pairs (x part, x *), and the goal is to learn a model that predicts the complete image x * given a partial image x part. Compared to generation, our likelihood is now conditioned on x part -i.e., p θ,φ (x | x part). Now, we describe how we modify each of our two models p θ (x | s, c) and p φ (s, c | z) to incorporate this extra information. First, the programmatic structure is no longer fully latent, since we can observe partial programmatic structure in x part. In particular, we can leverage our program synthesis algorithm to help perform completion. We first synthesize programs P * and P part representing the global structure in x * and x part, respectively. Then, we can train a model f that predicts P * given P part -i.e., it extrapolates P part to a programP = f (P part) representing the structure of the whole image. Thus, unlike generation, where we sample a programP = (s, c) ∼ p φ (s, c | z), we use the extrapolated programP = f (P part).The second model p θ (x | s, c) for the most part remains the same, except when we executeP = (s, c) to obtain a structure rendering x struct, we render onto the partial image x part instead of onto a blank image to obtain the final rendering x struct+part. Then, we complete the structure rendering x struct+part into a prediction of the full imagex as before (i.e., using a VAE, GLCIC, or CycleGAN).Our image completion pipeline is shown in Figure 3, including the given partial image (top left), the program P part synthesized from the partial image (top middle), the extrapolated programP (top right), the structure rendering x struct+part (bottom left), and the predicted full imagex (bottom middle). Image representation. Since the images we work with are very high dimensional, for tractability, we assume that each image x ∈ R N M ×N M is divided into a grid containing N rows and N columns, where each grid cell has size M × M pixels (where M ∈ N is a hyperparameter). For example, this grid structure is apparent in Figure 3 Program grammar. Given this structure, we consider programs that draw 2D repeating patterns of M × M sub-images on the grid. More precisely, we consider programs P = ((s 1, c 1),..., (s k, c k)) ∈ (S × C) k that are length k lists of pairs consisting of a sketch s ∈ S and a perceptual component c ∈ C; here, k ∈ N is a hyperparameter. 3 A sketch s ∈ S has form s = for (i, j) ∈ {1, ..., n} × {1, ..., n} do DISPLAYFORM0 where n, a, b, n, a, b ∈ N are undetermined parameters that must satisfy a · n + b ≤ N and a · n + b ≤ N, and where?? is a hole to be filled by a perceptual component, which is an M × M sub-image c ∈ R M ×M. 4 Then, upon executing the (i, j) iteration of the for-loop, the program renders sub-image I at position (t, u) = (a · i + b, a · j + b) in the N × N grid. Figure 3 (top middle) shows an example of a sketch s where its hole is filled with a sub-image c, and Figure 3 (bottom left) shows the image rendered upon executing P = (s, c). FIG0 shows another such example. Program synthesis problem. Given a training image x ∈ R N M ×N M, our program synthesis algorithm outputs the parameters n h, a h, b h, n h, a h, b h of each sketch s h in the program (for h ∈ [k]), along with a perceptual component c h to fill the hole in sketch s h. Together, these parameters define a program P = ((s 1, c 1),..., (s k, c k)).The goal is to synthesize a program that faithfully represents the global structure in x. We capture this structure using a boolean tensor B (x) ∈ {0, 1} N ×N ×N ×N, where DISPLAYFORM1 where ∈ R + is a hyperparameter, and d(I, I) is a distance metric between on the space of subimages. In our implementation, we use a weighted sum of earthmover's distance between the color histograms of I and I, and the number of SIFT correspondences between I and I.Algorithm 1 Synthesizes a program P representing the global structure of a given image DISPLAYFORM2.., k} do s h, c h = arg max (s,c)∈S×Ĉ (P h−1 ∪ {(s, c)}; x) P ← P ∪ {(s h, c h)} end for Output: P Additionally, we associate a boolean tensor with a given program P = ((s 1, c 1),..., (s k, c k)). First, for a sketch s ∈ S with parameters a, b, n, a, b, n, we define DISPLAYFORM3 i.e., the set of grid cells where sketch renders a sub-image upon execution. Then, we have DISPLAYFORM4 t,u,t,u indicates whether the sketch s renders a sub-image at both of the grid cells (t, u) and (t, u). Then, DISPLAYFORM5 where the disjunction of boolean tensors is defined elementwise. Intuitively, B (P) identifies the set of pairs of grid cells (t, u) and (t, u) that are equal in the image rendered upon executing each pair (s, c) in P. Finally, our program synthesis algorithm aims to solve the following optimization problem: DISPLAYFORM0 1, where ∧ and ¬ are applied elementwise, and λ ∈ R + is a hyperparameter. In other words, the objective of is the number of true positives (i.e., entries where B (P) = B (x) = 1), and the number of false negatives (i.e., entries where B (P) = B (x) = 0), and computes their weighted sum. Thus, the objective of measures for how well P represents the global structure of x. For tractability, we restrict the search space in to programs of the form P = ((s 1, c 1 DISPLAYFORM1 In other words, rather than searching over all possible sub-images c ∈ R M ×M, we only search over the sub-images that actually occur in the training image x. This may lead to a slightly sub-optimal solution, for example, in cases where the optimal sub-image to be rendered is in fact an interpolation between two similar but distinct sub-images in the training image. However, we found that in practice this simplifying assumption still produced viable . Program synthesis algorithm. Exactly optimizing is in general an NP-complete problem. Thus, our program synthesis algorithm uses a partially greedy heuristic. In particular, we initialize the program to P = ∅. Then, on each iteration, we enumerate all pairs (s, c) ∈ S ×Ĉ and determine the pair (s h, c h) that most increases the objective in, whereĈ is the set of all sub-images x tu for t, u ∈ [N]. Finally, we add (s h, c h) to P. We show the full algorithm in Algorithm 1. Table 2: Performance of our approach PS-GM versus the baseline (BL) for image completion. We report Fréchet distance for GANbased models, and negative log-likelihood (NLL) for the VED. We perform two experiments-one for generation from scratch and one for image completion. We find substantial improvement in both tasks. Details on neural network architectures are in Appendix A, and additional examples for image completion are in Appendix B. Synthetic dataset. We developed a synthetic dataset based on MNIST. Each image consists of a 9 × 9 grid, where each grid cell is 16 × 16 pixels. Each grid cell is either filled with a colored MNIST digit or a solid color . The program structure is a 2D repeating pattern of an MNIST digit; to add natural noise, we each iteration of the for-loop in a sketch s h renders different MNIST digits, but with the same MNIST label and color. Additionally, we chose the program structure to contain correlations characteristic of real-world images-e.g., correlations between different parts of the program, correlations between the program and the , and noise in renderings of the same component. Examples are shown in Figure 4. We give details of how we constructed this dataset in Appendix A. This dataset contains 10,000 training and 500 test images. Facades dataset. Our second dataset consists of 1855 images (1755 training, 100 testing) of building facades. 6 These images were all scaled to a size of 256 × 256 × 3 pixels, and were divided into a grid of 15 × 15 cells each of size 17 or 18 pixels. These images contain repeating patterns of objects such as windows and doors. Experimental setup. First, we evaluate our approach PS-GM applied to generation from scratch. We focus on the synthetic dataset-we found that our facades dataset was too small to produce meaningful . For the first stage of PS-GM (i.e., generating the program P = (s, c)), we use a LSTM architecture for the encoder p φ (s, c | z) and a feedforward architecture for the decoder qφ(z | s, c). As described in Section 2, we use Algorithm 1 to synthesize programs P x = (s x, c x) representing each training image x ∈ X train. Then, we train p φ and qφ on the training set of programs DISPLAYFORM0 For the second stage of PS-GM (i.e., completing the structure rendering x struct into an image x), we have applied a variational encoder-decoder (VED) architecture, in which the goal is to minimize the reconstruction error of the decoded image (trained on full images that had their representative program extracted). The encoder of the VAE architecture, q θ (w | x struct), maps x struct to a latent vector w; this model has a standard convolutional architecture with 4 layers. The decoder p θ (x | w) maps the latent vector to a whole image, and has a standard transpose convolutional architecture with 6 layers. We train p θ and q θ using the typical loss based on the VAE approach. We also trained a model based on Cycle-GAN which mapped renderings of programs to complete images. While it is difficult to compare objectively to a VAE architecture, it appeared to be at least as capable of Original Images DISPLAYFORM1 Figure 4: Examples of synthetic images generated using our approach, PS-GM (with VED and CycleGan), and the baseline (a VAE and a SpatialGAN). Images in different rows are unrelated since the task is generation from scratch.generating structured images, and the structure in the PS-CycleGAN images were more apparent than in the PS-VAE due to less blurriness. We compare our encoder-decoder approach to a baseline consisting of a vanilla VAE. The architecture of the vanilla VAE is the same as the architecture of the VED we used for the second stage of PS-GM, except the input to the encoder is the original training image x instead of the structure rendering x struct.Results. We measure performance for PS-GM with the VED and the baseline VAE using the variational lower bound on the negative log-likelihood (NLL) BID23 on a held-out test set. For our approach, we use the lower bound, 7 which is the sum of the NLLs of the first and second stages; we report these NLLs separately as well. Figure 4 in Appendix B shows examples of generated images. For PS-GM and SpatialGAN, we use Fréchet inception distance BID8. Table 1 shows these metrics of both our approach and the baseline. Discussion. The models based on our approach quantitatively improve over the respective baselines. The examples of images generated using our approach with VED completion appear to contain more structure than those generated using the baseline VAE. Similarly, the images generated using our approach with CycleGAN clearly capture more complex structure than the unbounded 2D repeating texture patterns captured by SpatialGAN. Experimental setup. Second, we evaluated our approach PS-GM for image completion, on both our synthetic and the facades dataset. For this task, we compare using three image completion models: , , and the VED architecture described in Section 4.2. GLCIC is a state-of-the-art image completion model. CycleGAN is a generic imageto-image transformer. It uses unpaired training data, but we found that for our task, it outperforms approaches such as Pix2Pix BID11 that take paired training data. For each model, we trained two versions:• Our approach (PS-GM): As described in Section 2 (for image completion), given a partial image x part, we use Algorithm 1 to synthesize a program P part. We extrapolate P part toOriginal Image (Synthetic) Original Image (Facades)PS-GM (GLCIC, Synthetic) PS-GM (GLCIC, Facades)Baseline (GLCIC, Synthetic) Baseline (GLCIC, Facades)Figure 5: Examples of images generated using our approach (PS-GM) and the baseline, using GLCIC for image completion. DISPLAYFORM0, and executeP to obtain a structure rendering x struct. Finally, we train the image completion model (GLCIC, CycleGAN, or VED) to complete x struct to the original image x *.• Baseline: Given a partial image x part, we train the image completion model (GLCIC, CycleGAN, or VED) to directly complete x part to the original image x *.Results. As in Section 4.2, we measure performance using Fréchet inception distance for GLCIC and CycleGAN, and negative log-likelihood (NLL) to evaluate the VED, reported on a held-out test set. We show these in Table 2. We show examples of completed image using GLCIC in Figure 5. We show additional examples of completed images including those completed using CycleGAN and VED in Appendix B.Discussion. Our approach PS-GM outperforms the baseline in every case except the VED on the facades dataset. We believe the last is since both VEDs failed to learn any meaningful structure (see Figure 7 in Appendix B).A key reason why the baselines perform so poorly on the facades dataset is that the dataset is very small. Nevertheless, even on the synthetic dataset (which is fairly large), PS-GM substantially outperforms the baselines. Finally, generative models such as GLCIC are known to perform poorly far away from the edges of the provided partial image BID10. A benefit of our approach is that it provides the global context for a deep-learning based image completion model such as GLCIC to perform local completion. We have proposed a new approach to generation that incorporates programmatic structure into state-ofthe-art deep learning models. In our experiments, we have demonstrated the promise of our approach to improve generation of high-dimensional data with global structure that current state-of-the-art deep generative models have difficulty capturing. We leave a number of directions for future work. Most importantly, we have relied on a custom synthesis algorithm to eliminate the need for learning latent program structure. Learning to synthesize latent structure during training is an important direction for future work. In addition, future work will explore more expressive programmatic structures, including if-then-else statements. A EXPERIMENTAL DETAILS To sample a random image, we started with a 9×9 grid, where each grid cell is 16×16 pixels. We randomly sample a program P = ((s 1, c 1),..., (s k, c k)) (for k = 12), where each perceptual component c is a randomly selected MNIST image (downscaled to our grid cell size and colorized). To create correlations between different parts of P, we sample (s h, c h) depending on (s 1, c 1),..., (s h−1, c h−1). First, to sample each component c h, we first sample latent properties of c h (i.e., its MNIST label {0, 1, ..., 4} and its color {red, blue, orange, green, yellow}). Second, we sample the parameters of s h conditional on these properties. To each of the 25 possible latent properties of c h, we associate a discrete distribution over latent properties for later elements in the sequence, as well as a mean and standard deviation for each of the parameters of the corresponding sketch s h.We then render P by executing each (s h, c h) in sequence. However, when executing (s h, c h), on each iteration (i, j) of the for-loop, instead of rendering the sub-image c h at each position in the grid, we randomly sample another MNIST image c DISPLAYFORM0 with the same label as c h, recolor c DISPLAYFORM1 to be the same color as c h, and render c DISPLAYFORM2. By doing so, we introduce noise into the programmatic structure. PS-GM architecture. For the first stage of PS-GM (i.e., generating the program P = (s, c)), we use a 3-layer LSTM encoder p φ (s, c | z) and a feedforward decoder qφ(z | s, c). The LSTM includes sequences of 13-dimensional vectors, of which 6 dimensions represent the structure of the for-loop being generated, and 7 dimensions are an encoding of the image to be rendered. The image compression was performed via a convolutional architecture with 2 convolutional layers for encoding and 3 deconvolutional layers for decoding. For the second stage of PS-GM (i.e., completing the structure rendering x struct into an image x), we use a VED; the encoder q θ (w | x struct) is a CNN with 4 layers, and the decoder p θ (x | w) is a transpose CNN with 6 layers. The CycleGAN model has a discriminator with 3 convolutional layers and a generator which uses transfer learning by employing the pre-trained ResNet architecture. Baseline architecture. The architecture of the baseline is a vanilla VAE with the same as the architecture as the VED we used for the second state of PS-GM, except the input to the encoder is the original training image x instead of the structure rendering x struct. The baselines with CycleGAN also use the same architecture as PS-GM with CycleGAN/GLCIC. The Spatial GAN was trained with 5 layers each in the generative/discriminative layer, and 60-dimensional global and 3-dimensional periodic latent vectors. PS-GM architecture. For the first stage of PS-GM for completion (extrapolation of the program from a partial image to a full image), we use a feedforward network with three layers. For the second stage of completion via VAE, we use a convolutional/deconvolutional architecture. The encoder is a CNN with 4 layers, and the decoder is a transpose CNN with 6 layers. As was the case in generation, the CycleGAN model has a discriminator with 3 convolutional layers and a generator which uses transfer learning by employing the pre-trained ResNet architecture. Baseline architecture. For the baseline VAE architecture, we used a similar architecture to the PS-GM completion step (4 convolutional and 6 deconvolutional layers). The only difference was the input, which was a partial image rather than an image rendered with structure. The CycleGAN architecture was similar to that used in PS-GM (although it mapped partial images to full images rather than partial images with structure to full images). In Figure 6, we show examples of how our image completion pipeline is applied to the facades dataset, and in Figure 7, we show examples of how our image completion pipeline is applied to our synthetic dataset.

Represent the TLDR for finding the corresponding scientific article

Applying program synthesis to the tasks of image completion and generation within a deep learning framework

represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR!

Question: Learning control policies in robotic tasks requires a large number of interactions due to small learning rates, bounds on the updates or unknown constraints. In contrast humans can infer protective and safe solutions after a single failure or unexpected observation. In order to reach similar performance, we developed a hierarchical Bayesian optimization algorithm that replicates the cognitive inference and memorization process for avoiding failures in motor control tasks. A Gaussian Process implements the modeling and the sampling of the acquisition function. This enables rapid learning with large learning rates while a mental replay phase ensures that policy regions that led to failures are inhibited during the sampling process. The features of the hierarchical Bayesian optimization method are evaluated in a simulated and physiological humanoid postural balancing task. We quantitatively compare the human learning performance to our learning approach by evaluating the deviations of the center of mass during training. Our show that we can reproduce the efficient learning of human subjects in postural control tasks which provides a testable model for future physiological motor control tasks. In these postural control tasks, our method outperforms standard Bayesian Optimization in the number of interactions to solve the task, in the computational demands and in the frequency of observed failures. Autonomous systems such as anthropomorphic robots or self-driving cars must not harm cooperating humans in co-worker scenarios, pedestrians on the road or them selves. To ensure safe interactions with the environment state-of-the-art robot learning approaches are first applied to simulations and afterwards an expert selects final candidate policies to be run on the real system. However, for most autonomous systems a fine-tuning phase on the real system is unavoidable to compensate for unmodelled dynamics, motor noise or uncertainties in the hardware fabrication. Several strategies were proposed to ensure safe policy exploration. In special tasks like in robot arm manipulation the operational space can be constrained, for example, in classical null-space control approaches;;;;; or constraint black-box optimizer;;;;. While this null-space strategy works in controlled environments like research labs where the environmental conditions do not change, it fails in everyday life tasks as in humanoid balancing where the priorities or constraints that lead to hardware damages when falling are unknown. Alternatively, limiting the policy updates by applying probabilistic bounds in the robot configuration or motor command space;;;; were proposed. These techniques do not assume knowledge about constraints. Closely related are also Bayesian optimization techniques with modulated acquisition functions;;; to avoid exploring policies that might lead to failures. However, all these approaches do not avoid failures but rather an expert interrupts the learning process when it anticipates a potential dangerous situation. Figure 1: Illustration of the hierarchical BO algorithm. In standard BO (clock-wise arrow), a mapping from policy parameters to rewards is learned, i.e., φ → r ∈ R 1. We propose a hierarchical process, where first features κ are sampled and later used to predict the potential of policies conditioned on these features, φ|κ → r. The red dots show the first five successive roll-outs in the feature and the policy space of a humanoid postural control task. All the aforementioned strategies cannot avoid harming the system itself or the environment without thorough experts knowledge, controlled environmental conditions or human interventions. As humans require just few trials to perform reasonably well, it is desired to enable robots to reach similar performance even for high-dimensional problems. Thereby, most approaches are based on the assumption of a "low effective dimensionality", thus most dimensions of a high-dimensional problem do not change the objective function significantly. a method for relevant variable selection based on Hierarchical Diagonal Sampling for both, variable selection and function optimization, has been proposed. Randomization combined with Bayesian Optimization is proposed in to exploit effectively the aforementioned "low effective dimensionality". a dropout algorithm has been introduced to overcome the high-dimensionality problem by only train onto a subset of variables in each iteration, evaluating a "regret gap" and providing strategies to reduce this gap efficiently. an algorithm has been proposed which optimizes an acquisition function by building new Gaussian Processes with sufficiently large kernellengths scales. This ensures significant gradient updates in the acquisition function to be able to use gradient-dependent methods for optimization. The contribution of this paper is a computational model for psychological motor control experiments based on hierarchical acquisition functions in Bayesian Optimization (HiBO). Our motor skill learning method uses features for optimization to significantly reduce the number of required roll-outs. In the feature space, we search for the optimum of the acquisition function by sampling and later use the best feature configuration to optimize the policy parameters which are conditioned on the given features, see also Figure 1. In postural control experiments, we show that our approach reduces the number of required roll-outs significantly compared to standard Bayesian Optimization. The focus of this study is to develop a testable model for psychological motor control experiments where well known postural control features could be used. These features are listed in Table 3. In future work we will extend our model to autonomous feature learning and will validate the approach in more challenging robotic tasks where'good' features are hard to hand-craft. In this section we introduce the methodology of our hierarchical BO approach. We start with the general problem statement and afterwards briefly summarize the concept of BO which we use here as a baseline. We then go through the basic principles required for our algorithm and finally we explain mental replay. The goal in contextual policy search is to find the best policy π * (θ|c) that maximizes the return with reward r t (x t, u t) at time step t for executing the motor commands u t in state x t. For learning the policy vector and the context, we collect samples of the return J(θ [k] ) ∈ R 1, the evaluated policy parameter vector θ [k] ∈ R m and the observed contextual features modeled by c [k] ∈ R n. All variables used are summarized in Table 1. The optimization problem is defined as The optimal parameter vector and the corresponding context vector can be found in an hierarchical optimization process which is discussed in Section 2.3. Bayesian Optimization (BO) has emerged as a powerful tool to solve various global optimization problems where roll-outs are expensive and a sufficient accurate solution has to be found with only few evaluations, e.g.;;. In this paper we use the standard BO as a benchmark for our proposed hierarchical process. Therefore, we now briefly summarize the main concept. For further details refer to. The main concept of Bayesian Optimization is to build a model for a given system based on the so far observed data D = {X, y}. The model describes a transformation from a given data point x ∈ X to a scalar value y ∈ y, e.g. from the parameter vector θ to the return J(θ). Such model can either be parametrized or non-parametrized and is used for choosing the next query point by evaluating an acquisition function α(D). Here, we use the non-parametric Gaussian Processes (GPs) for modeling the unknown system which are state-of-the-art model learning or regression approaches Williams & Rasmussen (1996; 2006) that were successfully used for learning inverse dynamics models in robotic applications;. For comprehensive discussions we refer to;. GPs represent a distribution over a partial observed system in the form of where D = {X, y} are the so far observed data points and D * = {X *, y *} the query points. This representation is fully defined by the mean m and the covariance K. We chose m(x) = 0 as mean {0, 1} flag indicating the success of rollout k function and as covariance function a Matérn kernel Matérn. It is a generalization of the squared-exponential kernel that has an additional parameter ν which controls the smoothness of the ing function. The smoothing parameter can be beneficial for learning local models. We used Matérn kernels with ν = 5 / 2 and the gamma function Γ, A = (2 √ ν||x p − x q ||) /l and modified Bessel function H ν. The length-scale parameter of the kernel is denoted by σ, the variance of the latent function is denoted by σ y and δ pq is the Kronecker delta function (which is one if p = q and zero otherwise). Note that for ν = 1/2 the Matérn kernel implements the squared-exponential kernel. In our experiments, we optimized the hyperparameters θ = [σ, l, σ y] by maximizing the marginal likelihood. Predictions for a query points D * = {x *, y *} can then be determined as The predictions are then used for choosing the next model evaluation point x n based on the acquisition function α(x; D). We use expected improvement (EI) which considers the amount of improvement where τ is the so far best measured value max(y), Φ the standard normal cumulative distribution function, φ the standard normal probability density function and ξ ∼ σ ξ U (−0.5, 0.5) a random value to ensure a more robust exploration. Samples, distributed over the area of interest, are evaluated and the best point is chosen for evaluation based on the acquisition function values. We learn a joint distribution p(J(θ .., K roll-outs of observed triples. This distribution is used for as a prior of the acquisition function in Bayesian optimization. However, instead of directly conditioning on the most promising policy vectors using α BO = α(θ; D), we propose an iterative conditioning scheme. Therefore, the two acquisition functions are employed, where for Equation, the evaluated mean µ(θ; c) and variance σ(θ; c) for the parameter θ are conditioned on the features c. The hierarchical optimization process works then as follows: In the first step we estimate the best feature values based on a GP model using the acquisition function from Equation These feature values are then used to condition the search for the best new parameter θ [k+1] using Equation θ We subsequently continue evaluating the policy vector θ [k+1] using the reward function presented in Equation To ensure robustness for Bayesian Optimization, mental replays can be generated. Therefore, the new training data set J(θ, generated by the policy parameter θ [k+1], will be enlarged by augmenting perturbed copies of the policy parameter θ [k+1]. These l copies are then used for generating the augmented training data sets Here, the transcript · l denotes l perturbed copies of the given set. Hence, perturbed copies of the parameters θ [k+1] and features c [k+1] are generated keeping the objective J(θ [k+1] ) constant. In Algorithm the complete method is summarized. We evaluate different replay strategies in the Section in 3.3. In this section we first present observations on human learning during perturbed squat-to-stand movements. We compare the learning of a simulated humanoid to the learning rates achieved by the human participants. Second, we evaluate our hierarchical BO approach in comparison to our baseline, the standard BO. Third we evaluate the impact of mental replays on the performance of our algorithm. To observe human learning, we designed an experiment where 20 male participants were subjected to waist pull perturbation during squat-to-stand movements, see Figure 2 (a). Participants had to stand up from a squat position without making any compensatory steps (if they made a step, such trial was considered a fail). Backward perturbation to the centre of mass (CoM) was applied by a pulling mechanism and was dependent on participants' mass and vertical CoM velocity. On average, participants required 6 trials (σ human = 3.1) to successfully complete the motion. On the left panel of Figure 3, a histogram of the required trials before the first success is shown. On the right panel, the evaluation for the simulated humanoid are presented (details on the implementation are discussed in the subsequent Subsection 3.2). The human learning behavior is faster and more reliable than the learning behavior of the humanoid. However, humans can exploit fundamental knowledge about whole body balancing whereas our humanoid has to learn everything from scratch. Only the gravity constant was set to zero in our simulation, as we are only interested in the motor adaptation and not in gravity compensation strategies. Adaptation was evaluated using a measure based on the trajectory area (TA) at every episode as defined in?. The Trajectory area represents the total deviation of the CoM trajectory with respect to a straight line. The trajectory area of a given perturbed trajectory is defined as the time integral of the distance of the trajectory points to the straight line in the sagittal plane: A positive sign represents the anterior direction while a negative sign represents the posterior direction. The mean and standard deviation for the trajectory area over the number of training episodes for all participants are depicted in Figure 4. Comparing these with the simulation of our humanoid shows that the learning rate using our approach is similar to the learning rate of real humans. Figure 4: Mean and standard deviaton of the trajectory area (T A) with regard to the number of episodes for both, the human experiments and the simulated humanoid. For the humanoid the xcoordinates have been shifted about −0.5 to account for the stretched arms. In addition, the trajectory area of the humanoid has been scaled with the factor 0.1 and shifted about −0.2 to allow easier comparison. To test the proposed algorithm we simulated a humanoid postural control task as shown in Figure 2(b). The simulated humanoid has to stand up and is thereby exposed to an external pertubation proportional to the velocity of the CoM in the superior direction in the sagittal plane. The pertubation is applied during standing up motion such that the robot has to learn to counter balance. The simulated humanoid consist of four joints, connected by rigid links, where the position of the first joint is fixed onto the ground. A PD-controller is used with K P,i and K D,i for i = 1, 2, 3, 4 being the proportional and derivative gains. In our simulations the gains are set to K P,i = 400 and K D,i = 20 and an additive control noise ∼ N has been inserted such that the control input for a certain joint becomes where e P,i, e D,i are the joint errors regarding the target position and velocity. The control gains can also be learned. The goal positions and velocities for the joints are given. As parametrized policy, we use a via point [φ i,φ i], where φ i is the position of joint i at time t via andφ i the corresponding velocity. Hence, the policy is based on 9, respectively 17 parameters (if the gains are learned), which are summarized in Table 2. For our simulations we handcrafted 7 features, namely the overall success, the maximum deviation of the CoM in x and y direction and the velocities of the CoM for the x and y directions at 200 ms respectively 400 ms. In Table 3 the features used in this paper are summarized. Simultaneously learning of the features is out of scope of this comparison to human motor performance but part of future work. We simulated the humanoid in each run for a maximum of t max = 2 s with a simulation time step of dt = 0.002 s, such that a maximum of N = 1000 simulation steps are used. The simulation has been stopped at the simulation step N end if either the stand up has been failed or the maximum simulation time has been reached. The return of a roll-out J(θ) is composed according to J(θ) = −(c balance + c time + c control) with the balancing costs c balance = 1/N end the time dependent costs c time = (N − N end) and control costs of c control = 10 ij. We compared our approach with our baseline, standard Bayesian Optimization. For that we used the features 4, 5 in 3 and set the number of mental replays to l = 3. We initialized both, the BO and the HiBO approach with 3 seed points and generated average statistics over 20 runs. In Figure 5 the comparison between the rewards of the algorithms over 50 episodes is shown. In Figure 6 (a) the number of successful episodes is illustrated. Our approach requires significantly fewer episodes to improve the reward then standard Bayesian Optimization (10 ± 3 vs 45 ± 5) and has a higher success quote (78% ± 24% vs 60% ± 7%). We further evaluated the impact of the different features on the learning behavior. In Figure 6 (b) the average statistics over 20 runs for different selected features with 3 mental replays are shown. All feature pairs generate better on average then standard BO, whereas for the evaluated task no significant difference in the feature choice was observed. We evaluated our approach with additional experience replays. For that we included an additive noise of rep ∼ N (0, 0.05) to perturb the policy parameters and features. In Figure 6 (c) average statistics over 20 runs of the success rates for different number of replay episodes are shown (rep3 = 3 replay episodes). Our proposed algorithm works best with a number of 3 replay episodes. Five or more replays in every iteration steps even reduce the success rate of the algorithm. We introduced HiBO, a hierarchical approach for Bayesian Optimization. We showed that HiBO outperforms standard BO in a complex humanoid postural control task. Moreover, we demonstrated the effects of the choice of the features and for different number of mental replay episodes. We compared our to the learning performance of real humans at the same task. We found that the learning behavior is similar. We found that our proposed hierarchical BO algorithm can reproduce the rapid motor adaptation of human subjects. In contrast standard BO, our comparison method, is about four times slower. In future work, we will examine the problem of simultaneously learning task relevant features in neural nets.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper presents a computational model for efficient human postural control adaptation based on hierarchical acquisition functions with well-known features.

Represent the short summary for finding the corresponding scientific article (long).

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Deep reinforcement learning has achieved great success in many previously difficult reinforcement learning tasks, yet recent studies show that deep RL agents are also unavoidably susceptible to adversarial perturbations, similar to deep neural networks in classification tasks. Prior works mostly focus on model-free adversarial attacks and agents with discrete actions. In this work, we study the problem of continuous control agents in deep RL with adversarial attacks and propose the first two-step algorithm based on learned model dynamics. Extensive experiments on various MuJoCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that our proposed framework is much more effective and efficient than model-free based attacks baselines in degrading agent performance as well as driving agents to unsafe states. Deep reinforcement learning (RL) has revolutionized the fields of AI and machine learning over the last decade. The introduction of deep learning has achieved unprecedented success in solving many problems that were intractable in the field of RL, such as playing Atari games from pixels and performing robotic control tasks (; ;). Unfortunately, similar to the case of deep neural network classifiers with adversarial examples, recent studies show that deep RL agents are also vulnerable to adversarial attacks. A commonly-used threat model allows the adversary to manipulate the agent's observations at every time step, where the goal of the adversary is to decrease the agent's total accumulated reward. As a pioneering work in this field, show that by leveraging the FGSM attack on each time frame, an agent's average reward can be significantly decreased with small input adversarial perturbations in five Atari games. further improve the efficiency of the attack in by leveraging heuristics of detecting a good time to attack and luring agents to bad states with sample-based Monte-Carlo planning on a trained generative video prediction model. Since the agents have discrete actions in Atari games , the problem of attacking Atari agents often reduces to the problem of finding adversarial examples on image classifiers, also pointed out in , where the adversaries intend to craft the input perturbations that would drive agent's new action to deviate from its nominal action. However, for agents with continuous actions, the above strategies can not be directly applied. Recently, studied the problem of adversarial testing for continuous control domains in a similar but slightly different setting. Their goal was to efficiently and effectively find catastrophic failure given a trained agent and to predict its failure probability. The key to success in is the availability of agent training history. However, such information may not always be accessible to the users, analysts, and adversaries. Therefore, in this paper we study the robustness of deep RL agents in a more challenging setting where the agent has continuous actions and its training history is not available. We consider the threat models where the adversary is allowed to manipulate an agent's observations or actions with small perturbations, and we propose a two-step algorithmic framework to find efficient adversarial attacks based on learned dynamics models. Experimental show that our proposed modelbased attack can successfully degrade agent performance and is also more effective and efficient than model-free attacks baselines. The contributions of this paper are the following: Figure 1: Two commonly-used threat models. • To the best of our knowledge, we propose the first model-based attack on deep RL agents with continuous actions. Our proposed attack algorithm is a general two-step algorithm and can be directly applied to the two commonly-used threat models (observation manipulation and action manipulation). • We study the efficiency and effectiveness of our proposed model-based attack with modelfree attack baselines based on random searches and heuristics (rand-U, rand-B, flip, see Section 4). We show that our model-based attack can degrade agent performance more significantly and efficiently than model-free attacks, which remain ineffective in numerous MuJoCo domains ranging from Cartpole, Fish, Walker, and Humanoid. Adversarial attacks in reinforcement learning. Compared to the rich literature of adversarial examples in image classifications and other applications (including natural language processing , speech , etc), there is relatively little prior work studying adversarial examples in deep RL. One of the first several works in this field are and , where both works focus on deep RL agent in Atari games with pixels-based inputs and discrete actions. In addition, both works assume the agent to be attacked has accurate policy and the problem of finding adversarial perturbation of visual input reduces to the same problem of finding adversarial examples on image classifiers. Hence, applied FGSM (to find adversarial perturbations and further improved the efficiency of the attack by heuristics of observing a good timing to attack -when there is a large gap in agents action preference between most-likely and leastlikely action. In a similar direction, study the problem of adversarial testing by leveraging rejection sampling and the agent training histories. With the availability of training histories, successfully uncover bad initial states with much fewer samples compared to conventional Monte-Carlo sampling techniques. Recent work by consider an alternative setting where the agent is attacked by another agent (known as adversarial policy), which is different from the two threat models considered in this paper. Finally, besides adversarial attacks in deep RL, a recent work study verification of deep RL agent under attacks, which is beyond the scope of this paper. Learning dynamics models. Model-based RL methods first acquire a predictive model of the environment dynamics, and then use that model to make decisions . These model-based methods tend to be more sample efficient than their model-free counterparts, and the learned dynamics models can be useful across different tasks. Various works have focused on the most effective ways to learn and utilize dynamics models for planning in RL (; ; ;). In this section, we first describe the problem setup and the two threat models considered in this paper. Next, we present an algorithmic framework to rigorously design adversarial attacks on deep RL agents with continuous actions. Let s i ∈ R N and a i ∈ R M be the observation vector and action vector at time step i, and let π: R N → R M be the deterministic policy (agent). Let f: R N × R M → R N be the dynamics model of the system (environment) which takes current state-action pair (s i, a i) as inputs and outputs the next state s i+1. We are now in the role of an adversary, and as an adversary, our goal is to drive the agent to the (un-safe) target states s target within the budget constraints. We can formulate this goal into two optimization problems, as we will illustrate shortly below. Within this formalism, we can consider two threat models: Threat model (i): Observation manipulation. For the threat model of observation manipulation, an adversary is allowed to manipulate the observation s i that the agent perceived within an budget: where ∆s i ∈ R N is the crafted perturbation and U s ∈ R N, L s ∈ R N are the observation limits. Threat model (ii): Action manipulation. For the threat model of action manipulation, an adversary can craft ∆a i ∈ R M such that M are the limits of agent's actions. Our formulations. Given an initial state s 0 and a pre-trained policy π, our (adversary) objective is to minimize the total distance of each state s i to the pre-defined target state s target up to the unrolled (planning) steps T. This can be written as the following optimization problems in Equations 3 and 4 for the Threat model (i) and (ii) respectively: A common choice of d(x, y) is the squared 2 distance x − y 2 2 and f is the learned dynamics model of the system, and T is the unrolled (planning) length using the dynamics models. In this section, we propose a two-step algorithm to solve Equations 3 and 4. The core of our proposal consists of two important steps: learn a dynamics model f of the environment and deploy optimization technique to solve Equations 3 and 4. We first discuss the details of each factor, and then present the full algorithm by the end of this section. Step 1: learn a good dynamics model f. Ideally, if f is the exact (perfect) dynamics model of the environment and assuming we have an optimization oracle to solve Equations 3 and 4, then the solutions are indeed the optimal adversarial perturbations that give the minimal total loss with -budget constraints. Thus, learning a good dynamics model can conceptually help on developing a strong attack. Depending on the environment, different forms of f can be applied. For example, if the environment of concerned is close to a linear system, then we could let f (s, a) = As + Bu, where A and B are unknown matrices to be learned from the sample trajectories (s i, a i, s i+1) pairs. For a more complex environment, we could decide if we still want to use a simple linear model (the next state prediction may be far deviate from the true next state and thus the learned dynamical model is less useful) or instead switch to a non-linear model, e.g. neural networks, which usually has better prediction power but may require more training samples. For either case, the model parameters A, B or neural network parameters can be learned via standard supervised learning with the sample trajectories pairs (s i, a i, s i+1). Step 2: solve Equations 3 and 4. Once we learned a dynamical model f, the next immediate task is to solve Equation 3 and 4 to compute the adversarial perturbations of observations/actions. When the planning (unrolled) length T > 1, Equation 3 usually can not be directly solved by off-theshelf convex optimization toolbox since the deel RL policy π is usually a non-linear and non-convex neural network. Fortunately, we can incorporate the two equality constraints of Equation 3 into the objective and with the remaining -budget constraint (Equation 1), Equation 3 can be solved via projected gradient descent (PGD) 1. Similarly, Equation 4 can be solved via PGD to get ∆a i. We note that, similar to the n-step model predictive control, our algorithm could use a much larger planning (unrolled) length T when solving Equations 3 and 4 and then only apply the first n (≤ T) adversarial perturbations on the agent over n time steps. Besides, with the PGD framework, f is not limited to feed-forward neural networks. Our proposed attack is summarized in Algorithm 2 for Step 1, and Algorithm 3 for Step 2. Algorithm 1 Collect trajectories 1: Input: pre-trained policy π, MaxSampleSize n s, environment env 2: Output: a set of trajectory pairs k ← k + 1 10: end while 11: Return S Algorithm 2 learn dynamics 1: Input: pre-trained policy π, MaxSampleSize n s, environment env, trainable parameters W 2: Output: learned dynamical model f (s, a; W) 3: S agent ← Collect trajectories(π, n s, env) 4: S random ← Collect trajectories(random policy, n s, env) 5: f (s, a; W) ← supervised learning algorithm(S agent ∪ S random, W) 6: Return f (s, a; W) Algorithm 3 model based attack 1: Input: pre-trained policy π, learned dynamical model f (s, a; W), threat model, maximum perturbation magnitude, unroll length T, apply perturbation length n (≤ T) 2: Output: a sequence of perturbation δ 1,..., δ n 3: if threat model is observation manipulation (Eq. 1) then In this section, we conduct experiments on standard reinforcement learning environment for continuous control . We demonstrate on 4 different environments in and corresponding tasks: Cartpole-balance/swingup, Fish-upright, Walkerstand/walk and Humanoid-stand/walk. For the deep RL agent, we train a state-of-the-art D4PG Evaluations. We conduct experiments for 10 different runs, where the environment is reset to different initial states in different runs. For each run, we attack the agent for one episode with 1000 time steps (the default time intervals is usually 10 ms) and we compute the total loss and total return reward. The total loss calculates the total distance of current state to the unsafe states and the total return reward measures the true accumulative reward from the environment based on agent's action. Hence, the attack algorithm is stronger if the total return reward and the total loss are smaller. Baselines. We compare our algorithm with the following model-free attack baselines with random searches and heuristics: • rand-U: generate m randomly perturbed trajectories from Uniform distribution with interval [−,] and return the trajectory with the smallest loss (or reward), • rand-B: generate m randomly perturbed trajectories from Bernoulli distribution with probability 1/2 and interval [−,], and return the trajectory with the smallest loss (or reward), • flip: generate perturbations by flipping agent's observations/actions within the budget in ∞ norm. For rand-U and rand-B, they are similar to Monte-Carlo sampling methods, where we generate m sample trajectories from random noises and report the loss/reward of the best trajectory (with minimum loss or reward among all the trajectories). We set m = 1000 throughout the experiments. Our algorithm. A 4-layer feed-forward neural network with 1000 hidden neurons per layer is trained as the dynamics model f respectively for the domains of Cartpole, Fish, Walker and Humanoid. We use standard 2 loss (without regularization) to learn a dynamics model f. Instead of using recurrent neural network to represent f, we found that the 1-step prediction for dynamics with the 4-layer feed-forward network is already good for the MuJoCo domains we are studying. Specifically, for the Cartpole and Fish, we found that 1000 episodes (1e6 training points) are sufficient to train a good dynamics model (the mean square error for both training and test losses are at the order of 10 −5 for Cartpole and 10 −2 for Fish), while for the more complicated domain like Walker and Humanoid, more training points (5e6) are required to achieve a low test MSE error (at the order of 10 −1 and 10 0 for Walker and Humanoid respectively). Consequently, we use larger planning (unrolled) length for Cartpole and Fish (e.g. T = 10, 20), while a smaller T (e.g. 3 or 5) is used for Walker and Humanoid. Meanwhile, we focus on applying projected gradient descent (PGD) to solve Equation 3 and 4. We use Adam as the optimizer with optimization steps equal to 30 and we report the best for each run from a combination of 6 learning rates, 2 unroll length {T 1, T 2} and n steps of applying PGD solution with n ≤ T i. For observation manipulation, we report the on Walker, Humanoid and Cartpole domains with tasks (stand, walk, balance, swingup) respectively. The unsafe states s target for Walker and Humanoid are set to be zero head height, targeting the situation of falling down. For Cartpole, the unsafe states are set to have 180 • pole angle, corresponding to the cartpole not swinging up and nor balanced. For the Fish domain, the unsafe states for the upright task target the pose of swimming fish to be not upright, e.g. zero projection on the z-axis. The full of both two threat models on observation manipulation and action manipulation are shown in Table 1a, b and c, d respectively. Since the loss is defined as the distance to the target (unsafe) state, the lower the loss, the stronger the attack. It is clear that our proposed attack achieves much lower loss in Table 1a & c than the other three model-free baselines, and the averaged ratio is also listed in 1b & d. Notably, over the 10 runs, our proposed attack always outperforms baselines for the threat model of observation perturbation and the Cartpole domain for the threat model of action perturbation, while still superior to the baselines despite losing two times to the flip baseline on the Fish domain. Only our proposed attack can constantly make the Walker fall down (since we are minimizing its head height to be zero). To have a better sense on the numbers, we give some quick examples below. For instance, as shown in Table 1a and b, we show that the average total loss of walker head height is almost unaffected for the three baselines -if the walker successfully stand or walk, its head height usually has to be greater than 1.2 at every time step, which is 1440 for one episode -while our attack can successfully lower the walker head height by achieving an average of total loss of 258, which is roughly 0.51(0.68) per time step for the stand (walk) task. Similarly, for the humanoid , a successful humanoid usually has head height greater than 1.4, equivalently a total loss of 1960 for one episode, and Table 1a shows that the d4pg agent is robust to the perturbations generated from the three modelfree baselines while being vulnerable to our proposed attack. Indeed, as shown in Figure 2, the walker and humanoid falls down quickly (head height is close to zero) under our specially-designed attack while remaining unaffected for all the other baselines. Evaluating on the total reward. Often times, the reward function is a complicated function and its exact definition is often unavailable. Learning the reward function is also an active research field, which is not in the coverage of this paper. Nevertheless, as long as we have some knowledge of unsafe states (which is often the case in practice), then we can define unsafe states that are related to low reward and thus performing attacks based on unsafe states (i.e. minimizing the total loss of distance to unsafe states) would naturally translate to decreasing the total reward of agent. As demonstrated in Table 2, the have the same trend of the total loss in Table 1, where our proposed attack significantly outperforms all the other three baselines. In particular, our method can lower the average total reward up to 4.96× compared to the baselines , while the baseline are close to the perfect total reward of 1000. Evaluating on the efficiency of attack. We also study the efficiency of the attack in terms of sample complexity, i.e. how many episodes do we need to perform an effective attack? Here we adopt the convention in control suite where one episode corresponds to 1000 time steps (samples), and we learn the neural network dynamical model f with different number of episodes. Figure 3 plots the total head height loss of the walker (task stand) for the three baselines and our method with dynamical model f trained with three different number of samples: {5e5, 1e6, 5e6}, or equivalently {500, 1000, 5000} episodes. We note that the sweep of hyper parameters is the same for all the three models, and the only difference is the number of training samples. The show that for the baselines rand-U and flip, the total losses are roughly at the order of 1400-1500, while 809 959 193 walk 934 913 966 608 (a stronger baseline rand-B still has total losses of 900-1200. However, if we solve Equation 3 with f trained by 5e5 or 1e6 samples, the total losses can be decreased to the order of 400-700 and are already winning over the three baselines by a significant margin. Same as our expectation, if we use more samples (e.g. 5e6, which is 5-10 times more), to learn a more accurate dynamics model, then it is beneficial to our attack method -the total losses can be further decreased by more than 2× and are at the order of 50-250 over 10 different runs. Here we also give a comparison between our model-based attack to existing works on the sample complexity. In , 3e5 episodes of training data is used to learn the adversarial value function, which is roughly 1000× more data than even our strongest adversary (with 5e3 episodes). Similarly, use roughly 2e4 episodes to train an adversary via deep RL, which is roughly 4× more data than ours 2. In this paper, we study the problem of adversarial attacks in deep RL with continuous control for two commonly-used threat models (observation manipulation and action manipulation). Based on the threat models, we proposed the first model-based attack algorithm and showed that our formulation can be easily solved by off-the-shelf gradient-based solvers. Through extensive experiments on 4 MuJoCo domains (Cartpole, Fish, Walker, Humanoid), we show that our proposed algorithm outperforms all the model-free based attack baselines by a large margin. There are several interesting future directions can be investigated based on this work and is detailed in Appendix. A.1 MORE ILLUSTRATION ON FIGURE 3 The meaning of Fig 3 is to show how the accuracy of the learned models affects our proposed technique: 1. we first learned 3 models with 3 different number of samples: 5e5, 1e6, 5e6 and we found that with more training samples (e.g. 5e6, equivalently 5000 episodes), we are able to learn a more accurate model than the one with 5e5 training samples; 2. we plot the attack of total loss for our technique with 3 learned models (denoted as PGD, num train) as well as the baselines (randU, randB, Flip) on 10 different runs (initializations). We show with the more accurate learned model (5e6 training samples), we are able to achieve a stronger attack (the total losses are at the order of 50-200 over 10 different runs) than the less accurate learned model (e.g. 5e5 training samples). However, even with a less accurate learned model, the total losses are on the order of 400-700, which already outperforms the best baselines by a margin of 1.3-2 times. This in Fig 3 also suggest that a very accurate model isn't necessarily needed in our proposed method to achieve effective attack. Of course, if the learned model is more accurate, then we are able to degrade agent's performance even more. For the baselines (rand-U and rand-B), the adversary generates 1000 trajectories with random noise directly and we report the best loss/reward at the end of each episode. The detailed steps are listed below: Step 1: The perturbations are generated from a uniform distribution or a bernoulli distribution within the range [-eps, eps] for each trajectory, and we record the total reward and total loss for each trajectory from the true environment (the MuJoCo simulator) Step 2: Take the best (lowest) total reward/loss among 1000 trajectories and report in Table 1 and 2. We note that here we assume the baseline adversary has an "unfair advantage" since they have access to the true reward (and then take the best attack among 1000 trials), whereas our techniques do not have access to this information. Without this advantage, the baseline adversaries (rand-B, rand-U) may be weaker if they use their learned model to find the best attack sequence. In any case, Table 1 and 2 demonstrate that our proposed attack can successfully uncover vulnerabilities of deep RL agents while the baselines cannot. For the baseline'flip', we add the perturbation (with the opposite sign and magnitude) on the original state/action and project the perturbed state/action are within its limits. We use default total timesteps = 1000, and the maximum total reward is 1000. We report the total reward of the d4pg agents used in this paper below. The agents are well-trained and have total reward close to 1000. There are several interesting future directions can be investigated based on this work, including learning reward functions to facilitate a more effective attack, extending our current approach to develop effective black-box attacks, and incorporating our proposed attack algorithm to adversarial training of the deep RL agents. In particular, we think there are three important challenges that need to be addressed to study adversarial training of RL agents along with our proposed attacks: 1. The adversary and model need to be jointly updated. How do we balance these two updates, and make sure the adversary is well-trained at each point in training? 2. How do we avoid cycles in the training process due to the agent overfitting to the current adversary? 3. How do we ensure the adversary doesn't overly prevent exploration / balance unperturbed vs. robust performance?

Represent the short summary for finding the corresponding scientific article (long)!

We study the problem of continuous control agents in deep RL with adversarial attacks and proposed a two-step algorithm based on learned model dynamics.

Represent the TLDR for finding the corresponding scientific article

We show that a relatively simple black-box adversarial attack scheme using Bayesian optimization and dimension upsampling  is preferable to existing methods when the number of available queries is very low.