Question: Represent the article for finding a single-sentence TLDR

Question: Adversarial training has been demonstrated as one of the most effective methods for training robust models to defend against adversarial examples. However, adversarially trained models often lack adversarially robust generalization on unseen testing data. Recent works show that adversarially trained models are more biased towards global structure features. Instead, in this work, we would like to investigate the relationship between the generalization of adversarial training and the robust local features, as the robust local features generalize well for unseen shape variation. To learn the robust local features, we develop a Random Block Shuffle (RBS) transformation to break up the global structure features on normal adversarial examples. We continue to propose a new approach called Robust Local Features for Adversarial Training (RLFAT), which first learns the robust local features by adversarial training on the RBS-transformed adversarial examples, and then transfers the robust local features into the training of normal adversarial examples. To demonstrate the generality of our argument, we implement RLFAT in currently state-of-the-art adversarial training frameworks. Extensive experiments on STL-10, CIFAR-10 and CIFAR-100 show that RLFAT significantly improves both the adversarially robust generalization and the standard generalization of adversarial training. Additionally, we demonstrate that our models capture more local features of the object on the images, aligning better with human perception. Deep learning has achieved a remarkable performance breakthrough on various challenging benchmarks in machine learning fields, such as image classification and speech recognition. However, recent studies have revealed that deep neural network models are strikingly susceptible to adversarial examples, in which small perturbations around the input are sufficient to mislead the predictions of the target model. Moreover, such perturbations are almost imperceptible to humans and often transfer across diverse models to achieve black-box attacks (; ;). Though the emergence of adversarial examples has received significant attention and led to various defend approaches for developing robust models;; a), many proposed defense methods provide few benefits for the true robustness but mask the gradients on which most attacks rely (a; ; ;). Currently, one of the best techniques to defend against adversarial attacks is adversarial training a), which improves the adversarial robustness by injecting adversarial examples into the training data. Among substantial works of adversarial training, there still remains a big robust generalization gap between the training data and the testing data b;; ). The robustness of adversarial training fails to generalize on unseen testing data. Recent works further show that adversarially trained models capture more on global structure features but normally trained models are more biased towards local features. In intuition, global structure features tend to be robust against adversarial perturbations but hard to generalize for unseen shape variations, instead, local features generalize well for unseen shape variations but are hard to generalize on adversarial perturbation. It naturally raises an intriguing question for adversarial training: For adversarial training, is it possible to learn the robust local features, which have better adversarially robust generalization and better standard generalization? To address this question, we investigate the relationship between the generalization of adversarial training and the robust local features, and advocate for learning robust local features for adversarial training. Our main contributions are as follows: • To our knowledge, this is the first work that sheds light on the relationship between adversarial training and robust local features. Specifically, we develop a Random Block Shuffle (RBS) transformation to study such relationship by breaking up the global structure features on normal adversarial examples. • We propose a novel method called Robust Local Features for Adversarial Training (RLFAT), which first learns the robust local features, and then transfers the information of robust local features into the training on normal adversarial examples. • To demonstrate the generality of our argument, we implement RLFAT in two currently stateof-the-art adversarial training frameworks, PGD Adversarial Training (PGDAT) and TRADES (a). Empirical show consistent and substantial improvements for both adversarial robustness and standard accuracy on several standard datasets. Moreover, the salience maps of our models on images tend to align better with human perception. In this section, we introduce some notations and provide a brief description on current advanced methods for adversarial attacks and adversarial training. Let F (x) be a probabilistic classifier based on a neural network with the logits function f (x) and the probability distribution p F (·|x). Let L(F ; x, y) be the cross entropy loss for image classification. The goal of the adversaries is to find an adversarial example x ∈ B p (x):= {x : x − x p ≤} in the p norm bounded perturbations, where denotes the magnitude of the perturbations. In this paper, we focus on p = ∞ to align with previous works. Projected Gradient Descent. Projected Gradient Descent (PGD) ) is a stronger iterative variant of Fast Gradient Sign Method (FGSM) , which iteratively solves the optimization problem max x: x −x ∞ < L (F ; x, y) with a step size α: where U denotes the uniform distribution, and Π B ∞ (x) indicates the projection of the set B ∞ (x). Carlini-Wagner attack. Carlini-Wagner attack (CW) (2017b) is a sophisticated method to directly solve for the adversarial example x adv by using an auxiliary variable w: The objective function to optimize the auxiliary variable w is defined as: where The constant k controls the confidence gap between the adversarial class and the true class. N attack. N attack is a derivative-free black-box adversarial attack and it breaks many of the defense methods based on gradient masking. The basic idea is to learn a probability density distribution over a small region centered around the clean input, such that a sample drawn from this distribution is likely to be an adversarial example. Despite a wide range of defense methods, and have broken most previous defense methods (; ; ; a), and revealed that adversarial training remains one of the best defense method. The basic idea of adversarial training is to solve the min-max optimization problem, as shown in Eq.: min Here we introduce two currently state-of-the-art adversarial training frameworks. PGD adversarial training. PGD Adversarial Training (PGDAT) leverages the PGD attack to generate adversarial examples, and trains only with the adversarial examples. The objective function is formalized as follows: where x PGD is obtained via the PGD attack on the cross entropy L(F ; x, y). Zhang et al. (2019a) propose TRADES to specifically maximize the trade-off of adversarial training between adversarial robustness and standard accuracy by optimizing the following regularized surrogate loss: where and λ is a hyper-parameter to control the trade-off between adversarial robustness and standard accuracy. Unlike adversarially trained models, normally trained models are more biased towards the local features but vulnerable to adversarial examples . It indicates that, in contrast to global structural features, local features seems be more well-generalized but less robust against adversarial perturbation. Based on the basic observation, in this work, we focus on the learning of robust local features on adversarial training, and propose a novel form of adversarial training called RLFAT that learns the robust local features and transfers the robust local features into the training of normal adversarial examples. In this way, our adversarially trained models not only yield strong robustness against adversarial examples but also show great generalization on unseen testing data. It's known that adversarial training tends to capture global structure features so as to increase invariance against adversarial perturbations . To advocate for the learning of robust local features during adversarial training, we propose a simple and straightforward image transformation called Random Block Shuffle (RBS) to break up the global structure features of the images, at the same time retaining the local features. Specifically, for an input image, we randomly split the target image into k blocks horizontally and randomly shuffle the blocks, and then we perform the same split-shuffle operation vertically on the ing image. As illustrated in Figure 1, RBS transformation can destroy the global structure features of the images to some extent and retain the local features of the images. Then we apply the RBS transformation on adversarial training. Different from normal adversarial training, we use the RBS-transformed adversarial examples rather than normal adversarial examples as the adversarial information to encourage the models to learn robust local features. Note that we only use the RBS transformation as a tool to learn the robust local features during adversarial training and will not use RBS transformation in the inference phase. we refer to the form of adversarial training as RBS Adversarial Training (RBSAT). To demonstrate the generality of our argument, we consider two currently state-of-the-art adversarial training frameworks, PGD Adversarial Training (PGDAT) and TRADES (a), to demonstrate the effectiveness of the robust local features. We use the following loss function as the alternative to the objective function of PGDAT: where RBS(·) denotes the RBS transformation; x PGD is obtained via the PGD attack on the cross entropy L(F ; x, y). Similarly, we use the following loss function as the alternative to the objective function of TRADES: where Since the type of input images in the training phase and the inference phase is different (RBS transformed images for training, versus original images for inference), we consider to transfer the knowledge of the robust local features learned by RBSAT to the normal adversarial examples. Specifically, we present a knowledge transfer scheme, called Robust Local Feature Transfer (RLFT). The goal of RLFT is to learn the representation that minimizes the feature shift between the normal adversarial examples and the RBS-transformed adversarial examples. In particular, we apply RLFT on the logit layer for high-level feature alignment. Formally, the objective functions of robust local feature transfer for PGDAT and TRADES are formalized as follows, respectively: where f (·) denotes the mapping of the logit layer, and · 2 2 denotes the squared Euclidean norm. Since the quality of robust local feature transfer depends on the quality of the robust local features learned by RBSAT, we integrate RBSAT and RLFT into an end-to-end training framework, which we refer to as RLFAT (Robust Local Features for Adversarial Training). The general training process of RLFAT is summarized in Algorithm 1. Note that the computational cost of RBS transformation (line 7) is negligible in the total computational cost. Algorithm 1 Robust Local Features for Adversarial Training (RLFAT). 1: Randomly initialize network F (x); 2: Number of iterations t ← 0; 3: repeat 4: Read a minibatch of data {x 1, ..., x m} from the training set; 6: Generate the normal adversarial examples {x Calculate the overall loss following Eq.. Update the parameters of network F through back propagation; 10: until the training converges. We implement RLFAT in two currently state-of-the-art adversarial training frameworks, PGDAT and TRADES, and have new objective functions to learn the robust and well-generalized feature representations, which we call RLFAT P and RLFAT T: where η is a hyper-parameter to balance the two terms. In this section, to validate the effectiveness of RLFAT, we empirically evaluate our two implementations, denoted as RLFAT P and RLFAT T, and show that our models make significant improvement on both robust accuracy and standard accuracy on standard benchmark datasets, which provides strong support for our main hypothesis. Codes are available online 1. Baselines. Since most previous defense methods provide few benefit in true adversarially robustness , we compare the proposed methods with state-of-theart adversarial training defenses, PGD Adversarial Training (PGDAT) and TRADES (a). Adversarial setting. We consider two attack settings with the bounded ∞ norm: the white-box attack setting and the black-box attack setting. For the white-box attack setting, we consider existing strongest white-box attacks: Projected Gradient Descent (PGD) ) and CarliniWagner attack (CW) (b). For the black-box attack setting, we perform the powerful black-box attack, N attack , on a sample of 1,500 test inputs as it is timeconsuming. Datasets. We compare the proposed methods with the baselines on widely used benchmark datasets, namely CIFAR-10 and CIFAR-100 (Hyper-parameters. To avoid posting much concentrate on optimizing the hyper-parameters, for all datasets, we set the hyper-parameter λ in TRADES as 6, set the hyper-parameter η in RLFAT P as 0.5, and set the hyper-parameter η in RLFAT T as 1. For the training jobs of all our models, we set the hyper-parameters k of the RBS transformation as 2. More details about the hyper-parameters are provided in Appendix A. We first validate our main hypothesis: for adversarial training, is it possible to learn the robust local features that have better adversarially robust generalization and better standard generalization? In Table 1, we compare the accuracy of RLFAT P and RLFAT T with the competing baselines on three standard datasets. The proposed methods lead to consistent and significant improvements on adversarial robustness as well as standard accuracy over the baseline models on all datasets. With the robust local features, RLFAT T achieves better adversarially robust generalization and better standard generalization than TRADES. RLFAT P also works similarly, showing a significant improvement on the robustness against all attacks and standard accuracy than PGDAT. The demonstrate that, the robust local features can significantly improve both the adversarially robust generalization and the standard generalization over the state-of-the-art adversarial training frameworks, and strongly support our hypothesis. That is, for adversarial training, it is possible to learn the robust local features, which have better robust and standard generalization. Motivation. and Zhang et al. (2019b) found that the effectiveness of adversarial training is highly sensitive to the "semantic-loss" shift of the test data distribution, such as gamma mapping. To further investigate the performance of the proposed methods, we quantify the smoothness of the models under the distribution shifts of brightness perturbation and gamma mapping. Loss sensitivity on brightness perturbation. To quantify the smoothness of models on the shift of the brightness perturbation, we propose to estimate the Lipschitz continuity constant F by using the gradients of the loss function with respect to the brightness perturbation of the testing data. We adjust the brightness factor of images in the HSV (hue, saturation, value) color space, which we refer to as x b = V(x, α), where α denotes the magnitude of the brightness adjustment. The lower the value of b F (α) is, the smoother the loss function of the model is: Loss sensitivity on gamma mapping. Gamma mapping is a nonlinear elementwise operation used to adjust the exposure of images by applyingx (γ) = x γ on the original image x. Similarly, we approximate the loss sensitivity under gamma mapping, by using the gradients of the loss function with respect to the gamma mapping of the testing data. A smaller value indicates a smoother loss function. Sensitivity analysis. The for the loss sensitivity of the adversarially trained models under brightness perturbation are reported in Table 2a, where we adopt various magnitude of brightness adjustment on each testing data. In Table 2b, we report the loss sensitivity of adversarially trained models under various gamma mappings. We observe that RLFAT T provides the smoothest model under the distribution shifts on all the three datasets. The suggest that, as compared to PGDAT and TRADES, both RLFAT P and RLFAT T show lower gradients of the models on different data distributions, which we can directly attribute to the robust local features. To further gain insights on the performance obtained by the robust local features, we perform ablation studies to dissect the impact of various components (robust local feature learning and robust local feature transfer). As shown in Figure 2, we conduct additional experiments for the ablation studies of RLFAT P and RLFAT T on STL-10, CIFAR-10 and CIFAR-100, where we report the standard accuracy over the clean data and the average robust accuracy over all the attacks for each model. We first analyze that as compared to adversarial training on normal adversarial examples, whether adversarial training on RBS-transformed adversarial examples produces better generalization and more robust features. As shown in Figure 2, we observe that Robust Local Features Learning (RLFL) exhibits stable improvements on both standard accuracy and robust accuracy for RLFAT P and RLFAT T, providing strong support for our hypothesis. Does robust local feature transfer help? We further add Robust Local Feature Transfer (RLFT), the second term in Eq., to get the overall loss of RLFAT. The robust accuracy further increases on all datasets for RLFAT P and RLFAT T. The standard accuracy further increases also, except for RLFAT P on CIFAR-100, but it is still clearly higher than the baseline model PGDAT. It indicates that transferring the robust local features into the training of normal adversarial examples does help promote the standard accuracy and robust accuracy in most cases. We would like to investigate the features of the input images that the models are mostly focused on. Following the work of , we generate the salience maps using SmoothGrad on STL-10 dataset. The key idea of SmoothGrad is to average the gradients of class activation with respect to noisy copies of an input image. As illustrated in Figure 3, all the adversarially trained models basically capture the global structure features of the object on the images. As compared to PGDAT and TRADES, both RLFAT P and RLFAT T capture more local feature information of the object, aligning better with human perception. Note that the images are correctly classified by all these models. For more visualization , see Appendix B. Differs to existing adversarially trained models that are more biased towards the global structure features of the images, in this work, we hypothesize that robust local features can improve the generalization of adversarial training. To validate this hypothesis, we propose a new stream of adversarial training approach called Robust Local Features for Adversarial Training (RLFAT) and implement it in currently state-of-the-art adversarial training frameworks, PGDAT and TRADES. We provide strong empirical support for our hypothesis and show that the proposed methods based on RLFAT not only yield better standard generalization but also promote the adversarially robust generalization. Furthermore, we show that the salience maps of our models on images tend to align better with human perception, uncovering certain unexpected benefit of the robust local features for adversarial training. Our findings open a new avenue for improving adversarial training, whereas there are still a lot to explore along this avenue. First, is it possible to explicitly disentangle the robust local features from the perspective of feature disentanglement? What is the best way to leverage the robust local features? Second, from a methodological standpoint, the discovered relationship may also serve as an inspiration for new adversarial defenses, where not only the robust local features but also the global information is taken into account, as the global information is useful for some tasks. These questions are worth investigation in future work, and we hope that our observations on the benefit of robust local features will inspire more future development. Here we show the details of the training hyper-parameters and the attack hyper-parameters for the experiments. Training Hyper-parameters. For all training jobs, we use the Adam optimizer with a learning rate of 0.001 and a batch size of 32. For CIFAR-10 and CIFAR-100, we run 79,800 steps for training. For STL-10, we run 29,700 steps for training. For STL-10 and CIFAR-100, the adversarial examples are generated with step size 0.0075, 7 iterations, and = 0.03. For CIFAR-10, the adversarial examples are generated with step size 0.0075, 10 iterations, and = 0.03. Attack Hyper-parameters. For the PGD attack, we use the same attack parameters as those of the training process. For the CW attack, we use PGD to minimize its loss function with a high confidence parameter (k = 50) following the work of. For the N attack, we set the maximum number of optimization iterations to T = 200, b = 300 for the sample size, the variance of the isotropic Gaussian σ 2 = 0.01, and the learning rate η = 0.008. We provide more salience maps of the adversarially trained models on sampled images in Figure 4. Original PGDAT TRADES RLFATP RLFATT Original PGDAT TRADES RLFATP RLFATT Figure 4: More Salience maps of the four models. For each group of images, we have the original image, and the salience maps of the four models sequentially.

Represent the TLDR for finding the corresponding scientific article

We propose a new stream of adversarial training approach called Robust Local Features for Adversarial Training (RLFAT) that significantly improves both the adversarially robust generalization and the standard generalization.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We present a new method that combines transfer-based and scored black-box adversarial attack, improving the success rate and query efficiency of black-box adversarial attack across different network architectures.


Question: Represent the scientific article to find a short TLDR

Question: The verification of planning domain models is crucial to ensure the safety, integrity and correctness of planning-based automated systems. This task is usually performed using model checking techniques. However, directly applying model checkers to verify planning domain models can in false positives, i.e. counterexamples that are unreachable by a sound planner when using the domain under verification during a planning task. In this paper, we discuss the downside of unconstrained planning domain model verification. We then propose a fail-safe practice for designing planning domain models that can inherently guarantee the safety of the produced plans in case of undetected errors in domain models. In addition, we demonstrate how model checkers, as well as state trajectory constraints planning techniques, should be used to verify planning domain models so that unreachable counterexamples are not returned. Planning and task scheduling techniques are increasingly applied to real-world problems such as activity sequencing, constraint solving and resource management. These processes are implemented in planning-based automated systems which are already used in space missions BID14 BID3 BID0, search and rescue BID12, logistics BID19 and many other domains. Since the failure of such systems could have catastrophic consequences, these applications are regarded as safety-critical. Therefore, verification methods that are robust, trustworthy and systematic are crucial to gain confidence in the safety, integrity and correctness of these systems. The literature is rich with studies on verification of planning systems. For instance, BID17 carried out scenario-based testing and model-based validation of the remote agent that controlled the Deep Space 1 mission. Another example is the verification of the safety of the autonomous science agent design that was deployed on the Earth Orbiter 1 spacecraft.A typical planning system consists of a planning domain model, planning problem, planner, plan, executive, and mon-itor. Planners take as an input a domain model which describes application-specific states and actions, and a problem that specifies the goal and the initial state. From these inputs, a sequence of actions that can achieve the goal starting from the initial state is returned as plan. The plan is then executed by an executive to change the world state to match the desired goal. Our research focuses on the verification of planning domain models wrt. safety properties. Domain models provide the foundations for planning. They describe real-world actions by capturing their pre-conditions and effects. Due to modelling errors, a domain model might be inconsistent, incomplete, or inaccurate. This could cause the planner to fail in finding a plan or to generate unrealistic plans that will fail to execute in the real world. Moreover, erroneous domain models could lead planners to produce unsafe plans that, when executed, could cause catastrophic consequences in the real world. This paper addresses the fact that the state-of-the-art verification methods for planning domain models are vulnerable to false positive counterexamples. In particular, unconstrained verification tasks might return counterexamples that are unreachable by planners. Such counterexamples can mislead designers to unnecessarily restrict domain models, thereby potentially blocking valid and possibly necessary behaviours. In addition, false positive counterexamples can lead verification engineers to overlook counterexamples that are reachable by planners. To overcome these deficiencies, we propose to employ planning goals as constraints during verification. Thus, we introduce goal-constrained planning domain model verification, a novel concept that eliminates unreachable counterexamples per se. We formally prove that goal-constrained planning domain model verification of safety properties is guaranteed to return reachable counterexamples if and only if any exist. We also demonstrate two different ways to perform goal-constrained planning domain model verification, one using model checkers and the other using state trajectory constraints planning techniques. To the best of our knowledge, this work is the first to recommend fail-safe planning domain model design practice; introduce the concept of goal-constrained planning domain model verification. and demonstrate how model checkers, as well as state trajectory constraints planning techniques, can be used to perform goal-constrained planning domain model verification The rest of this paper is organised as follows. First, Section 2, contrasts the concepts presented here with related work. Second, Section 3 discusses the problem of unreachable counterexamples in planning domain model verification. Third, Section 4 proposes a design practice for planning domain models that can inherently guarantee domain model safety even in the case of undetected modelling errors. A verification concept of planning domain models that avoids returning unreachable counterexamples is presented in Section 5. Then, Section 6 discusses the implementation of this concept on the Cave Diving planning domain using Spin and MIPS-XXL. Finally, Section 7 concludes the paper and suggests future work. Closely related, but different, is the work by BID1. Their main objective is to treat verification as a planning task, whereas our aim is to demonstrate how model checkers and planners can be used for domain model verification. They proposed to perform system model verification using classical planners. To do this, they first translated the model of the system under verification into a planning domain model. Then, the negation of the safety property to be established, is used as the goal for the planner, which is then consulted to find a plan that acts as counterexample for the given property. In our study, because our aim is to verify domain models against a given property with respect to a specific goal and initial state, we used state trajectory constraints to restrict counterexamples to identify plans that can achieve the planning goal while falsifying the safety property. Unlike BID1, where the negation of the safety property is used as the goal, in our verification as planning method, the negation of the safety property is represented as state trajectory constraint and the goal is the given planning goal. BID16 ) also apply verification as planning to verify planning domain models, starting from LTL specifications. This work fundamentally differs from our work. BID16 focused their work on translating specification properties into trap formulas which can help in testing the impact of individual atomic propositions on the validity of the overall verified property. However, their method does not consider the interaction between property testing and the original planning goal. Note that finding a planning constraint to exercise a specific atomic proposition is not enough to ensure the constraint itself would be exercised during the planning process. For example, a planning goal might be achieved through a state trajectory that does not exercise the hard constraint used to represent the tested property. Our work is mainly based on investigating this interaction. Therefore, we used state trajectory constraints to guarantee the property is tested while achieving the planning goal. Additionally, their work, just like other similar methods, requires a complete planner to give deterministic , whereas our work, as discussed in Section 5, guarantees definite verification without this requirement.(Goldman, Kuter, and Schneider 2012) also used classical planners for planning systems verification, but they examined verifying plans rather than domain models. They proposed an approach that uses classical planners to find counterexamples for a given planning problem and plan instance. Their work and ours are related in that both suggest performing planning verification for a specific planning problem rather than attempting ungrounded verification of a planning system. However, their work is limited to the verification of single plan instances, whereas our method verifies all potential plans that can be spun from a domain model for a specific goal and initial state. Among others, BID15 BID13 BID18 BID10 BID2 ) used model checkers to verify planning domain models. They translated the respective domain models into the input language of the selected model checker. The model checker is then applied to verify the domain model wrt. a given specification property. Similarly, we also proposed a method to verify domain models using model checkers. However, our method differs from the others in two aspects. First, in the way we define the planning domain model verification problem, and, second, in the way we use model checkers to perform verification. As explained in Section 5, we consider the verification of planning domain models to be constrained by a specific goal and initial state pair. In contrast, previous studies perform ungrounded verification of domain models, i.e. leaving the goal and initial state open. As discussed in Section 3, the ungrounded goal and initial state may cause the model checker to return counterexamples that are unreachable when a planner uses the DUV. These unreachable counterexamples can mislead the designers to over-restrict the DUV during the debugging process. On the other side, when the goal and initial state are constrained for verification, then we have shown that the returned counterexamples, if any, are guaranteed to be reachable by any sound planner. The second difference is that, after the planning domain model is translated to the model checker's input language, we augment the model transitions, introducing the negation of the goal as a new constraint that forces the model checker to terminate once the goal is reached. This modification prevents the model checker from returning counterexamples that falsify the given property after satisfying the goal; these are unreachable by planners. Planning domain model verification aims to demonstrate that any produced plan satisfies a set of properties. To achieve this, formal planning domain model verification methods leave the planning goal open. This, we define as unconstrained verification of planning domain models, i.e. the verification is expected to hold for any potential goal. unconstrained verification searches the domain model for a sequence of actions that can falsify the given property, regardless of any other conditions. In particular, whether or not a planner would consider this sequence to be a plan, is not taken into account. This is a critical oversight, because, when the domain model is used to solve a specific planning problem, the sequence of actions that constitutes such a counterexample might, in fact, be "pruned away" by the planner, if it does not satisfy the planning goal. Therefore, for a specific planning problem, counterexamples that do not achieve the planning goal are deemed unreachable counterexamples from the planner's perspective. To illustrate this, we use a modified version of the microwave oven example, introduced in BID5, as presented in FIG0. A safety requirement would be that the domain model does not allow the generation of erroneous plans, in LTL p 0 = G(¬Error), where G is the LTL globally operator. Unconstrained verification will return StartOven as a counterexample that when applied to s 0 will produce s 2 which is an error state. However, when this model is used to find a plan that achieves the goal (g = Heat), this sequence will not be considered by the planner as it does not lead to a state that achieves the goal. Moreover, we observe that the valid plan CloseDoor, StartOven does satisfy the property p 0, i.e. is error-free. Thus, the sequence StartOven from s 0 to s 2 is an unreachable counterexample for the planner; it does not achieve the goal, nor is it part of a valid plan towards the goal. DISPLAYFORM0 Counterexamples that are unreachable by planners exist in the literature. For example, BID18 ) used the Spin model checker to verify whether a planning domain model would permit an automated planning system to select plans that would waste resources and therefore not meet the mission's science goals. To express this requirement, they used "five data-producing activities must be scheduled by any returned plan" as a property for model checking. The automated system has two data-producing and two data-consuming activities, and a buffer that can hold four data blocks. The goal of the planner is to schedule five data-producing activity instances. The counterexample returned by the model checker represented a plan with the two data-consuming activities scheduled before four dataproducing activities. This plan did not contain a fifth dataproducing task, because the data buffer was full after four data-producing activities and the only two data-consuming tasks that would have cleared the buffer, were scheduled at the beginning of the plan with no data in the buffer. Though the model checker found a counterexample to falsify the property, we argue that any sound planner would not generate such a plan, because it does not achieve the planning goal. As such, this counterexample would have been pruned during the planner's goal search, and consequently, it would never have been returned as a plan, i.e. it is unreachable for the planner, yet reachable by a goal-ignorant model checker. The problem with unreachable counterexamples is that they mislead the designer to unnecessarily restrict the domain model in the process of removing them. Consequently, debugging is made harder and genuine counterexamples could potentially be introduced in the process. To overcome this, we observe that planning is performed for a specific goal and initial state. To exploit this observation for domain model verification, we propose to use the goal and initial state given to the planner as constraints to ensure that the counterexamples returned by a model checker, or other tools used in this context, falsify the given property while also achieving the planning goal. Thus, instead of performing unconstrained domain model verification, we propose goal-constrained verification of planning domain models. The details of this method are further explained in Section 5. Next, we describe an inherently safe domain model design practice which can help to make domain models safer. The ultimate objective of planning domain model verification is to ensure that the plans produced by the verified domains satisfy a given specification. An alternative and guaranteed way of achieving this goal is to extract plan constraints from the specification, then include them in the domain model. A sound planner using this constrained domain model cannot produce any plan that could violate these constraints. This idea was first noticed in 2005 BID18 ) but was dismissed as it was not possible to describe overall plan constraints using PDDL 2.2. However, in BID7 BID7 proposed an extension to the PDDL 2.2 language that allows the expression of plan state trajectory constraints. The extended language, called PDDL3.0, was proposed for the fifth international planning competition (IPC-5). BID18 provided an example of a system consisting of a camera, solid-state recorder and a radio, and a requirement that for all plans, if an image is taken and stored, then it is eventually uplinked. With the hard state trajectory constraints, this property can be expressed as sometimeafter((image is taken and image is stored) image is uplinked). With this constraint, any sequence of actions that does not respect this property would not be returned as a plan. Though including specification properties in the domain model as strong constraints is enough to guarantee that sound planners using the constrained domain models will produce plans that meet the specification, this method will not be able to find any errors in the domain model. Instead, it will just ensure these errors, if any, are masked and prevented from affecting any plans that could possibly be generated using the modified domain model. As such, this method can be seen as a safety defence layer, a firewall, that prevents any potential property violation. Nevertheless, note that undetected bugs in a domain model could cause what would have been valid plans to be masked, thus unnecessarily restricting the planner. Therefore, further verification efforts are needed to reveal and rectify any underlying errors. We consider including plan constraints in the domain model to be a good practice to design inherently safe domain models. The effort of extracting formal properties from the specification and inserting them as constraints in the domain model is a small investment in return to the huge benefit of guaranteed safe plans, i.e. plans that are safe "by construction". This, together with our new concept of goalconstrained verification, as introduced in the next section, can deliver safe and error-free models.5 Goal-constrained verification of planning domain modelsPlanning domain model verification covers different objectives, including the domain's correctness, completeness, robustness, effectiveness and safety. The intent of safety verification in this context is to verify that any plan produced from the DUV will satisfy a given safety property. In other words, a domain is considered safe if the domain is guaranteed only to produce plans that satisfy the given safety property when used by a sound planner. This verification task can be performed using advanced search algorithms, such as model checkers or classical planners, to find a valid counterexample for the given safety property. We define a valid counterexample to be a sequence of actions that, firstly, can falsify the given safety property, secondly, can achieve the planning goal from the given initial state, and, thirdly, none of the sub-sequences of the counterexample can achieve the goal. Formally, this is defined as follows: Let the planning problem P be a tuple (D, s 0, g), where D is the domain model that describes the set of all available actions A, s 0 is the initial state and g is the desired goal. π is a solution to P, a plan, defined as a sequence of actions, where these actions are chosen from A. π = a 0, a 1,..., a n such that π |= g i.e. when π is applied to the initial state s 0 it yields a sequence of states S, S = s 0, s 1,..., s n where the last state s n satisfies the planning goal g, s n |= g. We say a plan π satisfies a property p, π |= p, if the sequence of states S, generated by the plan π, satisfies the property p, S |= p. Furthermore, as defined in (Ghallab, Nau, and Traverso 2004), we call a plan π a redundant plan, if π contains a subsequence, π, π | π, that achieves the goal g. Definition 1: A valid counterexample for a safety property, p, of a planning problem is a non-redundant plan, π, that falsifies the safety property, π |= p. Plans are required to be non-redundant in the definition of valid counterexamples to exclude any plans that are enriched with action sequences which are unnecessary to achieve the planning goal but required to falsify the given safety property. Such plans represent counterexamples that are unreachable by any sound planner when searching for a plan to achieve a given planning problem in a planning task. Such plans represent counterexamples that are unreachable by any sound planner. To ensure the returned counterexamples are valid, we constrain the verification problem with a goal and initial state, and we exclude any counterexample that is a redundant plan. More formally, the verification problem associated with planning task P is defined as the tuple V = (D, (s 0, g), p). Where p is a formal safety property extracted from a given specification and required to hold over all valid paths that achieve the goal g from the initial state s 0.In this section, we introduced and formally defined the concept of goal-constrained verification of planning domain models. In the following subsections, we demonstrate how this concept can be realized using model checkers and state trajectory constraints planning techniques. Model checkers verify safety properties by searching for counterexamples that falsify those properties. In the case of planning applications, any sequence of actions that does not achieve the given goal, will be pruned by any sound planner. Therefore, in the verification of planning problems, any counterexample that does not achieve the goal of the planning problem should be eliminated on the bases that this counterexample is unreachable by the planner. To force model checkers to only return valid counterexamples, the safety property is first negated and then joined with the planning goal in a conjunction. This conjunction is then negated and supplied to the model checker as an input property. The final property requires the model checker to find a counterexample that both, falsifies the safety property and satisfies the planning goal. Note that, unlike Def. 1, this permits sequences that falsify the property after satisfying the goal. However, once the goal is achieved, planners terminate the search, thus rendering such sequences unreachable. To eliminate these sequences, model transitions should be augmented with an additional guard, representing the negation of the goal, to restrict all transitions once the goal is achieved. With this modification, the model checker is forced to return counterexamples that falsify the safety property before achieving the goal, because once the goal is satisfied no further transitions will be permitted. For a verification problem V = (D, (s 0, g), p) we first translate the domain model D into the model checker's input language, obtaining the model M that incorporates the initial state s 0. Then, a model checker is applied to the verification problem V = (M, ¬F (g)) to establish that DISPLAYFORM0 where F is the LTL eventually operator. The model M is modified to M by augmenting the guards of all transitions with the negation of the goal condition. The model checker is then applied to the verification problem V = (M, p) where p is defined as follows: DISPLAYFORM1 There are two possible outcomes of the verification task V. If the model checker returns a counterexample, π, then: DISPLAYFORM2 From the definition of the LTL eventually operator F: DISPLAYFORM3 It follows that there is at least one sequence S that falsifies the property p, and there is a state s j in that sequence which satisfies the goal g, according to and. In addition to that, in the sequence S, p is guaranteed to be falsified before g is satisfied, due to the modification we introduced in the model M. Thus, the plan π is a valid counterexample for the original safety property p as per Def. 1. This proves that the DUV does not satisfy the safety property p with respect to the goal and initial state. The other potential outcome is that the model checker fails to find a counterexample, then for all plans, π: DISPLAYFORM4 It follows that π |= F (¬p) ∨ π |= F (g). Furthermore, from we know that ∃π. π |= F (g). Therefore for all plans, π: DISPLAYFORM5 That means p is always true for all possible plans. Which proves that the DUV satisfies the original property with respect to the goal and initial state. Domain models can be verified to only produce valid plans, in terms of satisfying given a property, for a specific goal and initial state pair using sound planners. This is achieved by consulting the planner over the DUV to produce a plan that can satisfy the goal and the negation of the property. If the domain model permits producing plans that, along with achieving the goal, contradict the safety property, then an unsafe plan can be found. Thus, the returned plan is a counterexample that demonstrates that the safety property does not hold. On the other hand, if the domain model does not permit the generation of plans that can satisfy the negation of the safety property while achieving the goal, then the planner will fail. Thus, the property holds in any plan produced for the given goal. A benefit of goal-constrained planning domain verification is, where a planner is used to perform the verification task, there is no need to for this planner to be complete, as long as the planner used for the verification is also the planner that will be used during the planning task. This is because any counterexample not found by that planner during verification, will then also not be reached by the same planner during the planning task. The following subsection provides a description of how state trajectory constraints can be used to verify planning domain models for a specific goal and initial state. Goal-constrained planning domain verification using planning techniques with state trajectory constraints The PDDL3.0 state trajectory constraints, first mentioned in Section 4, can be used to perform planning domain model verification. First, the negation of the given property is expressed using PDDL3.0 modal operators and embedded in the original domain model as state trajectory constraint. The modified model is then used by a planner, as described earlier, to perform the verification. For a verification problem V = (D, (s 0, g), p), we first apply a planner to the planning problem P = (D, s 0, g) to establish that there is a plan that solves P ∃π. π |= g. Then, the safety property p is negated and expressed in terms of PDDL3.0 modal operators as shown in BID8 ). The is added as state trajectory constraint to the original domain model. Using the algorithm proposed in (Edelkamp, Jabbar, and Nazih 2006), the new model is transformed into a PDDL2 compatible version. This is performed by first translating the state trajectory constraint into a non-deterministic finite state automaton (NFA) which can monitor property violations by inserting additional predicates and actions conditional effects into the model to simulate and observe the behaviour of the automaton that represents the constraint. This yields a new planning problem P = (D, s 0, g), where D, s 0, g are modified instances of D, s 0, g that are supplemented with the additional predicates and actions conditional effects of the automaton that represents the introduced constraint. Then, a planner is applied to P with two possible outcomes. If the planner finds a plan then: DISPLAYFORM0 Since the satisfaction of g implies both, the satisfaction of the original goal g at the last state of the sequence S, and the satisfaction of the state trajectory constraint by the sequence S, can be rewritten as DISPLAYFORM1 Furthermore, from it follows that π |= p, confirming that there is at least one plan that achieves the goal while not respecting the safety property. Therefore, this plan is a valid counterexample for that property as per Def. 1. Hence, the DUV does not satisfy the property wrt. the planning goal and initial state. Alternatively, if the planner fails to find a plan, then, as opposed to FORMULA1, we have DISPLAYFORM2 Given FORMULA1, FORMULA1 can be simplified to: DISPLAYFORM3 Hence, all plans satisfy the safety property. Therefore, the property holds for the planning domain model wrt. the given goal and initial state. In this section, we discuss how goal-constrained planning domain verification can verify safety properties using both the Spin model checker BID11 ) and the MIPS-XXL planner BID6. We perform constrained and unconstrained verification tasks to show how unlike the latter task our method does not return unreachable counterexamples. As an example, we consider the classical cave diving planning domain taken from the Satisfying Track of the IPC-2014 (IPC2014 2014 . The problem consists of an underwater cave system with a finite number of partially interconnected locations. Divers can enter the cave from a specific location, entrance, and swim from one location to an adjacent connected one. They can hold up to four oxygen tanks and they consume one for every swim and take-photo action. Only one diver can be in the cave at a time. Finally, divers have to perform a decompression manoeuvre to go to the surface and this can be done only at the entrance. Additionally, divers can drop tanks in or take tanks from any location if they hold at least one tank or there is one tank available at the location respectively. The planning goals of this domain, as provided in the problem files in the IPC-2014, consist of two parts. The first part dictates the required underwater location of which the photo is to be taken (we call it mission target) and the second part which mandates the divers should return to the surface after completing the mission (we call it safety target).A critical safety property is that divers should not drown i.e. they should not be in an underwater location, other than the entrance, where neither the divers nor the location has one full oxygen tank at least. To enable the planner and the model checkers to explore the entire state space, we simplified this domain by ignoring the "precludes" condition from the original domain as it does not affect the verification of the drowning safety property. Consequently, we considered only one diver and we modified some actions to enable the diver to go back into the water after a dive. These modifications are further explained in the commented simplified planning domain model PDDL file which is provided along with the tasks problem PDDL and Promela files online [address hidden for blind review].First, we translated the planning domain model from PDDL to Promela. Thus, the verification using the translated model only hold provided that the translation is valid. The verification of the translation is outside the scope and focus of this paper and left for future work. In this example, the chosen planning goal is to have a photo of the first location, L 1, and to get the diver outside the water. The verification tasks are:1 -Unconstrained verification with only the safety property: Both Spin and MIPS-XXL found a counterexample prepare-tank, enter-water, swim(L 0, L 1). Indeed, this counterexample leads the diver to a drowning state. At the end of this sequence, the diver will be in underwater location L 1 which is not the entrance so they can not surface and with no oxygen tank to swim back to the entrance. However, this is not a plan because it does not achieve any useful goal. Therefore, it will not be produced by any sound planner when it is used in a practical scenario (taking a photo of any location).2-Verification with safety property and incomplete goal (mission target only): Both Spin and MIPS-XXL returned prepare-tank, prepare-tank, enter-water, swim(L 0, L 1), take-photo. This counterexample achieves the goal and violates the property. However, without the safety part of the goal, it would be possible to generate plans that imply divers should swim to an underwater location and take a photo of it without requiring the divers to return to the surface. These kind of plans are illegal as they do not respect the safety part of the goal. Therefore, such sequences are unreachable counterexamples i.e. will never be produced by any sound planner while planning for a legal goal.3-Verification using Spin with both safety property and proper goal but without the augmented model M: Spin found a counterexample prepare-tank, prepare-tank, prepare-tank, prepare-tank, enter-water, swim(L 0, L 1), take-photo, swim(L 1, L 0), decompress, enter water, swim(L 0, L 1). This counterexample achieves the goal and violates the safety property but only after the goal is achieved. Therefore, this is also an unreachable counterexample because a sound planner will terminate after achieving the goal and any counterexample that violates the property after achieving the goal will not be returned. Hence, it is unreachable. 4-Goal-constrained planning domain verification, as presented in this paper, the was: No plan is returned by the planner MIPS-XXL with complete exploration and no counterexample is returned by Spin with exhaustive verification mode. This means the planning domain model has no provision of producing a plan that can violate the safety property before achieving the goal. I.e. this model is safe with respect to the given property and goal pair. Though the counterexamples returned by the incomplete verification tasks number one, two and three are obviously unreachable and should not misguide the designers to overcomplicate the model, in a real world sized application such unreachable counterexamples can be critical and much more diffcult to recognise and avoid. We expect that our proposed concept can save practitioners a huge amount of personhours trying to alter planning domain models for behaviours that their planners will never experience in practice. The verification of planning domain models is essential to guarantee the safety of planning-based automated systems. Unreachable counterexamples returned by unconstrained planning domain model verification techniques undermine the verification . In this paper, we have discussed the potential deficiencies of this problem and provided an example of an unreachable counterexample form the literature. We then introduced goal-constrained verification, a new concept to address this problem, which restricts the verification task to a specific goal and initial state pair. This limits counterexamples to those practically reachable by a planner that is tasked with achieving the goal given the initial state. Consequently, our method verifies the domain model only wrt. a specific goal and initial state. This is an acceptable limitation, given that planners also operate on this basis. We have demonstrated how model checkers and planning techniques can be used to perform goal-constrained planning domain model verification. In addition, we have recommended an inherently safe practice for domain model design that guarantees the safety of domain models "by construction" in case of undetected modelling errors. Goalconstrained domain model verification ensures accurate verification and complements the inherently safe domain model design practice to generate safe and error-free planning domain models. In , the main message of this paper is that the direct application of verification algorithms to the planning domain model verification problem can return counterexamples that would never be reached by planners in real planning tasks. These unreachable counterexamples can mislead the designers to perform unnecessary remediations that can be prone to errors. The proposed solution is simple which makes it readily usable in practice. It is also effective as formally proven in the paper. Currently, we are investigating the use of Temporally Extended Goals (TEGs) translators BID20 to perform goal-constrained domain model verification. As future work, we intend to automate the proposed methods, so that they can be applied to real-world sized planning domain models. Finally, we would like to perform an empirical comparison of the proposed methods to assess their scalability and performance.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Why and how to constrain planning domain model verification with planning goals to avoid unreachable counterexamples (false positives verification outcomes).

Represent the TLDR for finding the corresponding scientific article

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems


Question: We've seen tremendous success of image generating models these years. Generating images through a neural network is usually pixel-based, which is fundamentally different from how humans create artwork using brushes. To imitate human drawing, interactions between the environment and the agent is required to allow trials. However, the environment is usually non-differentiable, leading to slow convergence and massive computation. In this paper we try to address the discrete nature of software environment with an intermediate, differentiable simulation. We present StrokeNet, a novel model where the agent is trained upon a well-crafted neural approximation of the painting environment. With this approach, our agent was able to learn to write characters such as MNIST digits faster than reinforcement learning approaches in an unsupervised manner. Our primary contribution is the neural simulation of a real-world environment. Furthermore, the agent trained with the emulated environment is able to directly transfer its skills to real-world software. To learn drawing or writing, a person first observes (encodes) the target image visually and uses a pen or a brush to scribble (decode), to reconstruct the original image. For an experienced painter, he or she foresees the consequences before taking any move, and could choose the optimal action. Stroke-based image generation is fairly different from traditional image generation problems due to the intermediate rendering program. Raster-based deep learning approaches for image generation allow effective optimization using back-propagation. While for stroke-based approaches, rather than learning to generate the image, it is more of learning to manipulate the painting program. An intuitive yet potentially effective way to tackle the problem is to first learn this mapping from "stroke data" to the ing image with a neural network, which is analogous to learning painting experience. An advantage of such a mapping over software is that it provides a continuous transformation. For any painting program, the pixel values of an image are calcuated based on the coordinate points along the trajectory of an action. Specific pixels are indexed by the discrete pixel coordinates, which cuts the gradient flow with respect to the action. In our implementation, the indexing is done by an MLP described in Section 3.We further define "drawing" by giving a formal definition of "stroke". In our context, a "stroke" consists of color, brush radius, and a sequence of tuples containing the coordinate and pressure of each point along the trajectory. We will later describe this in detail in Section 3.Based on these ideas, we train a differentiable approximator of our painting software, which we call a "generator". We then tested the generator by training a vanilla CNN as an agent that encodes the image into "stroke" data as an input for the environment. Our proposed architecture, StrokeNet, basically comprises the two components, a generator and an agent. Finally, an agent is trained to write and draw pictures of several popular datasets upon the generator. For the MNIST digits, we evaluated the quality of the agent with a classifier trained solely on the original MNIST dataset, and tested the classifier on generated images. We also compared our method with others to show the efficiency. We explored the latent space of the agent as well. Generative models such as VAEs BID13 BID26 and GANs BID5 BID17 BID21 BID0 have achieved huge success in image generation in recent years. These models generate images directly to pixel-level and thus could be trained through back-propagation effectively. To mimic human drawing, attempts have been made by both graphics and machine learning communities. Traditionally, trial-and-error algorithms BID9 are designed to optimize stroke placement by minimizing an energy function, incorporating heuristics, e.g., constraining the number of strokes. Concept learning is another example tackling this problem using Bayesian program learning BID14. Recent deep learning based approaches generally falls into two categories: RNN-based approaches and reinforcement learning. For RNN-based approaches such as SketchRNN BID7 and handwriting generation with RNN by Graves BID6, they both rely on sequential datasets. Thus for unpaired data, those models cannot be applied. Another popular solution is to adopt reinforcement learning such as "artist agent" BID27 and SPIRAL BID4. These methods train an agent that interact with the painting environment. For reinforcement learning tasks with large, continuous action space like this, the training process can be computationally costly and it could take the agent tens of epochs to converge. To mitigate this situation, we simulate the environment in a differentiable manner much alike the idea in World Models BID8 BID22 BID23, where an agent learns from a neural network simulated environment. Similar approach is also used in character reconstruction for denoising BID11. In our scenario, we train our generator (auto-encoder) by parts for flexible stroke sequence length and image resolution, discussed in Section 3 and 4.Differentiable rendering is an extensively researched topic in computer graphics. It is used to solve inverse rendering problems. Some differentiable renderers explicitly model the relationship between the parameters and observations BID16, others use neural network to approximate the BID18 since neural nets are powerful function approximators. While little has been done on simulating 2D rendering process adopted in digital painting software, we used a generator neural network to meet our needs. We define a single stroke as follows, DISPLAYFORM0 where c ∈ R 3 stands for RGB color, scalar r for brush radius, and tuple (x i, y i, p i) for an anchor point on the stroke, consisting of x, y coordinate and pressure p, and n is the maximum number of points in a single stroke, in this case, 16. These values are normalized such that the coordinates correspond to the default OpenGL coordinate system. DISPLAYFORM1 for k = 1, 2, 3 and i = 1, 2, · · ·, n. We used absolute coordinates for each point. It is notable that compared to the QuickDraw BID7 dataset which contains longer lines, our strokes consist of much fewer points. We consider many trajectory points redundant since the stroke lines can be fitted by spline curves with fewer anchor points. For example, to fit a straight line, only two end-points are needed regardless of the length, in other words, stroke curves are usually scaleinvariant. However, if we are to sample the data from a user input, we could have dozens of points along the trajectory. Hence we made the assumption of being able to represent curves with a few anchors. We later showed that a single stroke with only 16 anchors is able to fit most MNIST digits and generate twisted lines in Section 5. We further assumed that longer and more complicated lines can be decomposed into simple segments and extended our experiments to include recurrent drawing of multiple strokes to generate more complex drawings. The outline of the StrokeNet architecture is shown in FIG0. The generator takes s as input, and projects the stroke data with two MLPs. One is the position encoder which encodes (x i, y i, p i) into 64 × 64 feature maps, the other, brush encoder encodes the color and radius of the brush to a single 64 × 64 feature map. The color c is a single gray scale scalar whose value equals to 1 3 3 k=1 c k, while color strokes are approximated by channel mixing described in Section 3.4. The features are then concatenated and passed to the (de)convolution layers. To preserve the sequential and pressure information of each point (x i, y i, p i), the position encoder first maps (x i, y i) to the corresponding position onto a 64 × 64 matrix by putting a bright dot on that point. This is modeled by a 2D Gaussian function with its peak scaled to 1, which simplifies to: DISPLAYFORM0 for i = 1, 2, · · ·, n where the value is calculated for each point (x, y) on the 64 × 64 map. Denote this mapping from (x i, y i) to R 64×64 as pos: DISPLAYFORM1 By multiplying the corresponding pressure p i, we now have n position features, in our setup, sixteen. This part of the generator is trained separately with random coordinates until it generates accurate and reliable signals. However, if we directly feed these features into the (de)convolutional layers of the network, the generator fails partly due to the sparsity of the single brightness feature. Instead, we take every two neighbouring feature maps and add them together (denoted by "reduce" in FIG0 .), DISPLAYFORM2 Now, each feature map f i represents a segment of the stroke. By learning to connect and curve the n − 1 "segments", we are able to reconstruct the stroke. By appending the encoded color and radius data we now have the feature with shape 64 × 64 × n. We then feed the features into three (de)convolutional layers with batch-normalization BID12 activated by LeakyReLU BID28. The last layer is activated by tanh. The agent is a VGG BID25 )-like CNN that encodes the target image into its underlying stroke representation s. Three parallel FC-decoders with different activations are used to decode position (tanh), pressure (sigmoid) and brush data (sigmoid) from the feature. We used average-pooling instead of max-pooling to improve gradient flow. For the recurrent version of StrokeNet, two separate CNNs are trained for the target image and the drawing frame, as shown in FIG1. In practice the target image feature is computed once for all steps. We first built a painting software using JavaScript and WebGL. We later tailored this web application for our experiment.2 The spline used to fit the anchor points is centripetal CatmullRom BID2 BID1. A desirable feature about Catmull-Rom spline is that the curve goes through all control points, unlike the more commonly used Bezier curve BID24.We then interpolate through the sampled points and draw circles around each center point as shown in Figure 3. For each pixel inside a circle, its color depends on various factors including attributes of the brush, blending algorithm, etc. Our generator is trained on the naive brush. When it comes to the color blending of two frames, the generator is fed with the mean value of input RGB color as a gray scale scalar, and its output is treated as an alpha map. Normalization and alpha-blending is then performed to yield the next color frame, to simulate real blending algorithm underlying the software. Denote the generator output at time-step t by q (t) ∈ R 256×256, the frame image by r (t) ∈ R 3×256×256, RGB color of the brush by c ∈ R 3, the blending process is approximated as follows, DISPLAYFORM0 DISPLAYFORM1 for k = 1, 2, 3 corresponding to the RGB channels, where J denotes a 256 × 256 all-one matrix. For the generator, we synthesize a large amount of samples, each of length n. We would like to capture both the randomness and the smoothness of human writing, thus it is natural to incorporate chaos, most notably, the motion of three-body BID19 ). Figure 3: Illustration of how a stroke is rendered. Figure 4: Images from our three-body dataset. There is no closed-form solution to three-body problem, and error accumulates in simulation using numerical methods, leading to unpredictable and chaotic . We simulate three-body motion in space (z-component for pressure) with random initial conditions and sample the trajectories as strokes for our dataset. The simulation is done with a set of equations using Newton's universal law of gravitation: DISPLAYFORM0 where P i (i = 1, 2, 3, P i ∈ R 3) denotes the position of the three objects respectively, F i denotes the gravitational force exerted on the ith object. In our simulation we set mass m 1 = m 2 = m 3 = 1 and gravitational constant G = 5 × 10 −5. We also always keep our camera (origin point) at the center of the triangle formed by the three objects to maintain relatively stable "footage".Using this method we collected about 600K images since there is virtually no cost to generate samples. Samples from the dataset are shown in Figure 4. To prove the effectivess of our neural environment, we trained an agent to perform drawing task on several popular datasets, from characters to drawings, with the generator part frozen. For MNIST and Omniglot, we trained an agent to draw the characters within one stroke. We later trained the recurrent StrokeNet on more complex datasets like QuickDraw and KanjiVG BID20. We resized all the input images to 256 × 256 with anti-alias and paddings. At first we train the position encoder guided by function pos that maps a coordinate to a 64×64 matrix with l 2 distance to measure the loss. Next we freeze the position encoder and train the other parts of the generator, again with l 2 loss to measure the performance on the three-body dataset. It can be found that smaller batch size in more accurate images. We trained the generator with a batch size of 64 until the loss no longer improves. We then set the batch size to 32 to sharpen the neural network. To train the agent, we freeze the generator. Denote the agent loss as l agent, the generated image and ground-truth image as i gen and i gt respectively, the loss is defined as: DISPLAYFORM0 where DISPLAYFORM1 T is the data describing the kth anchor point on the stroke. Here the summation term constrains the average distance between neighbouring points, where λ denotes the penalty strength. If we drop this term, the agent fails to learn the correct order of the points in a stroke because the generator itself is, after all, not robust to all cases of input, and is very likely to produce wrong for sequences with large gaps between neighbouring points. All experiments are conducted on a single NVIDIA Tesla P40 GPU. We first experimented with single step StrokeNet on MNIST and Omniglot, then we experimented recurrent StrokeNet with QuickDraw and Kanji. For the MNIST dataset, we later enforced a Gaussian prior to the latent variable and explored the latent space of the agent by linear interpolation. Finally, for a quantitative evaluation of the model, we trained a classifier on MNIST, and tested the classifier with images generated by the agent. The close accuracies indicate the quality of the generated images. It can be seen that a single stroke provides rich expressive power for different shapes. The generator generalizes well to unseen stroke patterns other than the synthesized three-body dataset. On the Omniglot dataset, since many characters consist of multiple strokes while the agent can only draw one, the agent tries to capture the contour of the character. For more complex datasets, multiple steps of strokes are needed. Again the agent does pretty well to capture the contour of the given image. However, it seems that the agent has trouble to recover the details of the pictures, and tends to smear inside the boundaries with thick strokes. To convert the agent into a latent space generative model, we experimented with the VAE version of the agent, where the feature obtained from the last layer of CNN is projected into two vectors representing the means µ and standard deviations (activated by softplus) σ, both of 1024 dimensions. A vector noise of i.i.d. Gaussian U ∼ N (0, I) is sampled, latent variable z is given by DISPLAYFORM0 We did latent space interpolation with the agent trained on MNIST. The simple data led to easily interpretable . Since the images are generated by strokes, the digits transform smoothly to one another. That is to say, the looked as if we were directly interpolating the stroke data. Results are shown in Figure 9 and 10. In order to evaluate the agent, we trained a 5-layer CNN classifier solely on pre-processed MNIST dataset, which is also the input to the MNIST agent. The size of the image is 256 × 256, so there is some performance drop to the classification task compared to standard 28 × 28 images. The classifier is then used to evaluate the paired test-set image generated by the agent. The accuracies reflect the quality of the generated images. We also compared the l 2 loss with SPIRAL on MNIST to illustrate that our method has the advantage of faster convergence over reinforcement learning approaches, shown in FIG0. Pre-processed images 90.82% Agent Output (3 steps) 88.43% Agent Output TAB0 79.33% Agent Output (1 step, VAE) 67.21% (a) (b) FIG0: Comparison of stroke orders between human and agent. We can see the stroke order is completely chaotic compared to natural order. For future work, there are several major improvements we want to make both to the network structure and to the algorithm. The recurrent structure adopted here is of the simplest form. We use this setup because we consider drawing as a Markov process, where the current action only depends on what the agent sees, the target image and the previous frame. More advanced structures like LSTM BID10 or GRU BID3 may boost the performance. A stop sign can be also introduced to determine when to stop drawing, which can be useful in character reconstruction. For the agent, various attention mechanism could be incorporated to help the agent focus on undrawn regions, so that smear and blurry scribbles might be prevented. Secondly, The generator and the agent were trained as two separate parts throughout the experiment. We can somehow train them as a whole: during the training of the agent, store all the intermediate stroke data. After a period of training, sample images from the real environment with the stroke data just collected, and train the generator with the data. By doing so in an iterative manner, the generator could fit better to the current agent and provide more reliable reconstructions, while a changing generator may potentially provide more valuable overall gradients. It is also found useful to add a bit of randomness to the learning rate. Since different decoders of the agent learn at different rates, stochasticity in more appealing . For example, the agent usually fails to generalize to color images because it always sticks with one global average color (as shown in FIG0). However, it sometimes generates appealing with some randomness added during the training. As a of this immobility, the way agent writes is dull compared to humans and reinforcement learning agents like SPIRAL. For instance, when writing the digit "8", the agent is simply writing "3" with endpoints closed. Also, the agent avoids to make intersecting strokes over all datasets, although such actions are harmless and should be totally encouraged and explored! Thus, random sampling techniques could be added to the decision making process to encourage bolder moves. Finally, for the evaluation metrics, the naive l 2 loss can be combined with adversarial learning. If paired sequential data is available, we believe adding it to training will also improve the . In this paper we bring a proof-of-concept that an agent is able to learn from its neural simulation of an environment. Especially when the environment is deterministic given the action, or contains a huge action space, the proposed approach could be useful. Our primary contribution is that we devised a model-based method to approximate non-differentiable environment with neural network, and the agent trained with our method converges quickly on several datasets. It is able to adapt its skills to real world. Hopefully such approaches can be useful when dealing with more difficult reinforcement learning problems. T denote the coordinate of a sampled point. For a curve defined by points P 0, P 1, P 2, P 3, the spline can be produced by: DISPLAYFORM0 where DISPLAYFORM1 with α = 0.5, t 0 = 0 and i = 0, 1, 2, 3By interpolating t from t 1 to t 2 linearly, we generate the curve between P 1 and P 2. The pressure values between neighbouring points are interpolated linearly. The agent loss equals to the l 2 distance between the generator output and agent input plus the penalty term constraining the average point distance within a stroke. For (c) and (d) the learning rate is set to 10 −4, batch size equals to 64. Figure 15: A trained StrokeNet generates images that resemble the output of painting software. The first row depicts generated by our model (left) and by the software (right) given the same input. The second row shows the model could produce strokes with color and texture using simple arithmetic operations. The third and fourth row shows the model's ability to draw MNIST digits (left) on both its own generative model (middle) and real-world painting software (right).

Represent the short summary for finding the corresponding scientific article (long)

StrokeNet is a novel architecture where the agent is trained to draw by strokes on a differentiable simulation of the environment, which could effectively exploit the power of back-propagation.

represent this text to find a much much longer article about it with say hundreds-thousands of words:

To understand the information stored in the latent space, we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space.


Question: Represent the article for finding a single-sentence TLDR

Question: We demonstrate how machine learning is able to model experiments in quantum physics. Quantum entanglement is a cornerstone for upcoming quantum technologies such as quantum computation and quantum cryptography. Of particular interest are complex quantum states with more than two particles and a large number of entangled quantum levels. Given such a multiparticle high-dimensional quantum state, it is usually impossible to reconstruct an experimental setup that produces it. To search for interesting experiments, one thus has to randomly create millions of setups on a computer and calculate the respective output states. In this work, we show that machine learning models can provide significant improvement over random search. We demonstrate that a long short-term memory (LSTM) neural network can successfully learn to model quantum experiments by correctly predicting output state characteristics for given setups without the necessity of computing the states themselves. This approach not only allows for faster search but is also an essential step towards automated design of multiparticle high-dimensional quantum experiments using generative machine learning models. In the past decade, artificial neural networks have been applied to a plethora of scientific disciplines, commercial applications, and every-day tasks with outstanding performance in, e.g., medical diagnosis, self-driving, and board games . In contrast to standard feedforward neural networks, long short-term memory (LSTM) architectures have recurrent connections, which allow them to process sequential data such as text and speech . Such sequence-processing capabilities can be particularly useful for designing complex quantum experiments, since the final state of quantum particles depends on the sequence of elements, i.e. the experimental setup, these particles pass through. For instance, in quantum optical experiments, photons may traverse a sequence of wave plates, beam splitters, and holographic plates. Highdimensional quantum states are important for multiparticle and multisetting violations of local realist models as well as for applications in emerging quantum technologies such as quantum communication and error correction in quantum computers . Already for three photons and only a few quantum levels, it becomes in general infeasible for humans to determine the required setup for a desired final quantum state, which makes automated design procedures for this inverse problem necessary. One example of such an automated procedure is the algorithm MELVIN, which uses a toolbox of optical elements, randomly generates sequences of these elements, calculates the ing quantum state, and then checks whether the state is interesting, i.e. maximally entangled and involving many quantum levels. The setups proposed by MELVIN have been realized in laboratory experiments b). Recently, also a reinforcement learning approach has been applied to design new experiments . Inspired by these advances, we investigate how LSTM networks can learn quantum optical setups and predict the characteristics of the ing quantum states. We train the neural networks using millions of setups generated by MELVIN. The huge amount of data makes deep learning approaches the first choice. We use cluster cross validation to evaluate the models. Let us consider a quantum optical experiment using three photons with orbital angular momentum (OAM) (; a). The OAM of a photon is characterized by an integer whose size and sign represent the shape and handedness of the photon wavefront, respectively. For instance, after a series of optical elements, a three particle quantum state may have the following form: This state represents a physical situation, in which there is 1/4 chance (modulus square of the amplitude value 1/2) that all three photons have OAM value 0 (first term), and a 1/4 chance that photons 1 and 3 have OAM value 1, while photon 2 has OAM value 0 (second term), and so on for the two remaining terms. We are generally interested in two main characteristics of the quantum states: Are they maximally entangled? Are they high-dimensional? The dimensionality of a state is represented by its Schmidt rank vector (SRV). State |Ψ is indeed maximally entangled because all terms on the right hand side have the same amplitude value. Its SRV is, as the first photon is four-dimensionally entangled with the other two photons, whereas photons two and three are both only two-dimensionally entangled with the rest. A setup is labeled "positive" (y E = 1) if its output state is maximally entangled and if the setup obeys some further restrictions, e.g., behaves well under multi-pair emission, and otherwise labeled "negative" (y E = 0). The target label capturing the state dimensionality is the SRV y SRV = (n, m, k). We train LSTM networks to directly predict these state characteristics (entanglement and SRV) from a given experimental setup without actually predicting the quantum state itself. For classification, we use binary cross entropy (BCE) in combination with logistic sigmoid output activation for learning. For regression, it is always possible to reorder the photon labels such that the SRV has entries in non-increasing order. An SRV label is thus represented by 3-tuple y SRV = (n, m, k) which satisfies n ≥ m ≥ k. With slight abuse of notation, we model n ∼ P(λ) as a Poisson-distributed random variable and m ∼ B(n, p), k ∼ B(m, q) as Binomials with ranges m ∈ {1, . . . n} and k ∈ {1, . . ., m} and success probabilities p and q, respectively. The ing log-likelihood objective (omitting all terms not depending on λ, p, q) for a data point x with label (n, m, k) is whereλ,p,q are the network predictions (i.e. functions of x) for the distribution parameters of n, m, k respectively. The Schmidt rank value predictions aren =λ,m =pλ,k =pqλ. To see this, we need to consider the marginals of the joint probability mass function To obtain the marginal distribution of m, we can first sum over all possible k, which is easy. To sum out n we first observe that n m = 0 for n < m, i.e. the first m terms are zero and we may write Sequence processing model for a many-to-one mapping. The target valueŷ can be either an estimate for y E (entanglement classification) or y SRV (SRV regression). capturing only non-zero terms. It follows that which is P(pλ)-distributed. Using the same argument for k we get that the marginal of k is P(pqλ)-distributed. The estimatesn,m,k are obtained by taking the means of their respective marginals. The used sequence processing model is depicted in Figure 1. We train two networks, one for entanglement classification (target y E), and one for SRV regression (target y SRV). The reason why we avoid multitask learning in this context is that we do not want to incorporate correlations between entanglement and SRV into our models. For instance, the SRV was only observed in nonmaximally entangled samples so far, which is a perfect correlation. This would cause a multitask network to automatically label such a sample as negative only because of its SRV. By training separate networks we lower the risk of incorporating such correlations. A setup of N elements is being fed into a network by its sequence of individual optical components x = (x 1, x 2, ..., x N), where in our data N ranges from 6 to 15. We use an LSTM with 2048 hidden units and a component embedding space with 64 dimensions. The component embedding technique is similar to word embeddings . The dataset produced by MELVIN consists of 7,853,853 different setups of which 1,638,233 samples are labeled positive. Each setup consists of a sequence x of optical elements, and the two target values y E and y SRV. We are interested in whether the trained model is able to extrapolate to unseen SRVs. Therefore, we cluster the data by leading Schmidt rank n. Figure 2 shows the the number of positive and negative samples in the data set for each n. All samples with n ≥ 9 are moved to a special extrapolation set consisting of only 1,754 setups (gray cell in Table 1). The remainder of the data, i.e. all samples with n < 9, is then divided into a training set and a conventional test set with 20 % of the data drawn at random (iid). This workflow is shown in Figure 3. 0,1 0,1 2 3 4 5 6 7 8 9-12 Table 1: Cluster cross validation folds and extrapolation set characterized by leading Schmidt rank n. Samples with n = 0 and samples with n = 1 are combined and then split into two folds at random. The test set is used to estimate the conventional generalization error, while the extrapolation set is used to shed light on the ability of the learned model to perform on higher Schmidt rank numbers. If the model extrapolates successfully, we can hope to find experimental setups that lead to new interesting quantum states. Cluster cross validation (CCV) is an evaluation method similar to standard cross validation. Instead of grouping the folds iid, CCV groups them according to a clustering method. Thus, CCV removes similarities between training and validation set and simulates situations in which the withheld folds have not been obtained yet, thereby allowing us to investigate the ability of the network to discover these withheld setups. We use CCV with nine folds (white cells in Table 1). Seven of these folds correspond to the leading Schmidt ranks 2,..., 8. The samples with n = 1 (not entangled) and n = 0 (not even a valid three-photon state) are negative by definition. These samples represent special cases of y E = 0 setups and it is not necessary to generalize to these cases without training on them. Therefore, the 4,300,268 samples with n < 2 are divided into two folds at random such that the model will always see some of these special samples while training. Let us examine if the LSTM network has learned something about quantum physics. A good model will identify positive setups correctly while discarding as many negative setups as possible. This behavior is reflected in the metrics true positive rate TPR = TP/(TP + FN) and true negative rate TNR = TN/(TN + FP), with TP, TN, FP, FN the true positives, true negatives, false positives, false negatives, respectively. A metric that quantifies the success rate within the positive predictions is the hit rate (i.e. precision or positive predicted value), defined as HR = TP/(TP + FP) . For each withheld CCV fold n, we characterize a setup to be "interesting" when it fulfills the following two criteria: (i) It is classified positive (ŷ E > τ) with τ the classification threshold of the sigmoid output activation. (ii) The SRV predictionŷ SRV = (n,m,k) is such that there exists a y SRV = (n, m, k) with y SRV −ŷ SRV 2 < r. We call r the SRV radius. We denote samples which are classified as interesting (uninteresting) and indeed positive (negative) as true positives We split the entire data by their leading Schmidt rank n. All samples with n ≥ 9 constitute the extrapolation set, which we use to explore the out-of-distribution capabilities of our model. For the remaining samples (i.e. n < 9) we make a random test split at a ratio of 1/4. The test set is used to estimate the conventional generalization error of our model. We use the training set to perform cluster cross validation. (negatives). And we denote samples which are classified as interesting (uninteresting) and indeed negative (positive) as false positives (false negatives). We employ stochastic gradient descent for training the LSTM network with momentum 0.5 and batch size 128. We sample mini-batches in such a way that positive and negative samples appear equally often in training. For balanced SRV regression, the leading Schmidt rank vector number n is misused as class label. The models were trained using early stopping after 40000 weight update steps for the entanglement classification network and 14000 update steps for the SRV regression network. Hyperparameter search was performed in advance on a data set similar to the training set. Figure 4 shows the TNR, TPR, and rediscovery ratio for sigmoid threshold τ = 0.5 and SRV radius r = 3. The rediscovery ratio is defined as the number of distinct SRVs, for which at least 20% of the samples are rediscovered by our method, i.e. identified as interesting, divided by the number of distinct SRVs in the respective cluster. The TNR for fold 0,1 is 0.9996, and the hit rate HR on the extrapolation set 9-12 is 0.659. Error bars in Figure 4 and later in the text are 95 % binomial proportion confidence intervals. Model performance depends heavily on parameters τ and r. Figure 5 shows the "beyond distribution" for a variety of sigmoid thresholds and SRV radii. Figure 5: True negative rate (scale starts at 0.6), true positive rate, rediscovery ratio, and hit rate for the extrapolation set 9-12 for varying sigmoid threshold τ and SRV radius r. For too restrictive parameter choices (τ → 1 and r → 0.5) the TNR approaches 1, while TPR and rediscovery ratio approach 0, such that no interesting new setups would be identified. For too loose choices (small τ, large r), too few negative samples would be rejected, such that the advantage over random search becomes negligible. For a large variety of τ and r the models perform satisfyingly well, allowing a decent compromise between TNR and TPR. This is reflected in large values for the hit rate, which is 0.736 on average over all depicted thresholds. Finally, we investigate the conventional in-distribution generalization error using the test set (20 % of the data). Entanglement classification: The entanglement training BCE loss value is 10.2. TNR and TPR are 0.9271 ± 0.00024 and 0.9469 ± 0.00041, respectively. The corresponding test error is 10.4. TNR and TPR are 0.9261 ± 0.00038 and 0.9427 ± 0.00065, respectively. SRV regression: The SRV training loss value according to Equation is 2.247, the accuracy with r = 3 is 93.82 % and the mean distance between label and prediction is 1.3943. The SRV test error is 2.24, the accuracy with r = 3 is 0.938 and the mean distance between label and prediction is 1.40. These figures are consistent with a clean training procedure. Our experiments demonstrate that an LSTM-based neural network can be trained to model certain properties of complex quantum systems. Our approach is not limited to entanglement and Schmidt rank but may be generalized to employ other objective functions such as multiparticle transformations, interference and fidelity qualities, and so on. Another possible next step to expand our approach towards the goal of automated design of multiparticle high-dimensional quantum experiments is the exploitation of generative models. Here, we consider Generative Adversarial Networks (GANs) and beam search as possible approaches. Generating sequences such as text in adversarial settings has been done using 1D CNNs and LSTMs . The LSTM-based approaches employ ideas from reinforcement learning to alleviate the problem of propagating gradients through the softmax outputs of the network. Since our data is in structure similar to text, these approaches are directly applicable to our setting. For beam search, there exist two different ideas, namely a discriminative approach and a generative approach. The discriminative approach incorporates the entire data set (positive and negative samples). The models trained for this work can be used for the discriminative approach in that one constructs new sequences by maximizing the belief of the network that the outcome will be a positive setup. For the generative approach, the idea is to train a model on the positive samples only to learn their distribution via next element prediction. On inference, beam search can be used to approximate the most probable sequence given some initial condition . Another option to generate new sequences is to sample from the softmax distribution of the network output at each sequence position as has been used for text generation models . In general, automated design procedures of experiments has much broader applications beyond quantum optical setups and can be of importance for many scientific disciplines other than physics. We have shown that an LSTM-based neural network can be trained to successfully predict certain characteristics of high-dimensional multiparticle quantum states from the experimental setup without any explicit knowledge of quantum mechanics. The network performs well even on unseen data beyond the training distribution, proving its extrapolation capabilities. This paves the way to automated design of complex quantum experiments using generative machine learning models.

We demonstrate how machine learning is able to model experiments in quantum physics.

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.