Question: Represent the article for finding a single-sentence TLDR

Question: An unintended consequence of feature sharing is the model fitting to correlated tasks within the dataset, termed negative transfer. In this paper, we revisit the problem of negative transfer in multitask setting and find that its corrosive effects are applicable to a wide range of linear and non-linear models, including neural networks. We first study the effects of negative transfer in a principled way and show that previously proposed counter-measures are insufficient, particularly for trainable features. We propose an adversarial training approach to mitigate the effects of negative transfer by viewing the problem in a domain adaptation setting. Finally, empirical on attribute prediction multi-task on AWA and CUB datasets further validate the need for correcting negative sharing in an end-to-end manner. Advances in machine learning have led to proficient supervised learning models with powerful representations in various prediction tasks. We now expect an ideal classification model to restrict itself to a pertinent set of evidences available to it from the input for prediction. Further, we expect the model to disregard any unrelated evidences in the data to enable better generalization. Figure 1: A supervised classifier'cheetah vs. snow-leopard' that uses unrelated evidence (of habitat) over relevant evidence (of fur patterns). As shown by the pixel importance maps, the model suffers from the negative transfer prevalent in a typical animal image dataset skewed towards the animal's typical habitat and fails to generalize to rare samples. Let us consider the task of training an animal classifier "cheetah vs. snow-leopards" from a dataset of images of these animals, such as those illustrated in Figure 1 -a task which ideally should focus on the animal's appearance features. However, a large portion of these images also contain various cues of the typical habitat of the animals in the , i.e., tall grass and snow (see Figures 1 (a) and (b)) which are, in principle, unrelated to the animal's appearance. An archetypal model is deceived by the co-occurrence of such unrelated, yet easily detectable cues of habitat over the animal's appearance features such as complex fur patterns. However, a proficient supervised learning model must identify relevant evidences for the label of interest and at the same time discard various unrelated evidences such as presence of snow, even though it tends to co-occur frequently with snow-leopard. Consequently, it would be more likely that such a model would perform better on rare-instances (such as those in Figures 1 (c) and (d)) and generalize better to unseen instances. This phenomenon of co-occurring but unrelated evidences being present in training data and thereby having a debilitating effect on model performance has been described in literature BID8; BID16; BID9; BID13; BID15 ). These techniques utilize the easy availability of labels for unrelated evidences (e.g. habitat labels above), called negative labels which constitutes an auxiliary task, and seek to mitigate its debilitating performance on the primary task (e.g. animal classification above) with techniques referred to as negative-sharing or negative-transfer. While all of these techniques have tackled this problem utilizing various forms of regularization, we describe several shortcomings of this class of approaches, most notable of which is their inapplicability to the popular paradigm of trainable features obtained via neural representation learning. Motivated by these limitations, in this paper we depart from the direction of regularization-based approaches and examine methods inspired from a domain-adaptation viewpoint to propose an adversarial training-based formulation. We uniquely view such a scenario as an instance of adversarial multi-task learning, where the classification tasks are either the primary task of interest (i.e., predicting the presence of fur pattern and color) or the auxiliary negative tasks (i.e., characteristics of habitat) to be avoided. Since the 2 tasks are unrelated, any label correlation between primary and auxiliary labels in the training data is only by chance and therefore from a domain-adaptation perspective, we envision a target-domain as possibly having a different correlation between the primary and auxiliary labels. The effects of negative transfer are hence mitigated when the classification task is trained in this domain. We discuss advantages of our proposed formulation, inspired from domain-adaptation, to alleviate the negative transfer over existing techniques, including ready applicability to neural networks in an end-to-end fashion. It must be noted that, while the formulation of the problem is motivated with multi-task learning, negative-transfer is a disposition of any supervised learning task from simple binary classification to recent popular supervised tasks such as image detection, captioning, or visual dialog. We present motivating literature that prelude this work next. Image classification literature often characterize mid-level features as attribute predictors in multilabel learning. The inability of models to learn predictors with the correct semantic concept is attributed to negative transfer. To our knowledge, the predominent approach to tackle negative transfer in such setting was the use of specific regularizers BID9 BID8. Specifically, the primary model avoid using features which are important for the auxiliary task, leading to models competing for features. BID8 further extends this idea to attribute groups, where feature selection sparsity is induced across group, but encourage within them. We highlight three limitations of feature competition techniques below:• Repeated features: Consider the simple scenario where some features in the feature representations are repeated or dependent on others. Feature competition enforces tasks to pick unique features, however they are implicitly the same. Going back to the case of'cheetah vs snow leopard' example mentioned earlier, when there are two copies of a feature which captures'snow in the ', then both primary and auxiliary classifiers would just pick different copies of that feature. Here, the idea of resisting cheetah/snow leopard classifier from picking features which capture snow is negated.• Trainable features: Neural representations have become extremely popular owing the power of learning trainable features. However, feature competition techniques fail with prediction models with trainable features. If there are features that are important for both primary and auxiliary task, models involving a trainable feature setup will lead to duplicating the same features, thereby ing in the repeated feature scenario.• Easy auxiliary task with abundant features: Consider a scenario of an easy auxiliary task, that does not require a large number of features to be predicted correctly. Similar to the previous case, the spared features from the auxiliary task can be picked by the primary task ing in negative transfer. Motivated by these shortcomings, we proceed to examine the negative transfer problem in a domain adaptation setting. In this section we first formalize and explain the problem setting of negative transfer. We present scenarios where earlier attempts (regularization-based approaches) to tackle this problem fails. We then explain our formulation for posing then negative transfer problem in a domain adaptation setting and derive adversarial learning algorithms to solve negative transfer. Lastly, we present a thorough analysis of the proposed formulation to address negative transfer by experimenting on a carefully designed synthetic dataset. In typical 2-label classification problem, we assume that training data and all future test examples are drawn i.i.d from a probability distribution D on instance space X and a labeling function f p: X → {+1, −1} for the primary task labels and f a: X → {+1, −1} for the auxiliary task labels. Every instance x ∈ X has primary and auxiliary labels: y p and y a respectively. The goal is to learn a classifier which performs well on future samples from D, which may have a different label correlation since the tasks are unrelated. Formally, we capture this label correlation via a joint label distribution DISPLAYFORM0, and we assume that P (Y) in training and test are different. This problem setup is different from earlier works on negative transfer in the way how unrelated tasks are defined. We define unrelated tasks as the ones which can have different correlation in training and test data. In unrelated tasks is referred as tasks which are defined over orthogonal sets of features. BID17 uses the term negative correlation for unrelatedness among tasks. Two tasks are negatively correlated when one feature is deemed to be important to first task makes it unlikely to be important to the second task. Let the instances in training data be drawn from source distribution D S. The aim is to train a classifier on this which performs well on instances drawn from target distribution D T. Instances in D S has primary and auxiliary labels correlated with joint label distribution DISPLAYFORM0 We assume that the labelling function f p and f a are universal and common for both source and target domains. Typical unsupervised domain adaptation setting has labelled instances from source distribution and also unlabelled instances from target distribution. However, we have no information (neither labels nor instances) about the target distribution. The only information we have about the target domain is P T (Y). We are either provided an estimate of P T (Y) from an oracle, or we make an assumption on P T (Y) (for instance, a uniform distribution over the space of Y).Consider U T, the space of all distributions over X which has label distribution P T (Y). It is extremely challenging to adapt from the source domain to all members of U T with given P T (Y). However, we can settle for a particular D T. We pick such a D T ∈ U T as that distribution over X which is nearest to D S. We model this in terms of KL divergence as a constrained optimization as follows. DISPLAYFORM1 Before explaining the solution to the above optimization problem, we show the relationship between D and P (Y). To articulate our intuition, we use Figure 3 (a) which illustrates a sample source distribution D S over 2D instance space (green). The labelling functions (in this case, f p and f a) partitions the space of X into regions such that each region corresponds to a label combination (y = {y p, y a}). We denote each of these regions as R y. Notice that P (Y = y) becomes the integral of D over the region R y.Let φ S and φ T be density functions of D S and D T respectively. Then, the above optimization problem becomes: DISPLAYFORM2 Where ∆ ⊃ U T is the set of all distributions over X and region R y = {x : DISPLAYFORM3 The Lagrangian equivalent for the above problem can be stated as, DISPLAYFORM4 where λ y ∈ R. Using the Euler-Lagrange equation to solve above problem, DISPLAYFORM5 Intuitively we find that D T is a scaled version of D S with scaling factor as the ratio of P T (Y) and P S (Y). Note that this scaling factor may vary for different x ∈ X depending on which region R y it falls in. This is depicted in Figure. 3(b) which is a target distribution derived from source distribution in 3(a). The regions R −1,+1 and R +1,−1 are scaled up whereas regions R +1,+1 and R −1,−1 are scaled down. Though the above derivation is for two labels, one can see that Eq. 3 extends to any number of labels in y. As mentioned earlier in this section, we have no instances from target domain. However φ T (x) allows us to assign soft domain labels to the source domain instances, which indicates the probability of that instance belonging to the target domain. Specifically, let y D be the binary label indicating if an instance belongs to target domain. Then, DISPLAYFORM6 Two possible assumptions that could be made on P T (Y) (if not provided) are uniform distribution over space of y or uncorrelated (independent labels). Next we present two methods that leverage these soft domain labels. In the previous section we modeled negative transfer as a domain adaptation problem with soft domain labels. Next, we present methods to leverage the domain adversarial neural network (DANN) by BID3 to induce domain-invariance features. These methods are based on the theoretical analysis of domain adaptation by BID1 BID0. They provide a generalization bound on target error as following:Theorem 1 (Ben-David et al. FORMULA2) Let h ∈ H be a prediction model of the form h: X → {−1, +1} where H is the hypothesis space. Let S and T be generalization errors for source (D S) and target (D T) domains. For any classifier h ∈ H DISPLAYFORM0 Above theorem states that the target error is bounded by sum of source error and distance between source and target distributions. d H∆H can be seen as the maximum accuracy of classifying source and target domain by using a classifier from hypothesis space H∆H. Further, any classifier g ∈ H∆H is the function XOR of some h, h ∈ H. DANN introduced an objective function which minimizes both source error as well as divergence between domains. Divergence between the domains can be looked at as the prediction accuracy of domain classifier. DANN models starts with a mapping J f: X → R d with a parameter θ f, which projects instances to a latent representation space, we call this the feature extractor. These features are then mapped to primary label space with a mapping function J y (label predictor) with parameter θ y. The same features from latent representation space are mapped to domain label by J d (domain classifier), with parameter θ d. We denote the training set as DISPLAYFORM1, with every instance we are provided with a primary label y p and a soft domain label y D. The objective here is to find a feature extractor J f which projects instances to a latent representation space where achievable label prediction accuracy is high and domain prediction accuracy is low. Let L y (θ f, θ y) be the prediction loss for label prediction and let L d (θ f, θ d) be that for domain classifier, then objective function is DISPLAYFORM2 where DISPLAYFORM3 However, in using this formulation together with aforementioned soft domain labels, in a weak J D since the soft domain labels are highly skewed towards the source domain. In such settings, DISPLAYFORM4 We address this issue again in Section 4.The effort is to make J f provide a latent representation that is unable to discriminate source and target domain. As the soft domain labels are assigned according to y = {y p, y a} (see Eq. 4), if that latent representation can be used to correctly predict y p and y a with label predictors h p ∈ H and h a ∈ H, then there exists a g ∈ H∆H which can predict source and target domains well (with g(x) = h p (x) ⊕ h a (x)). Conversely, if the representations cannot be used to predict both the labels y p and y a correctly implies poor performance on domain classification. From this observation, we propose to replace domain classification loss (second loss term of Eq. 7) with auxiliary label classifier loss L a (θ f, θ a), where θ a is the parameter for the auxiliary label classifier J a. We solve the optimization problem in Eq. 7 by using gradient reversal layer BID3. Gradient reversal layer multiplies the gradient by a negative constant during the backpropagation. We call the auxiliary label classifier as an adversarial to the primary classifier. In this form, our idea is closely related to Zhang et al. FORMULA2 We extend this two label scenario to multilabels, by partitioning labels into groups, such that related labels are together. If a label is unrelated to all other labels than it forms a singleton group. We propose a model architecture, with a latent representation (with J f) for each group. Further, the latent representation of a given group must be unable to predict any task from all other groups. We achieve this by adding all label classification other than the group member as the adversarial. In the next section, we showcase the empirical performance of multitask models trained in this way. Another interesting alternative that departs from the usual view of feature projections is to utilize feature selection for J f (feature extractor). We discuss this alternative next. An intuitive method to prevent negative transfer between correlated features is to use a feature selection approach to explicitly select relevant features appropriate for a task. We use Recursive Feature Elimination (RFE) BID5 for the task of feature selection using CNN as the classifier, since wrapper method (J.) is computationally effective and noticeably time consuming when deep nets are used as the classifier. Recursive feature selection BID5 consider all the features and eliminated them at each iteration till the desired criterion is met. At each iteration, the current feature set is used to evaluate a task using a classifier. Each of the features obtain a score from the classifier, based on which one or more features are eliminated from the set. This step is repeated until the criterion is met, which can be in terms of the final number of features to be chosen or desired classification performance to be attain. In order to perform RFE, we need to score each of the feature based on its effect on the classifier. For classifiers like logistic regression, the feature importance can be obtained by the weights assigned by the classifier for each of the features. In adversarial settings for multilabel classification, importance of the k th feature can be calculated as: DISPLAYFORM0 where, |Y p | and |Y a | denote the number of labels to be predicted for the primary and the auxiliary task respectively. We study the adverse effects of negative transfer and how our proposed algorithms could resist this negative transfer, on synthetic data. Our synthetic dataset consists of 10-dimensional feature vectors, with two binary labels y p and y a (primary and auxiliary). We generate the data from a generative system with one class label distribution for training set and one for test set. First 5 features are sampled according to the primary label from a mixture of two Gaussian distributions with identity covariance matrix. If the primary label is 1 then 5 features are sampled from first Gaussian, otherwise from the second Gaussian. Similarly second set of 5 features are sampled same way from another set of mixture of two Gaussian distributions, this time sampled according to the auxiliary label. Note that for these experiments we are only interested in predicting primary label y p in the test set. In all the experiments we keep P (Y p = 1) = P (Y a = 1) = 0.5. We also Keep distance between 2 Gaussian distributions corresponding to primary label to fixed to 1.5. We use conditional label distribution P (Y p |Y a) as measure label correlation. We compare following algorithms in our experiments: baseline -A logistic regression classifier trained on primary label. Two sets of experiments where conducted to study the following aspects of negative transfer:• Difference between train and test label correlation: For this experiment, we trained a model on training set with P (Y p |Y a) = 0.8 and tested out it's performance on test set which only differs from training set in P (Y p |Y a). Figure 4.1(a) Shows the mean average precision of classifiers on test set. One can see that the baseline mAP drops as correlation in test set goes down. This indicates that baseline classifier captured wrong set of features for prediction. We can see that DadvC is performing marginally better than baseline. ALadvC and fs-adv are consistently performing well irrespective of varying label correlation in test. This shows that these methods picked the correct set of labels. • Easiness of auxiliary task: We fix P (Y p |Y a) = 0.8 for training set and P (Y p |Y a) = 0.5 for test. We vary the distance between Gaussian distribution corresponding to auxiliary label while keeping that of primary label fixed. By doing this we vary the easiness of auxiliary task. In Figure 4.1(b) we can see that, as the auxiliary task gets easier, more features related to auxiliary task will be used by the primary classifiers which in decreasing performance in test set. One can see that Both baseline and DadvC are performing bad as easiness increases. The proposed algorithms performs consistently better. We use the Animals-with-Attributes (AwA) datasets BID12 and Caltech-UCSD Birds 200-2011 (CUB) for the multilabel attribute prediction task. With both datasets we follow the experimental protocol from BID12. The Animals with Attributes-2 (AWA) consists of 30, 475 images of animals in natural settings, with 50 animals and 85 annotated attributes each. According to BID8, FORMULA13 into 10 relevant groups. The dataset is split across the animals as 27/13/10 for train, validation, and test respectively as done in BID12. The Caltech-UCSD Birds 200-2011 (CUB) consists of 11,788 images of birds, captured in natural settings, corresponding to 200 birds. The dataset also consists of 312 attributes of the birds annotated per image. They are aggregated into 28 groups corresponding to anatomical parts, and split across classes: 100/50/50 for train, validation and test over attributes, as done in BID12. The split ensures different correlations between attribute labels in the three splits, which highlights the problems of negative-transfer. We directly utilize the the feature representation (of length 2048) obtained from ResNet-101 BID6 model that is pre-trained on ImageNet BID2 for AwA and CUB. We explain below the specific architectures and parameters used for each experiment. Mean average precision (mAP) on validation and test sets are reported. Attribute prediction is an unbalanced prediction task, as individual attributes are rare. We use a balance corrected binary crossentropy loss function for all experiments, with the balance count obtained from the training set. Further, we utilize early stopping criteria based on the performance of the model on the validation set. LR, LR+FS and LR+FS-adv: As discussed in Section 3.3.1, one way to prevent the adverse effects of negative transfer is by selecting the optimal feature set for each of the task, based on the proposed adversarial objective function. We use Recursive Feature Elimination (RFE) BID5 method for the task of feature selection wrapped over a Logistic Regression (LR). We perform feature selection experimentation using Logistic Regression (instead of multi-layer perception, as The CUB dataset consists of 28 groups, increasing number of adversarial tasks from most correlated to least helps improve performance on test splits. done in next set of experiments) for time efficiency. For CUB, we transform the 2048 features into a 500 dimensional space by using a dense layer followed by ReLU, which is then used for feature selection. Based on the primary task at hand, the feature importance scores are calculated using Eq. 8, by using the learned weights of LR. At each iteration, we remove λ × 0.33 numbers of features, where we decrease λ at each iteration as: 0.95 × λ. The final number of features to be retained is decided using the validation based on the mAP. We also observe the performance of feature selection without the adversarial setting (LR+fs). In this case, we only select the features that perform well for the primary task, without considering any effect on the auxiliary tasks. Performances of RFE have been observed in both adversarial and non-adversarial settings, which is reported in TAB0, and group-wise test mAP in Figures 4(a) and 5 (a) respectively. As shown in TAB0, the mAP of RFE improves the baseline accuracy when LR has been used (compare rows 2 and 3 with row 1). The efficiency of the proposed criterion for scoring features, as shown by equation 8, can be observed when we compare between rows 2 and 3 in the Table. There is an improvement of 2.3% and 45% compared to LR (row 1), and 5.7% and 31.8% compared to LR+FS (w/o adv) (row 2) for AwA and CUB datasets respectively showcasing the advantage of the proposed method (LR+FS-adv). The average number of features selected are 321 (out of 2048) and 140 (out of 500) for AwA and CUB respectively. It is evident from the Table that the performance of MLP and ALadvC outperforms LR+FS with adv by a large margin. The performance can be attributed to MLP and ALadvS can apply projections to the feature vectors rather than selection/omission. We test the performance of the proposed adversarial approach as described in Section 3.3. We utilize the representation vector obtained from ResNet-101 as the base model. Next, we attach a trainable layers (AwA:500, CUB:500 100) with ReLU. Next, we add latent representation per group (AwA:10, CUB:5) with a linear connect to the group specific attribute prediction (with sigmoid activation). The smaller size of the latent layer ensures high feature transfer leading to more negative transfer. The baseline model (MLP) and the proposed model with auxiliary labels for each group as an adversarial classifier (ALadvC) are both trained with learning rate of lr = 0.01, which is decayed exponentially. Additionally, the gradient reversal weight λ = K 1+exp −10 * li, is related to the i th step of training, where l i = i/num steps. The scheduler increases the weight exponentially towards K (described in BID4). The model configuration (number and size of intermediate layers, K, lr) have all been picked from a large parameter sweep for best validation error. Due to the large number of adversarial branches (28 groups corresponding to 27 adversarial branches per group), we utilize the pairwise-task-label overlap to threshold the number of tasks (details below). TAB0 ALadvC performance improves by 2.8% and a 6.8% mAP improvements on the validation and test splits for AwA, respectively. Similarly, 4.8% and 3.5% improvements are observed in CUB dataset. Note that the splits ensure different correlations between attributes in the validation and test splits. Further, 4 (b) and 5 (b), show the effect of the gradient reversal on the learning process as the λ progressively improves. We notice a drop in the train mAP and simultaneous improvement over test mAP as the model sheds improvement obtained from negative transfer. Ultimately the performance on both datasets improves, suggesting the overall adverse effect of negative transfer. The group-wise performance measured on the test sets show drop in mAP for certain groups. We assert that the drop represents the true performance of the model on these tasks. As mentioned above, we identify an additional imbalance in the attribute prediction task, we term class-combination imbalance. The measure indicates the pairwise co-concurrence of attributes, which is computed as the pairwise Jaccard similarity between each attribute (illustration of the maximum Jaccard similarity cross group for both datasets in Figure 6, for their corresponding train splits). We utilize the measure to identify the subset of adversarial tasks per group for the CUB dataset. As shown in FIG4 (c), the performance of ALadvC improves when all group tasks are utilized. The empirical eludes to the advantage of the number or hardness of the adversarial task in improving prediction performance. In this work, we show that adversarial learning is the natural answer to prevent negative transfer. This leads to potential improvement in any supervised learning of natural data that is seeking generalization. We find that even in relatively straight-forward linear models presented above, co-occurrence of unrelated labels hampers performance and must be explicitly treated. We address the problem of negative transfer in a multi-task scenario, and also show the applicability of our solution in any supervised task. Supervised learning practitioners can utilize domain expertise to acquire and leverage additional negative labels for this purpose. Recent work in explainability of machine learning models can also be appropriately leveraged to facilitate this task.

Represent the TLDR for finding the corresponding scientific article

We look at negative transfer from a domain adaptation point of view to derive an adversarial learning algorithm.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the text to find a very short summary (1-2 sentences max)!

Question: Recent theoretical and experimental suggest the possibility of using current and near-future quantum hardware in challenging sampling tasks. In this paper, we introduce free-energy-based reinforcement learning (FERL) as an application of quantum hardware. We propose a method for processing a quantum annealer’s measured qubit spin configurations in approximating the free energy of a quantum Boltzmann machine (QBM). We then apply this method to perform reinforcement learning on the grid-world problem using the D-Wave 2000Q quantum annealer. The experimental show that our technique is a promising method for harnessing the power of quantum sampling in reinforcement learning tasks. Reinforcement learning (RL) BID33; BID6 has been successfully applied in fields such as engineering BID11; BID35, sociology BID12; BID30, and economics BID22; BID31. The training samples in reinforcement learning are provided by the interaction of an agent with an ambient environment. For example, in a motion planning problem in uncharted territory, it is desirable for the agent to learn to correctly navigate in the fastest way possible, making the fewest blind decisions. That is, neither exploration nor exploitation can be pursued exclusively without either facing a penalty or failing at the task. Our goal is, therefore, not only to design an algorithm that eventually converges to an optimal policy, but for the algorithm to be able to generate suboptimal policies early in the learning process. Free-energy-based reinforcement learning (FERL) using a restricted Boltzmann machine (RBM), as suggested by BID27, relies on approximating a utility function for the agent, called the Q-function, using the free energy of an RBM. RBMs have the advantage that their free energy can be efficiently calculated using closed formulae. RBMs can represent any joint distribution over binary variables BID20; BID15; Le BID19; however, this property of universality may require exponentially large RBMs BID20; Le BID19.General Boltzmann machines (GBM) are proposed in an effort to devise universal Q-function approximators with polynomially large Boltzmann networks BID10. Traditionally, Monte Carlo simulation is used to perform the computationally expensive tasks of approximating the free energy of GBMs under a Boltzmann distribution. One way to speed up the approximation process is to represent a GBM by an equivalent physical system and try to find its Boltzmann distribution. An example of such a physical system is a quantum annealer consisting of a network of pair-wise interacting quantum bits (qubits). Although quantum annealers have already been used in many areas of computational science, including combinatorial optimization and machine learning, their application in RL has not been explored. In order to use quantum annealing for RL, we first represent the Q-function as the free energy of a physical system, that is, that of a quantum annealer. We then slowly evolve the state of the physical system from a well-known initial state toward a state with a Boltzmann-like probability distribution. Repeating the annealing process sufficiently long can provide us with samples from the Boltzmann distribution so that we can empirically approximate the free energy of the physical system under this distribution. Finally, approximating the free energy of the system would give us an estimate of the Q-function. Up until the past few years, studies were limited to the classical Boltzmann machines. 1 Recently, BID10 generalized the classical method toward a quantum or quantum-inspired algorithm for approximating the free energy of GBMs. Using simulated quantum annealing (SQA) BID10 showed that FERL using a deep Boltzmann machine (DBM) can provide a drastic improvement in the early stages of learning, yet performing the same procedure on an actual quantum device remained a difficult task. This is because sampling from a quantum system representing a quantum Boltzmann machine is harder than the classical case, since at the end of each anneal the quantum system is in a superposition. Any attempt to measure the final state of the quantum system is doomed to fail since the superposition would collapse into a classical state that does not carry the entirety of information about the final state. In this work, we have two main contributions. We first employ a quantum annealer as a physical device to approximate the free energy of a classical Boltzmann machine. Second, we generalize the notion of classical Boltzmann machines to quantum Boltzmann machines within the field of RL and utilize a quantum annealer to approximate the free energy of a quantum system. In order to deal with the issue of superposition mentioned above, we propose a novel stacking procedure in that we attempt to reconstruct the full state of superposition from the partial information that we get from sampling after each anneal. Finally we report proof-of-concept using the D-Wave 2000Q quantum processor to provide experimental evidence for the applicability of a quantum annealer in reinforcement learning as predicted by BID10. We refer the reader to BID33 and BID37 for an exposition on Markov decision processes (MDP), controlled Markov chains, and the various broad aspects of reinforcement learning. A Q-function is defined by mapping a tuple pπ, s, aq of a given stationary policy π, a current state s, and an immediate action a of a controlled Markov chain to the expected value of the instantaneous and future discounted rewards of the Markov chain that begins with taking action a at initial state s and continuing according to π: Qpπ, s, aq " Err ps, aqs`E DISPLAYFORM0 Here, rps, aq is a random variable, perceived by the agent from the environment, representing the immediate reward of taking action a from state s, and Π is the Markov chain ing from restricting the controlled Markov chain to the policy π. The fixed real number γ P p0, 1q is the discount factor of the MDP. From Q˚ps, aq " max π Qpπ, s, aq, the optimal policy for the MDP can be retrieved via π˚psq " argmax a Q˚ps, aq. This reduces the MDP task to that of computing Q˚ps, aq. Through the Bellman optimality equation BID4, we get Q˚ps, aq " Err ps, aqs`γ ÿ DISPLAYFORM1 so Q˚is the fixed point of the following operator defined on L 8 pSˆAq:T pQq: ps, aq Þ Ñ Err ps, aqs`γ ż max a 1 In this paper, we focus on the TD Q-learning method, with the Q-function parametrized by neural networks in order to find π˚psq and Q˚ps, aq, which is based on minimizing the distance between T pQq and Q. A clamped Boltzmann machine is a GBM in which all visible nodes v are prescribed fixed assignments and removed from the underlying graph. Therefore, the energy of the clamped Boltzmann machine may be written as DISPLAYFORM0 where V and H are the sets of visible and hidden nodes, respectively, and by a slight abuse of notation, the letter v stands both for a graph node v P V and for the assignment v P t0, 1u. The interactions between the variables represented by their respective nodes are specified by real-valued weighted edges of the underlying undirected graph represented by w vh, and w hh 1 denotes the weights between visible and hidden, or hidden and hidden, nodes of the Boltzmann machine, respectively. A clamped quantum Boltzmann machine (QBM) has the same underlying graph as a clamped GBM, but instead of a binary random variable, qubits are associated to each node of the network. The energy function is substituted by the quantum Hamiltonian of an induced transverse field Ising model (TFIM), which is mathematically a Hermitian matrix DISPLAYFORM1 where σ z h represent the Pauli z-matrices and σ x h represent the Pauli x-matrices. Thus, a clamped QBM with Γ " 0 is equivalent to a clamped classical Boltzmann machine. This is because, in this case, H v is a diagonal matrix in the σ z -basis, the spectrum of which is identical to the range of the classical Hamiltonian. We note that is a particular instance of a TFIM. Let us begin with the classical Boltzmann machine case. Following BID27, for an assignment of visible variables v, F pvq denotes the equilibrium free energy, and is given via where β " 1 k B T is a fixed thermodynamic beta. In BID27, it was proposed to use the negative free energy of a GBM to approximate the Q-function through the relationship Qps, aq «´F ps, aq "´F ps, a; wq for each admissible state-action pair ps, aq P SˆA. Here, s and a are binary vectors encoding the state s and action a on the state nodes and action nodes, respectively, of a GBM. In RL, the visible nodes of a GBM are partitioned into two subsets of state nodes S and action nodes A. Here, w represents the vector of weights of a GBM as in. Each entry w of w can now be trained using the TD update rule: DISPLAYFORM0 ∆w vh " εpr n ps n, a n q`γQps n`1, a n`1 q´Qps n, a n qqvxhy and ∆w hh 1 " εpr n ps n, a n q`γQps n`1, a n`1 q´Qps n, a n qqxhh 1 y,where xhy and xhh 1 y are the expected values of the variables and the products of the variables, respectively, in the binary encoding of the hidden nodes with respect to the Boltzmann distribution of the classical Hamiltonian. 1 k B T be a fixed thermodynamic beta as in the classical case. As before, for an assignment of visible variables v, F pvq denotes the equilibrium free energy, and is given via DISPLAYFORM1 Here, Z v " trpe´β Hv q is the partition function of the clamped QBM and ρ v is the density matrix DISPLAYFORM2 Hv. The term´trpρ v ln ρ v q is the entropy of the system. Note that FORMULA7 is a generalization of. The notation x¨¨¨y is used for the expected value of any observable with respect to the Gibbs measure (i.e., the Boltzmann distribution), in particular, DISPLAYFORM3 This is also a generalization of the weighted sum ř h Pph|vqE v phq in. Inspired by the ideas of BID27 and BID1, we use the negative free energy of a QBM to approximate the Q-function exactly as in the classical case:Qps, aq «´F ps, a; wq for each admissible state-action pair ps, aq P SˆA. As before, s and a are binary vectors encoding the state s and action a on the state nodes and action nodes, respectively, of a Boltzmann machine. In RL, the visible nodes of a Boltzmann machine are partitioned into two subsets of state nodes S and action nodes A. Here, w represents the vector of weights of a QBM as in. Each entry w of w can now be trained using the TD update rule:∆w "´εpr n ps n, a n q´γF ps n`1, a n`1 q`F ps n, a n qq BF Bw.As shown in BID10, from we obtain ∆w vh " εpr n ps n, a n q γF ps n`1, a n`1 q`F ps n, a n qqvxσ z h y and ∆w hh 1 " εpr n ps n, a n q γF ps n`1, a n`1 q`F ps n, a n qqxσ z h σ z h 1 y. This concludes the development of the FERL method using QBMs. We refer the reader to Algorithm 3 in BID10 for more details. What remains to be done is to approximate values of the free energy F ps, aq and also the expected values of the observables xσ z h y and xσ z h σ z h 1 y. In this paper, we demonstrate how quantum annealing can be used to address this challenge. The evolution of a quantum system under a slowly changing, time-dependent Hamiltonian is characterized by BID7. The quantum adiabatic theorem (QAT) in BID7 states that a system remains in its instantaneous steady state, provided there is a gap between the eigen-energy of the steady state and the rest of the Hamiltonian's spectrum at every point in time. QAT motivated BID13 to introduce a paradigm of quantum computing known as quantum adiabatic computation which is closely related to the quantum analogue of simulated annealing, namely quantum annealing (QA), introduced by BID18.The history of QA and QAT inspired manufacturing efforts towards physical realizations of adiabatic evolution via quantum hardware BID17. In reality, the manufactured chips are operated at a non-zero temperature and are not isolated from their environment. Therefore, the existing adiabatic theory does not cover the behaviour of these machines. A contemporary investigation in quantum adiabatic theory was therefore initiated to study adiabaticity in open quantum systems BID28; BID36; Albash et al. FORMULA1; BID2; BID3. These sources prove adiabatic theorems for open quantum systems under various assumptions, in particular when the quantum system is coupled to a thermal bath satisfying the Kubo-Martin-Schwinger condition, implying that the instantaneous steady state is the instantaneous Gibbs state. This work in progress shows promising opportunities to use quantum annealers to sample from the Gibbs state of a TFIM. In practice, due to additional complications (e.g., level crossings and gap closure, described in the references above), the samples gathered from the quantum annealer are far from the Gibbs state of the final Hamiltonian. In fact, BID0 suggests that the distribution of the samples would instead correspond to an instantaneous Hamiltonian at an intermediate point in time, called the freeze-out point. Unfortunately, this point and, consequently, the strength Γ of the transverse field at this point, is not known a priori, and also depends on the TFIM undergoing evolution. Our goal is simply to associate a single (average) virual Γ to all TFIMs constructed through FERL. Another unknown parameter is the inverse temperature β, at which the Gibbs state, the partition function, and the free energy are attained. In a similar fashion, we wish to associate a single virtual β to all TFIMs encountered. The quantum annealer used in our experiments is the D-Wave 2000Q, which consists of a chip of superconducting qubits connected to each other according to a sparse adjacency graph called the Chimera graph. The Chimera graph structure looks significantly different from the frequently used models in machine learning, for example, RBMs and DBMs, which consist of consecutive fully connected bipartite graphs. FIG0 shows two adjacent blocks of the Chimera graph which consist of 16 qubits, which, in this paper, serve as the clamped QBM used in FERL.Another complication when using a quantum annealer as a QBM is that the spin configurations of the qubits can only be measured along a fixed axis (here the z-basis of the Bloch sphere). Once σ z is measured, all of the quantum information related to the projection of the spin along the transverse field (i.e., the spin σ x) collapses and cannot be retrieved. Therefore, even with a choice of virtual Γ, virtual β, and all of the measured configurations, the energy of the TFIM is still unknown. We propose a method for overcoming this challenge based on the Suzuki-Trotter expansion of the TFIM, which we call replica stacking, the details of which are explained in §3.4. In §4, we perform a grid search over values of the virtual parameters β and Γ. The accepted virtual parameters are the ones that in the most-effective learning for FERL in the early stages of training. By the Suzuki-Trotter decomposition BID34, the partition function of the TFIM defined by the Hamiltonian can be approximated using the partition function of a classical Hamiltonian denoted by H eff v and called an effective Hamiltonian, which corresponds to a classical Ising model of one dimension higher. More precisely, where r is the number of replicas, w`" 1 2β log coth´Γ β r¯, and h k represent spins of the classical system of one dimension higher. Note that each hidden node's Pauli z-matrices σ z h are represented by r classical spins, denoted by h k, with a slight abuse of notation. In other words, the original Ising model with a non-zero transverse field represented through non-commuting operators can be mapped to a classical Ising model of one dimension higher. FIG1 shows the underlying graph of a TFIM on a two-dimensional lattice and a corresponding 10-replica effective Hamiltonian in three dimensions. DISPLAYFORM0 The intuition behind the Suzuki-Trotter decomposition is that the superposition of the spins in a quantum system is represented classically by replicas in the z-basis. In other words, the measurement of the quantum system in the z-basis is interpreted as choosing one replica at random. Note that the probabilities of measuring`1 or´1 for each individual spin are preserved. This way, each hidden node in the quantum Boltzmann machine carries more information than a classical one; in fact, a classical representation of this system requires r classical binary units via the Suzuki-Trotter decomposition. Consequently, the connections between the hidden nodes become more complicated in the quantum case as well and can carry more information on the correlations between the hidden nodes. Note that the coupling strengths between the replicas are not arbitrary, but come from the mathematical decomposition following the Suzuki-Trotter formula. As a , the quantum Boltzmann machine can be viewed as an undirected graphical model but in one dimension higher than the classical Boltzmann machine. In the case of classical GBMs without further restrictions on the graph structure, xhy, xhh 1 y, and Qps, aq «´F ps, a; wq are not tractable. Consequently, to perform the weight update in one requires samples from the Boltzmann distribution corresponding to energy function to estimate xhy, xhh 1 y, and F ps, a; wq empirically. To approximate the right-hand side of and FORMULA1, we sample from the Boltzmann distribution of the energy function represented by the effective Hamiltonian using (, Theorem 6). We find the expected values of the observables xσ One way to sample spin values from the Boltzmann distribution of the effective Hamiltonian is to use the simulated quantum annealing algorithm (SQA) (see for an introduction). SQA is one of the many flavours of quantum Monte Carlo methods, and is based on the Suzuki-Trotter expansion described above. This algorithm simulates the quantum annealing phenomena of a TFIM by slowly reducing the strength of the transverse field at finite temperature to the desired target value. In our implementation, we have used a single spin-flip variant of SQA with a linear transverse-field schedule as in BID21 and BID14. Experimental studies have shown similarities in the behaviour of SQA and that of quantum annealing Isakov et al. The classical counterpart of SQA is conventional simulated annealing (SA), which is based on thermal annealing. This algorithm can be used to sample from Boltzmann distributions that correspond to an Ising spin model in the absence of a transverse field. Unlike SA, it is possible to use SQA not only to approximate the Boltzmann distribution of a classical Boltzmann machine, but also that of a quantum Hamiltonian in the presence of a transverse field. This can be done by reducing the strength of the transverse field to the desired value defined by the model, rather than to zero. It has been proven by BID24 that the spin system defined by SQA converges to the Boltzmann distribution of the effective classical Hamiltonian of one dimension higher that corresponds to the quantum Hamiltonian. Therefore, it is straightforward to use SQA to approximate the free energy in as well as the observables xσ z h y and xσ z h σ z h 1 y. However, any Boltzmann distribution sampling method based on Markov chain Monte Carlo (MCMC) has the major drawback of being extremely slow and computationally involved. Actually, it is an NP-hard problem to sample from the Boltzmann distribution. Another option is to use variational approximation BID26, which suffers from lack of accuracy and works in practice only in limited cases. As explained above, quantum annealers have the potential to provide samples from Boltzmann distributions (in the z-basis) corresponding to TFIM in a more efficient way. In what follows, we explain how to use quantum annealing to approximate the free energy corresponding to an effective Hamiltonian which in turn can be used to approximate the free energy of a QBM. (, Theorem 6) and translation invariance, each replica of the effective classical model is an approximation of the spin measurements of the TFIM in the measurement bases σ z. Therefore, a σ z -configuration sampled by a quantum annealer that operates at a given virtual inverse temperature β, and anneals up to a virtual transverse-field strength Γ, may be viewed as an instance of a classical spin configuration from a replica of the classical effective Hamiltonian of one dimension higher. This suggests the following method to approximate the free energy from for a TFIM. We gather a pool C of configurations sampled by the quantum annealer for the TFIM considered, allowing repetitions. Let r be the number of replicas. We write c eff " pc 1,..., c r q to indicate an effective configuration c eff with the classical configurations c 1 to c r as its replicas. We write c eff to denote the underlying set tc 1,..., c r u of replicas of c eff (without considering their ordering). We have P rc eff " pc 1,..., c r qs " P " c eff " pc 1,..., c r q|c eff " tc 1,..., c r u ‰ˆP " c eff " tc 1,..., c r u ‰ " P " c eff " pc 1,..., c r q|c eff " tc 1,..., c r u ‰ˆP " c eff " tc 1,..., c r u|c eff Ď C ‰ˆP " c eff Ď C ‰.The argument in the previous paragraph can now be employed to allow the assumption P " c eff Ď C ‰ » 1. In other words, the probability mass function of the effective configurations is supported in the subset of those configurations synthesized from the elements of C as candidate replicas. The conditional probability Prc eff " tc 1,..., c r u|c eff Ď C s can be sampled from by drawing r elements c 1,..., c r from C. We then sample from P " c eff " pc 1,..., c r q|c eff " tc 1,..., c r u ‰, according to the following distribution over c eff: DISPLAYFORM0 We consider πpc eff q our target distribution and construct the following MCMC method for which the limiting distribution is πpc eff q. We first attach the r classical spin configurations to the SQA's effective configuration structure uniformly at random. We then transition to a different arrangement with a Metropolis acceptance probability. For example, we may choose two classical configurations at random and exchange them with probability DISPLAYFORM1 where Epc eff q " w`ř h´ř r´1 k"1 h c k h c k`1`h c1 h cr¯. Such a stochastic process is known to satisfy the detailed balance condition. Consequently, the MCMC method allows us to sample from the effective spin configurations. This procedure of sampling and then performing the MCMC method creates a pool of effective spin configurations, which are then employed in equation FORMULA1 in order to approximate the free energy of the TFIM empirically. However, we consider a relatively small number of hidden nodes in our experiments, so the number of different σ z -configurations sampled by the quantum annealer is limited. As a consequence, there is no practical need to perform the MCMC method defined above. Instead, we attach classical spin configurations from the pool to the SQA effective configuration structure at random. In other words, in r iterations, a spin configuration is sampled from the pool of classical spin configurations described above and inserted as the next replica of the effective classical Hamiltonian consisting of r replicas. It is worthwhile to reiterate that this replica stacking technique yields an undirected graphical model. Specifically, the structure described in FIG1 is an undirected graphical model in the space of hidden nodes, where the node statistics are obtained from the Boltzmann distribution. One difference between this model and a classical Boltzmann machine is that each hidden node activation is governed by a series of r replicas in one dimension higher, and the undirected, replica-to-replica connections calculated therein. Moreover, the energy function of this extended model differs from the energy function of the classical Boltzmann machine (compare and FORMULA3 ). The free energy of the extended graphical model serves as the function approximator to the Q-function. We benchmark our various FERL methods on a 3ˆ5 grid-world problem BID32 with an agent capable of taking the actions up, down, left, or right, or standing still, on a grid-world with y, andPpc ef f |s i, a i q using Algorithm 2, for (i " 1, 2) calculate F ps i, a i q using for (i " 1, 2) Qps i, a i q Ð´F ps i, a i q for pi " 1, 2q update QBM weights using and πps 1 q Ð argmax a Qps 1, aq end for return π Algorithm 2 Replica stacking initialize the structure of the effective Hamiltonian in one dimension higher for i " 1, 2,..., m do for j " 1, 2,..., r do obtain spin configuration sample in z-basis from QA attach this spin configuration to j-th replica of the i-th effective configuration structure end for perform the MCMC technique described in §3.4 with transition probabilities to obtain the i-th instance of effective spin configurations end for obtain xH eff s i,a i y from the average energy of the m effective spin configurations obtain xhy and xhh 1 y by averaging over all h and h 1 replicas in each spin configuration gather statistics from Ppc eff |s i, a i q using the m effective spin configurations return xhy, xhh 1 y, xH DISPLAYFORM0 y, and Ppc eff |s i, a i q one deterministic reward, one wall, and one penalty, as shown in FIG5. The task is to find an optimal policy, as shown in FIG5, for the agent at each state in the grid-world. All of the Boltzmann machines used in our algorithms consist of 16 hidden nodes. The discount factor, as explained in §2, is set to 0.8. The agent attains the reward R " 200 in the top-left corner, the neutral value of moving to any empty cell is 100, and the agent is penalized by not receiving any reward if it moves to the penalty cell with value P " 0.For T r independent runs of every FERL method, T s training samples are used. The fidelity measure at the i-th training sample is defined by fidelitypiq " pT rˆ| S|q´1 DISPLAYFORM1 where π˚denotes the best known policy and Aps, i, lq denotes the action assigned at the l-th run and i-th training sample to the state s. In our experiments, each algorithm is run 100 times. An optimal policy for this problem instance can be represented as a selection of directional arrows indicating movement directions. Fig. 4 demonstrates the performance of a fully connected deep Q-network BID23 consisting of an input layer of 14 state nodes, two layers of eight hidden nodes each, and an output layer of five nodes representing the values of the Q-function for different actions, given a configuration of state nodes. We use the same number of hidden nodes in the fully connected deep Q-network as in the other networks described in this paper. We treat the network of superconducting qubits represented in FIG0 as a clamped QBM with two hidden layers, represented using blue and red colours. The state nodes are considered fully connected to the blue qubits and the action nodes are fully connected to the red qubits. For a choice of virtual parameters Γ ‰ 0 and β, which appear in FORMULA1 and FORMULA1, and for each query to the D-Wave 2000Q chip, we construct 150 effective classical configurations of one dimension higher, out of a pool of 3750 reads, according to the replica stacking method introduced in §3.4. The 150 configurations are, in turn, employed to approximate the free energy of the quantum Hamiltonian. We conduct 10 independent runs of FERL in this fashion, and find the average fidelity over the 10 runs and over the T s " 300 training samples. Fig. 6 shows the growth of the average fidelity of the best known policies generated by different FERL methods. For each method, the fidelity curve is an average over 100 independent runs, each with T s " 500 training samples. In this figure, the "D-Wave Γ " 0.5, β " 2.0" curve corresponds to the D-Wave 2000Q replica stacking-based method with the choice of the best virtual parameters Γ " 0.5 and β " 2.0, as shown in the heatmap in FIG6. The training is based on formulae,, and. The "SQA Bipartite Γ " 0.5, β " 2.0" and "SQA Chimera Γ " 0.5, β " 2.0" curves are based on the same formulae with the underlying graphs being a bipartite (DBM) and a Chimera graph, respectively, with the same choice of virtual parameters, but the effective Hamiltonian configurations generated using SQA as explained in §3.3.The "SA Bipartite β " 2.0" and "SA Chimera β " 2.0" curves are generated by using SA to train a classical DBM and a classical GBM on the Chimera graph, respectively, using formulae, FORMULA6, and. SA is run with a linear inverse temperature schedule, where β " 2.0 indicates the final value. The "D-Wave Classical β " 2.0" curve is generated using the same method, but with samples obtained using the D-Wave 2000Q. The "RBM" curve is generated using the method in BID27. We solve the grid-world problem using various Q-learning methods with the Q-function parametrized by different neural networks. For comparison, we demonstrate the performance of a fully connected deep Q-network method that can be considered state of the art. This method efficiently processes every training sample, but, as shown in Fig. 4, requires a very large number of training samples to converge to the optimal policy. Another conventional method is free-energy-based RL using an RBM. This method is also very successful at learning the optimal policy at the scale of the RL task considered in our experiment. Although this method does not outperform other FERL methods that take advantage of a highly efficient sampling oracle, the processing of each training sample is efficient, as it is based on closed formulae. In fact, for the size of problem considered, the RBM-based FERL outperforms the fully connected deep Q-network method. The comparison of in Fig. 6 suggests that replica stacking is a successful method for estimating effective classical configurations obtained from a quantum annealer, given that the spins can only be measured in measurement bases. For practical use in RL, this method provides a means of treating the quantum annealer as a QBM. FERL using the quantum annealer, in conjunction with the replica stacking technique, provides significant improvement over FERL using classical Boltzmann machines. The curve representing SQA-based FERL using a Boltzmann machine on the Chimera graph is almost coincident with the one obtained using the D-Wave 2000Q, whereas the SQA-based FERL using a DBM slightly outperforms it. This suggests that quantum annealing chips with greater connectivity and more control over annealing time can further improve the performance of the replica stacking method applied to RL tasks. This is further supported by comparing the performance of SA-based FERL using a DBM versus SA-based FERL using the Chimera graph. This shows that DBM is, due to its additional connections, a better choice of neural network compared to the Chimera graph. For practical reasons, we aim to associate an identical choice of virtual parameters β and Γ to all of the TFIMs constructed using FERL. BID5 and BID25 provide methods for estimating the effective inverse temperature β for other applications. However, in both studies, the samples obtained from the quantum annealer are matched to the Boltzmann distribution of a classical Ising model. In fact, the transverse-field strength is a second virtual parameter that we consider. The optimal choice Γ " 0.5 corresponds to 2{3 of the annealing time, in agreement with the work of BID0, who also considers TFIM with 16 qubits. The agreement of FERL using quantum annealer reads treated as classical Boltzmann samples with that of FERL using SA and classical Boltzmann machines suggests that, at least for this task and this size of Boltzmann machine, the measurements provided by the D-Wave 2000Q can be considered good approximations of Boltzmann distribution samples of classical Ising models. The extended undirected graphical model developed in this paper using the replica stacking method is not limited to Q-function approximation in RL tasks. Potentially, this method can be applied to tasks where Boltzmann machines can be used. This method provides a mechanism for approximating the activations and partition functions of quantum Boltzmann machines that have a significant transverse field. In this paper, we describe a free-energy-based reinforcement learning algorithm using an existing quantum annealer, namely the D-Wave 2000Q. Our method relies on the Suzuki-Trotter decomposition and the use of the measured configurations by the D-Wave 2000Q as replicas of an effective classical Ising model of one dimension higher. The presented here are first-step proofs of concept of a proposed quantum algorithm with a promising path towards outperforming reinforcement learning algorithms devised for digital hardware. Given appropriate advances in quantum annealing hardware, future research can employ the proposed principles to solve larger-scale reinforcement learning tasks in the emerging field of quantum machine learning.

Represent the short summary for finding the corresponding scientific article (long)

We train Quantum Boltzmann Machines using a replica stacking method and a quantum annealer to perform a reinforcement learning task.

We show that it is possible to recover the parameters of a 1-layer ReLU generative model from looking at samples generated by it


Question: represent the article for finding a single-sentence tldr:

Question: Deep learning models are vulnerable to adversarial examples crafted by applying human-imperceptible perturbations on benign inputs. However, under the black-box setting, most existing adversaries often have a poor transferability to attack other defense models. In this work, from the perspective of regarding the adversarial example generation as an optimization process, we propose two new methods to improve the transferability of adversarial examples, namely Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and Scale-Invariant attack Method (SIM). NI-FGSM aims to adapt Nesterov accelerated gradient into the iterative attacks so as to effectively look ahead and improve the transferability of adversarial examples. While SIM is based on our discovery on the scale-invariant property of deep learning models, for which we leverage to optimize the adversarial perturbations over the scale copies of the input images so as to avoid "overfitting” on the white-box model being attacked and generate more transferable adversarial examples. NI-FGSM and SIM can be naturally integrated to build a robust gradient-based attack to generate more transferable adversarial examples against the defense models. Empirical on ImageNet dataset demonstrate that our attack methods exhibit higher transferability and achieve higher attack success rates than state-of-the-art gradient-based attacks. Deep learning models have been shown to be vulnerable to adversarial examples ), which are generated by applying human-imperceptible perturbations on benign input to in the misclassification. In addition, adversarial examples have an intriguing property of transferability, where adversarial examples crafted by the current model can also fool other unknown models. As adversarial examples can help identify the robustness of models , as well as improve the robustness of models by adversarial training, learning how to generate adversarial examples with high transferability is important and has gained increasing attentions in the literature. Several gradient-based attacks have been proposed to generate adversarial examples, such as onestep attacks and iterative attacks (; . Under the white-box setting, with the knowledge of the current model, existing attacks can achieve high success rates. However, they often exhibit low success rates under the black-box setting, especially for models with defense mechanism, such as adversarial training (; and input modification). Under the black-box setting, most existing attacks fail to generate robust adversarial examples against defense models. In this work, by regarding the adversarial example generation process as an optimization process, we propose two new methods to improve the transferability of adversarial examples: Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and Scale-Invariant attack Method (SIM). • Inspired by the fact that Nesterov accelerated gradient is superior to momentum for conventionally optimization , we adapt Nesterov accelerated gradient into the iterative gradient-based attack, so as to effectively look ahead and improve the transferability of adversarial examples. We expect that NI-FGSM could replace the momentum iterative gradient-based method in the gradient accumulating portion and yield higher performance. • Besides, we discover that deep learning models have the scale-invariant property, and propose a Scale-Invariant attack Method (SIM) to improve the transferability of adversarial examples by optimizing the adversarial perturbations over the scale copies of the input images. SIM can avoid "overfitting" on the white-box model being attacked and generate more transferable adversarial examples against other black-box models. • We found that combining our NI-FGSM and SIM with existing gradient-based attack methods (e.g., diverse input method ) can further boost the attack success rates of adversarial examples. Extensive experiments on the ImageNet dataset show that our methods attack both normally trained models and adversarially trained models with higher attack success rates than existing baseline attacks. Our best attack method, SI-NI-TI-DIM (Scale-Invariant Nesterov Iterative FGSM integrated with translation-invariant diverse input method), reaches an average success rate of 93.5% against adversarially trained models under the black-box setting. For further demonstration, we evaluate our methods by attacking the latest robust defense methods;;; ). The show that our attack methods can generate adversarial examples with higher transferability than state-of-theart gradient-based attacks. 2.1 NOTATION Let x and y true be a benign image and the corresponding true label, respectively. Let J(x, y true) be the loss function of the classifier (e.g. the cross-entropy loss). Let x adv be the adversarial example of the benign image x. The goal of the non-targeted adversaries is to search an adversarial example x adv to maximize the loss J(x adv, y true) in the p norm bounded perturbations. To align with previous works, we focus on p = ∞ in this work to measure the distortion between x adv and x. That is x adv − x ∞ ≤, where is the magnitude of adversarial perturbations. Several attack methods have been proposed to generate adversarial examples. Here we provide a brief introduction. generates an adversarial example x adv by maximizing the loss function J(x adv, y true) with one-step update as: where sign(·) function restricts the perturbation in the L ∞ norm bound. Iterative Fast Gradient Sign Method (I-FGSM). extend FGSM to an iterative version by applying FGSM with a small step size α: where Clip x (·) function restricts generated adversarial examples to be within the -ball of x. Projected Gradient Descent (PGD). PGD attack is a strong iterative variant of FGSM. It consists of a random start within the allowed norm ball and then follows by running several iterations of I-FGSM to generate adversarial examples. Momentum Iterative Fast Gradient Sign Method (MI-FGSM). integrate momentum into the iterative attack and lead to a higher transferability for adversarial examples. Their update procedure is formalized as follows: where g t is the accumulated gradient at iteration t, and µ is the decay factor of g t. Diverse Input Method (DIM). optimize the adversarial perturbations over the diverse transformation of the input image at each iteration. The transformations include the random resizing and the random padding. DIM can be naturally integrated into other gradient-based attacks to further improve the transferability of adversarial examples. Translation-Invariant Method (TIM). Instead of optimizing the adversarial perturbations on a single image, use a set of translated images to optimize the adversarial perturbations. They further develop an efficient algorithm to calculate the gradients by convolving the gradient at untranslated images with a kernel matrix. TIM can also be naturally integrated with other gradientbased attack methods. The combination of TIM and DIM, namely TI-DIM, is the current strongest black-box attack method. Carlini & Wagner attack (C&W). C&W attack is an optimization-based method which directly optimizes the distance between the benign examples and the adversarial examples by solving: arg min It is a powerful method to find adversarial examples while minimizing perturbations for white-box attacks, but it lacks the transferability for black-box attacks. Various defense methods have been proposed to against adversarial examples, which can fall into the following two categories. Adversarial Training. One popular and promising defense method is adversarial training;; ), which augments the training data by the adversarial examples in the training process. develop a successful adversarial training method, which leverages the projected gradient descent (PGD) attack to generate adversarial examples. However, this method is difficult to scale to large-scale datasets . propose ensemble adversarial training by augmenting the training data with perturbations transferred from various models, so as to further improve the robustness against the black-box attacks. Currently, adversarial training is still one of the best techniques to defend against adversarial attacks. Input Modification. The second category of defense methods aims to mitigate the effects of adversarial perturbations by modifying the input data. discover that there exists a range of image transformations, which have the potential to remove adversarial perturbations while preserving the visual information of the images. mitigate the adversarial effects through random transformations. propose high-level representation guided denoiser to purify the adversarial examples. propose a JPEG-based defensive compression framework to rectify adversarial examples without impacting classification accuracy on benign data. leverage an end-to-end image compression model to defend adversarial examples. Although these defense methods perform well in practice, they can not tell whether the model is truly robust to adversarial perturbations. use randomized smoothing to obtain an ImageNet classifier with certified adversarial robustness. Similar with the process of training neural networks, the process of generating adversarial examples can also be viewed as an optimization problem. In the optimizing phase, the white-box model being attacked to optimize the adversarial examples can be viewed as the training data on the training process. And the adversarial examples can be viewed as the training parameters of the model. Then in the testing phase, the black-box models to evaluate the adversarial examples can be viewed as the testing data of the model. From the perspective of the optimization, the transferability of the adversarial examples is similar with the generalization ability of the trained models. Thus, we can migrate the methods used to improve the generalization of models to the generation of adversarial examples, so as to improving the transferability of adversarial examples. Many methods have been proposed to improve the generalization ability of the deep learning models, which can be split to two aspects: better optimization algorithm, such as Adam optimizer; data augmentation . Correspondingly, the methods to improve the transferability of adversarial examples can also be split to two aspects: better optimization algorithm, such as MI-FGSM, which applies the idea of momentum; model augmentation (i.e., ensemble attack on multiple models), such as the work of, which considers to attack multiple models simultaneously. Based on above analysis, we aim to improve the transferability of adversarial examples by applying the idea of Nesterov accelerated gradient for optimization and using a set of scaled images to achieve model augmentation. Nesterov Accelerated Gradient (NAG) is a slight variation of normal gradient descent, which can speed up the training process and improve the convergence significantly. NAG can be viewed as an improved momentum method, which can be expressed as: Typical gradient-based iterative attacks (e.g., I-FGSM) greedily perturb the images in the direction of the sign of the gradient at each iteration, which usually falls into poor local maxima, and shows weak transferability than single-step attacks (e.g., FGSM). show that adopting momentum into attacks can stabilize the update directions, which helps to escape from poor local maxima and improve the transferability. Compared to momentum, beyond stabilize the update directions, the anticipatory update of NAG gives previous accumulated gradient a correction that helps to effectively look ahead. Such looking ahead property of NAG can help us escape from poor local maxima easier and faster, ing in the improvement on transferability. We integrate NAG into the iterative gradient-based attack to leverage the looking ahead property of NAG and build a robust adversarial attack, which we refer to as NI-FGSM (Nesterov Iterative Fast Gradient Sign Method). Specifically, we make a jump in the direction of previous accumulated gradients before computing the gradients in each iteration. Start with g 0 = 0, the update procedure of NI-FGSM can be formalized as follows: where g t denotes the accumulated gradients at the iteration t, and µ denotes the decay factor of g t. Besides considering a better optimization algorithm for the adversaries, we can also improve the transferability of adversarial examples by model augmentation. We first introduce a formal definition of loss-preserving transformation and model augmentation as follows. Definition 1 Loss-preserving Transformation. Given an input x with its ground-truth label y true and a classifier f (x): x ∈ X → y ∈ Y with the cross-entropy loss J(x, y), if there exists an input transformation T (·) that satisfies J(T (x), y true ) ≈ J(x, y true) for any x ∈ X, we say T (·) is a loss-preserving transformation. Definition 2 Model Augmentation. Given an input x with its ground-truth label y true and a model f (x): x ∈ X → y ∈ Y with the cross-entropy loss J(x, y), if there exists a loss-preserving transformation T (·), then we derive a new model by f (x) = f (T (x)) from the original model f. we define such derivation of models as model augmentation. Intuitively, similar to the generalization of models that can be improved by feeding more training data, the transferability of adversarial examples can be improved by attacking more models simultaneously. enhance the gradient-based attack by attacking an ensemble of models. However, their approach requires training a set of different models to attack, which has a large computational cost. Instead, in this work, we derive an ensemble of models from the original model by model augmentation, which is a simple way of obtaining multiple models via the loss-preserving transformation. To get the loss-preserving transformation, we discover that deep neural networks might have the scale-invariant property, besides the translation invariance. Specifically, the loss values are similar for the original and the scaled images on the same model, which is empirically validated in Section 4.2. Thus, the scale transformation can be served as a model augmentation method. Driven by the above analysis, we propose a Scale-Invariant attack Method (SIM), which optimizes the adversarial perturbations over the scale copies of the input image: arg max where S i (x) = x/2 i denotes the scale copy of the input image x with the scale factor 1/2 i, and m denotes the number of the scale copies. With SIM, instead of training a set of models to attack, we can effectively achieve ensemble attacks on multiple models by model augmentation. More importantly, it can help avoid "overfitting" on the white-box model being attacked and generate more transferable adversarial examples. For the gradient processing of crafting adversarial examples, NI-FGSM introduces a better optimization algorithm to stabilize and correct the update directions at each iteration. For the ensemble attack of crafting adversarial examples, SIM introduces model augmentation to derive multiple models to attack from a single model. Thus, NI-FGSM and SIM can be naturally combined to build a stronger attack, which we refer to as SI-NI-FGSM (Scale-Invariant Nesterov Iterative Fast Gradient Sign Method). The algorithm of SI-NI-FGSM attack is summarized in Algorithm 1. In addition, SI-NI-FGSM can be integrated with DIM (Diverse Input Method), TIM (TranslationInvariant Method) and TI-DIM (Translation-Invariant with Diverse Input Method) as SI-NI-DIM, SI-NI-TIM and SI-NI-TI-DIM, respectively, to further boost the transferability of adversarial examples. The detailed algorithms for these attack methods are provided in Appendix A. In this section, we provide experimental evidence on the advantage of the proposed methods. We first provide experimental setup, followed by the exploration of the scale-invariance property for deep learning models. We then compare the of the proposed methods with baseline methods in Section 4.3 and 4.4 on both normally trained models and adversarially trained models. Beyond the defense models based on adversarial training, we also quantify the effectiveness of the proposed methods on other advanced defense in Section 4.5. Additional discussions, the comparison between NI-FGSM and MI-FGSM and the comparison with classic attacks, are in Section 4.6. Codes are available at https://github.com/JHL-HUST/SI-NI-FGSM. Input: A clean example x with ground-truth label y true; a classifier f with loss function J; Input: Perturbation size; maximum iterations T; number of scale copies m and decay factor µ. Update g t+1 by g t+1 = µ · g t + Dataset. We randomly choose 1000 images belonging to the 1000 categories from ILSVRC 2012 validation set, which are almost correctly classified by all the testing models. For normally trained models, we consider Inception-v3 (Inc-v3) , Inception-v4 (Inc-v4), Inception-Resnet-v2 (IncRes-v2) and Resnet-v2-101 (Res-101) . For adversarially trained models, we consider Inc-v3 ens3, Inc-v3 ens4 and IncRes-v2 ens . Additionally, we include other advanced defense models: high-level representation guided denoiser (HGD), random resizing and padding (R&P) , NIPS-r3 1, feature distillation (FD) , purifying perturbations via image compression model (Comdefend) and randomized smoothing (RS) . Baselines. We integrate our methods with DIM , TIM, and TI-DIM , to show the performance improvement of SI-NI-FGSM over these baselines. Denote our SI-NI-FGSM integrated with other attacks as SI-NI-DIM, SI-NI-TIM, and SI-NI-TIM-DIM, respectively. Hyper-parameters. For the hyper-parameters, we follow the settings in with the maximum perturbation as = 16, number of iteration T = 16, and step size α = 1.6. For MI-FGSM, we adopt the default decay factor µ = 1.0. For DIM, the transformation probability is set to 0.5. For TIM, we adopt the Gaussian kernel and the size of the kernel is set to 7 × 7. For our SI-NI-FGSM, the number of scale copies is set to m = 5. To validate the scale-invariant property of deep neural networks, we randomly choose 1,000 original images from ImageNet dataset and keep the scale size in the range of [0.1, 2.0] with a step size 0.1. Then we feed the scaled images into the testing models, including Inc-v3, Inc-v4, IncRes-2, and Res-101, to get the average loss over 1,000 images. As shown in Figure 1, we can easily observe that the loss curves are smooth and stable when the scale size is in range [0.1, 1.3]. That is, the loss values are very similar for the original and scaled images. So we assume that the scale-invariant property of deep models is held within [0.1, 1.3], and we leverage the scale-invariant property to optimize the adversarial perturbations over the scale copies of the input images. In this subsection, we integrate our SI-NI-FGSM with TIM, DIM and TI-DIM, respectively, and compare the black-box attack success rates of our extensions with the baselines under single model setting. As shown in Table 1, our extension methods consistently outperform the baseline attacks by 10% ∼ 35% under the black-box setting, and achieve nearly 100% success rates under the white-box setting. It indicates that SI-NI-FGSM can serve as a powerful approach to improve the transferability of adversarial examples. Following the work of , we consider to show the performance of our methods by attacking multiple models simultaneously. Specifically, we attack an ensemble of normally trained models (including Inc-v3, Inc-v4, IncRes-v2 and Res-101) with equal ensemble weights using TIM, SI-NI-TIM, DIM, SI-NI-DIM, TI-DIM and SI-NI-TI-DIM, respectively. As shown in Table 2, our methods improve the attack success rates across all experiments over the baselines. In general, our methods consistently outperform the baseline attacks by 10% ∼ 30% under the black-box setting. Especially, SI-NI-TI-DIM, the extension by combining SI-NI-FGSM with TI-DIM, can fool the adversarially trained models with a high average success rate of 93.5%. It indicates that these advanced adversarially trained models provide little robustness guarantee under the black-box attack of SI-NI-TI-DIM. Besides normally trained models and adversarially trained models, we consider to quantify the effectiveness of our methods on other advanced defenses, including the top-3 defense solutions in the NIPS competition (high-level representation guided denoiser (HGD, rank-1), random resizing and padding (R&P, rank-2) and the rank-3 submission (NIPS-r3), and three recently proposed defense methods (feature distillation (FD) , purifying perturbations via image compression model (Comdefend) and randomized smoothing (RS) ). We compare our SI-NI-TI-DIM with MI-FGSM, which is the top-1 attack solution in the NIPS 2017 competition, and TI-DIM , which is state-of-the-art attack. We first generate adversarial examples on the ensemble models, including Inc-v3, Inc-v4, IncResv2, and Res-101 by using MI-FGSM, TI-DIM, and SI-NI-TI-DIM, respectively. Then, we evaluate the adversarial examples by attacking these defenses. As shown in Table 3, our method SI-NI-TI-DIM achieves an average attack success rate of 90.3%, surpassing state-of-the-art attacks by a large margin of 14.7%. By solely depending on the trans- ferability of adversarial examples and attacking on the normally trained models, SI-NI-TI-DIM can fool the adversarially trained models and other advanced defense mechanism, raising a new security issue for the development of more robust deep learning models. Some adversarial examples generated by SI-NI-TI-DIM are shown in Appendix B.. The adversarial examples are crafted on Inc-v3 with various number of iterations ranging from 4 to 16, and then transfer to attack Inc-v4 and IncRes-v2. As shown in Figure 2, NI-FGSM yields higher attack success rates than MI-FGSM with the same number of iterations. In another view, NI-FGSM needs fewer number of iterations to gain the same attack success rate of MI-FGSM. The not only indicate that NI-FGSM has a better transferability, but also demonstrate that with the property of looking ahead, NI-FGSM can accelerate the generation of adversarial examples. Comparison with classic attacks. We consider to make addition comparison with classic attacks, including FGSM , I-FGSM , PGD and C&W . As shown in Table 4, our methods achieve 100% attack success rate which is the same as C&W under the white-box setting, and significantly outperform other methods under the black-box setting. In this work, we propose two new attack methods, namely Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and Scale-Invariant attack Method (SIM), to improve the transferability of adversarial examples. NI-FGSM aims to adopt Nesterov accelerated gradient method into the gradientbased attack, and SIM aims to achieve model augmentation by leveraging the scale-invariant property of models. NI-FGSM and SIM can be naturally combined to build a robust attack, namely SI-NI-FGSM. Moreover, by integrating SI-NI-FGSM with the baseline attacks, we can further improve the transferability of adversarial examples. Extensive experiments demonstrate that our methods not only yield higher success rates on adversarially trained models but also break other strong defense mechanism. Our work of NI-FGSM suggests that other momentum methods (e.g. Adam) may also be helpful to build a strong attack, which will be our future work, and the key is how to migrate the optimization method to the gradient-based iterative attack. Our work also shows that deep neural networks have the scale-invariant property, which we utilized to design the SIM to improve the attack transferability. However, it is not clear why the scale-invariant property holds. Possibly it is due to the batch normalization at each convolutional layer, that may mitigate the impact of the scale change. We will also explore the reason more thoroughly in our future work. The algorithm of SI-NI-TI-DIM attack is summarized in Algorithm 2. We can get the SI-NI-DIM attack algorithm by removing Step 10 of Algorithm 2, and get the SI-NI-TIM attack algorithm by removing T (·; p) in Step 7 of Algorithm 2. Input: A clean example x with ground-truth label y true; a classifier f with loss function J; Input: Perturbation size; maximum iterations T; number of scale copies m and decay factor µ. for i = 0 to m − 1 do sum the gradients over the scale copies of the input image 7: Get the gradients by ∇ x J(T (S i (x nes t); p), y true ) apply random resizing and padding to the inputs with the probability p Convolve the gradients by g = W * g convolve gradient with the pre-defined kernel W Update g t+1 by g t+1 = µ · g t + We visualize 12 randomly selected benign images and their corresponding adversarial images in Figure 3. The adversarial images are crafted on the ensemble models, including Inc-v3, Inc-v4, IncRes-v2 and Res-101, using the proposed SI-NI-TI-DIM. We see that these generated adversarial perturbations are human imperceptible.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We proposed a Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and a Scale-Invariant attack Method (SIM) that can boost the transferability of adversarial examples for image classification.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.