Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Planning in high-dimensional space remains a challenging problem, even with recent advances in algorithms and computational power. We are inspired by efference copy and sensory reafference theory from neuroscience. Our aim is to allow agents to form mental models of their environments for planning. The cerebellum is emulated with a two-stream, fully connected, predictor network. The network receives as inputs the efference as well as the features of the current state. Building on insights gained from knowledge distillation methods, we choose as our features the outputs of a pre-trained network, yielding a compressed representation of the current state. The representation is chosen such that it allows for fast search using classical graph search algorithms. We display the effectiveness of our approach on a viewpoint-matching task using a modified best-first search algorithm. As we manipulate an object in our hands, we can accurately predict how it looks after some action is performed. Through our visual sensory system, we receive high-dimensional information about the object. However, we do not hallucinate its full-dimensional representation as we estimate how it would look and feel after we act. But we feel that we understood what happened if there is an agreement between the experience of the event and our predicted experience. There has been much recent work on methods that take advantage of compact representations of states for search and exploration. One of the advantages of this approach is that finding a good representation allows for faster and more efficient planning. This holds in particular when the latent space is of a much lower dimensionality than the one where the states originally live in. Our central nervous system (CNS) sends a command (efferent) to our motor system, as well as sending a copy of the efferent to our cerebellum, which is our key organ for predicting the sensory outcome of actions when we initiate a movement and is responsible for fine motor control. The cerebellum then compares the of the action (sensory reafference) with the intended consequences. If they differ, then the cerebellum makes changes to its internal structure such that it does a better job next time -i.e., in no uncertain terms, it learns. The cerebellum receives 40 times more information than it outputs, by a count of the number of axons. This gives us a sense of the scale of the compression ratio between the high dimensional input and low dimensional output. Thus, we constrain our attention to planning in a low-dimensional space, without necessarily reconstructing the high-dimensional one. We apply this insight for reducing the complexity of tasks such that planning in high dimensionality space can be done by classical AI methods in low dimensionality space. Our contributions are thus twofold: provide a link between efference theory and classical planning with a simple model and introduce a search method for applying the model to reduced state-space search. We validate our approach experimentally on visual data associated with categorical actions that connect the images, for example taking an object and rotating it. We create a simple manipulation task using the NORB dataset , where the agent is presented with a starting viewpoint of an object and the task is to produce a sequence of actions such that the agent ends up with the target viewpoint of the object. As the NORB data set can be embedded on a cylinder (Schüler et al., 2018) or a sphere , we can visualize the actions as traversing the embedded manifold. We essentially aim at approximating a Markov decision process' state transition function. Thus, similarities abound in the reinforcement learning literature: many agents perform explicit latentspace planning computations as part of learning and executing policies. train a reinforcement learning (RL) agent to simultaneously predict rewards as well as future latent states. Our work is distinct from these as we are not assuming a reward signal and are not constrained to an RL setting during training. Similar to our training setup, predict future frames in ATARI environments conditioned on actions. The predicted frames are used for learning the dynamics of the environment, e.g. for improving exploration by informing agents of which actions are more likely to in unseen states. Our work differs as we are maneuvering within the latent space and not the full input space. Vision, memory, and controller modules are combined for learning a model of the world before learning a decision model in Ha and Schmidhuber's World Models . A predictive model is trained in an unsupervised manner, allowing the agent to learn policies entirely within its learned latent space representation of the environment. Instead of training the representation from scratch, we apply the machinery of transfer learning by using pre-trained networks. This allows the method to be applied to a generic representation rather than a specifically trained representation. Using the output of pre-trained networks as targets instead of the original pixels is not new. The case where the output of a larger model is the target for a smaller model is known as knowledge distillation (Buciluǎ et al., 2006). This is used for compressing a model ensemble into a single model. Vondrick et al. learn to make high-level semantic predictions of future frames in video data. Given a frame at a current time, a neural network is tasked with predicting the representation of a future frame, e.g. by AlexNet. Our approach extends this general idea by admitting an action as the input to our predictor network as well. Causal InfoGAN is a method based on generative adversarial networks (GANs) , inspired by InfoGAN in particular , for learning plannable representations. A GAN is trained for encoding start and target states and plans a trajectory in the representation space as well as reconstructing intermediate states in the plans. Our method differ from this by training a simple forward model and forgoing reconstruction, which is unnecessary for planning. To motivate our approach, we will briefly describe the concept of efference copies from neuroscience. As we act, our central nervous system (CNS) sends a signal, or an efference, to our peripheral nervous system (PNS). An additional copy of the efference is created, which is sent to an internal forward model . This model makes a prediction of the sensory outcome and learns by minimizing the errors between its prediction and the actual sensory inputs that are experienced after the action is performed. The sensory inputs that reach the CNS from the sensory receptors in the PNS are called afference. They can be exafference, signals that are caused by the environment, or reafference, signals that are caused by ourself acting. By creating an efference copy and training the forward model, we can tell apart exafference from reafference. This is how we can be tickled when grass straws are brushed against our feet (exafference), but we can walk barefeet over a field of grass without feeling tickled (reafference). We assume that the motor system and sensory system are fixed and not necessarily trainable. The motor system could be the transition function of a Markov decision process. The sensory system is assumed to be a visual feature extractor -in our experiments, a Laplacian Eigenmap or an intermediate layer of a pre-trained convolutional neural network (CNN) is used. Pre-trained CNNs can provide powerful, generic descriptors while eliminating the computational load associated with predicting the pixel values of the full images . ). A motor command, or efference, is issued by the CNS. A copy of the efference is made and is sent to an internal forward model, which predicts the sensory of the efference. The original efference is sent to the motor system to act. The consequence of the action in the world is observed by the sensory system, and a reafference is created and sent to the CNS. The forward model is then updated to minimize the discrepancy between predicted and actual reafference. Suppose we have a feature map φ and a training set of N data points (, where S t is the state at time step t and a t is an action ing in a state S t+1 . We train the predictor f, parameterized by θ, by minimizing the mean squared error loss over f 's parameters: is our set of training data. In our experiments, we construct f as a two-stream, fully connected, neural network (Fig. 2). Using this predictor we are able to carry out efficient planning in the latent space defined by φ. By planning we mean that there is a start state S start and a goal state S goal and we are tasked with finding a path between them. Assuming deterministic dynamics, we output the expected representation after performing the action. This allows us to formulate planning as a classical AI pathfinding or graph traversal problem. The environment is turned to a graph by considering each state as a node and each action as an edge. We used a modified best-first search algorithm with the trained EfferenceNets for our experiments (Algorithm 1). Each state is associated with a node as well as a score: the distance between its representation and the representation of the goal state. The edges between nodes are the available actions at each state. The output of the search is the sequence of actions that corresponds to the path connecting the start node to the proposed solution node, i.e. the node whose representation is the one closest to the goal. To make the algorithm faster, we only consider paths that do not take us to a state that has already been evaluated, even if there might be a difference in the predictions from going this roundabout way. That is, if a permutation of the actions in the next path to be considered is already in an evaluated path, it will be skipped. This is akin to transposition tables used to speed up search in game trees. This might yield paths with redundancies which can be amended with path-simplifying routines (e.g. take one step forward instead of one step left, one forward then one right). Selecting the optimal action in each step of a temporally-extended Markov decision process in a task effectively is a hard problem that is solved by different fields in different ways. For example, in Algorithm 1 Find a plan to reach S = S j, where j = argmin i≤k ||Ri − φ(S goal)|| and R = (r 0, . . ., r k) are the representations of explored states. Input: S start, S goal, max. trials m, action set A, feature map φ and EfferenceNet f Output: A plan of actions (a 0, . . ., a n) reaching # add j to the set of checked indices for k ← 0 to m do j = argmin i≤k,i / ∈D ||ri − φ(S goal)|| # take a new state, most similar to the goal for all a ∈ A do r k+1 ← f (rj, a) # try every action from current state R ← R ∪ {r k+1} # save the estimated representation p k+1 ← pj ∪ a P ← P ∪ {p k+1} # store path from goal state to current state # add j to the set of checked indices end for return pj the area of reinforcement learning, it is addressed by estimating the accumulated reward signal for a given behavioral strategy and adapting the strategy to optimize this estimate. This requires either a large number of actively generated samples or exact knowledge of the environment (in dynamic programming) to find a strategy that behaves optimally. In this paper, we choose a different approach and utilize a one-step prediction model to allow decisions to be made based on the predicted outcome of a number of one-step lookaheads started from an initial state. The actions are chosen so that each step greedily maximizes progress towards a known target. This method, sometimes called hillclimber or best-first search, belongs to the family of informed search algorithms . To be more efficient than random or exhaustive search, these kinds of algorithms rely on heuristics to provide sufficient evidence for a good -albeit not necessarily optimal -decision at every time step to reach the goal. Here we use the Euclidean distance in representation space: An action is preferred if its predicted is closest to the goal. The usefulness of this heuristics depends on how well and how coherently the Euclidean distance encodes the actual distance to the goal state in terms of the number of actions. Our experiments show that an easily attainable general purpose representation, such as a pre-trained VGG16 , can already provide sufficient guidance to apply such this heuristic effectively. One might, however, ask what a particularly suited representation might look like when attainability is ignored. It would need to take the topological structure of the underlying data manifold into account, such that the Euclidean distance becomes a good proxy for the geodesic distance. One class of methods that fulfill this are spectral embeddings, such as Laplacian Eigenmaps (LEMs) . Since they do not readily allow for out-of-sample embedding, they will only be applied in an in-sample fashion to serve as a control experiment in Section 5.2.1. In the experiments, we show that our method can be combined with simple, informed graph search algorithms to plan trajectories for manipulation tasks (Fig. 3, top row). We use the NORB dataset, which contains images of different objects each under 9 different elevations, 36 azimuths, and 6 lighting conditions. We derive from the dataset an object manipulation environment. The actions correspond to turning a turntable back and forth by 20 •, moving the camera up or down by 5 • or changing the illumination intensity. After the EfferenceNet is trained, we apply Algorithm 1 to the viewpoint matching task. The goal is to find a sequence of actions that transforms the start state to the goal state. The two states differ in their azimuth and elevation configuration. Given a feature map φ, we task the EfferenceNet f with predicting φ(S t+1) after the action a was performed in the state S t. We train f by minimizing the mean-squared error between f (φ(S t), a) and φ(S t+1). The network (Fig. 2 is built with Keras and optimized with Nadam and converges in two hours on a Tesla P40. It is a two-stream dense neural network. Each stream consists of one dense layer followed by a batch normalization (BatchNorm) layer. The outputs of these streams are then concatenated and passed through 3 dense layers, each one followed by a BatchNorm, and then an output dense layer. Every dense layer, except the last, is followed by a rectified linear unit (ReLU) activation. Figure 2: EfferenceNet architecture. The network takes as input the representation vector φ(S t) of the state S t as determined by the feature map φ, as well as the one-hot encoded action a. It outputs the estimated feature vector of the ing state S t+1 after action a is performed. Our φ is chosen to be a Laplacian Eigenmap for in-sample search and the second-to-last layer output of VGG16 , pre-trained on ImageNet , for out-ofsample search. As the representation made by φ do not change over the course of the training, they can be cached, expediting the training. Of the 10 car toys in the NORB dataset, we randomly chose 9 for our training set and test on the remaining one. Embedding a single toy's confgurations in three dimensions using Laplacian Eigenmaps will in a cylindrical embedding that encodes both, elevation and azimuth angles, as visible in Figure 4. Three dimensions are needed so that the cyclic azimuth can be embedded correctly as sin(θ) and cos(θ). If such a representation is now used to train the EfferenceNet which is subsequently applied in Algorithm 1, one would expect monotonically decreasing distance the closer the prediction comes to the target. Figure 5 shows that this is the case and that this behavior can be very effectively used for a greedy heuristic. While the monotinicity is not always exact due to imperfections in the approximate prediction, Figure 5 still qualitatively illustrates a best-case scenario. The goal and start states are chosen randomly, with the constraint that their distances along each dimension (azimuth and elevation) are uniformly distributed. As the states are searched, a heat map of similarities is produced (Fig. 3). To visualize the performance of the search we plot a histogram (Fig. 6) illustrating the accuracy of the search. The looks less accurate with respect to elevation than azimuth, but this is due to the elevation changes being more fine-grained than the azimuth changes, namely by a factor of 4. The difference between the elevation of the goal and solution viewpoints in Figure 3 left, for example, is hardly perceptible. If one would scale the histograms by angle and not by bins, the drop-off would be comparable. The heat maps of the type shown in Figure 3 can be aggregated to reveal basins of attraction during the search. Each heat map is shifted such that the goal position is at the bottom, middle row (Fig. 7, a). Here it is apparent that the goal and the 180 • flipped (azimuth) version of the goal are attractor states. This is due to the feature map being sensitive to the rough shape of the object, but being unable to distinguish finer details. In (Fig. 7, b) we display an aggregate heat map when the agent can alter the lighting conditions as well. In our work, we focus on learning a transition model. Doing control after learning the model is an established approach, with the mature field of model-based RL dedicated to it. This has the advantage of allowing for quick learning of new reward functions, since disentangling reward contingencies from the transition function is helpful when learning for multiple/changing reward functions and allows useful learning when there is no reward available at all. Thus, it might also be useful in a sparse or late reward setting. Another advantage of our approach is that it accommodates evaluations of reward trajectories with arbitrary discounts. Standard RL methods are usually restricted in their optimization problems. Often, there is a choice between optimizing discounted or undiscounted expected returns. Simulation/rollout-based planning methods are not restricted in that sense: If you are able to predict reward trajectories, you can (greedily) optimize arbitrary functions of these -possibly allowing for behavior regularization. For example, the risk-averse portfolio manager can prioritize smooth reward trajectories over volatile ones. We use a pre-trained network because we believe that a flexible algorithm should be based rather on generic, multi-purpose representions and not on very specific representations. This contributes to the flexibility of the system. However, a drawback of using pre-trained networks is that features might be encoded that are irrelevant for the current task. This has the effect that informed search methods, such as best-first search, are not guaranteed to output the accurate solution in the latent space, as there might be distracting pockets of erroneous local minima. Our visualizations reveal gradient towards the goal state as well as a visually similar, far away states. There is variation in the similarities, preventing the planning algorithm from finding the exact goal for every task, sometimes yielding solutions that are the polar-opposites of the goal, w.r.t. the azimuth. Pairing the EfferenceNet with a good but generic feature map allows us to perform an accurate search in the latent space of manipulating unseen objects. This remarkably simple method, inspired by the neurology of the cerebellum, reveals a promising line of future work. We validate our method by on a viewpoint-matching task derived from the NORB dataset. In the case of deterministic environments, EfferenceNets calculate features of the current state and action, which in turn define the next state. This opens up a future direction of research by combining EfferenceNets with successor features . Furthermore, the study of effective feature maps strikes us as an important factor in this line of work to consider. We utilize here Laplacian Eigenmaps and pre-trained deep networks. It is probably possible to improve the performance of the system by end-to-end training but we believe that it is more promising to work on generic multi-purpose representations. Possible further methods include Slow Feature Analysis (SFA) (Schüler et al., 2018). SFA has been previously shown to solve a special case of LEMs while it allows for natural out-of-sample embeddings.

We present a neuroscience-inspired method based on neural networks for latent space search

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.


Question:
represent the text to find a very short summary (1-2 sentences max)
Fewshots:
'Deep Neural Networks (DNNs) are vulnerable to adversarial attacks, especially white-box targeted attacks. This paper studies the problem of how aggressive white-box targeted attacks can be to go beyond widely used Top-1 attacks. We propose to learn ordered Top-k attacks (k>=1), which enforce the Top-k predicted labels of an adversarial example to be the k (randomly) selected and ordered labels (the ground-truth label is exclusive). Two methods are presented. First, we extend the vanilla Carlini-Wagner (C&W) method and use it as a strong baseline. Second, we present an adversarial distillation framework consisting of two components: (i) Computing an adversarial probability distribution for any given ordered Top-$k$ targeted labels. (ii) Learning adversarial examples by minimizing the Kullback-Leibler (KL) divergence between the adversarial distribution and the predicted distribution, together with the perturbation energy penalty. In computing adversarial distributions, we explore how to leverage label semantic similarities, leading to knowledge-oriented attacks. In experiments, we test Top-k (k=1,2,5,10) attacks in the ImageNet-1000 val dataset using two popular DNNs trained with the clean ImageNet-1000 train dataset, ResNet-50 and DenseNet-121. Overall, the adversarial distillation approach obtains the best , especially by large margin when computation budget is limited.. It reduces the perturbation energy consistently with the same attack success rate on all the four k's, and improve the attack success rate by large margin against the modified C&W method for k=10. Despite the recent dramatic progress, deep neural networks (DNNs) (; ; ;) trained for visual recognition tasks (e.g., image classification) can be easily fooled by so-called adversarial attacks which utilize visually imperceptible, carefully-crafted perturbations to cause networks to misclassify inputs in arbitrarily chosen ways in the close set of labels used in training (; ; ;), even with one-pixel attacks . The existence of adversarial attacks hinders the deployment of DNNs-based visual recognition systems in a wide range of applications such as autonomous driving and smart medical diagnosis in the long-run. In this paper, we are interested in learning visually-imperceptible targeted attacks under the whitebox setting in image classification tasks. In the literature, most methods address targeted attacks in the Top-1 manner, in which an adversarial attack is said to be successful if a randomly selected label (not the ground-truth label) is predicted as the Top-1 label with the added perturbation satisfying to be visually-imperceptible. One question arises, • The "robustness" of an attack method itself: How far is the attack method able to push the underlying ground-truth label in the prediction of the learned adversarial examples? Table 1 shows the evaluation of the "robustness" of different attack methods. The widely used C&W method does not push the GT labels very far, especially when smaller perturbation energy is aimed using larger search range (e.g., the average rank of the GT label is 2.6 for C&W 9×1000). Consider Top-5, if the ground-truth labels of adversarial examples still largely appear in the Top-5 of the prediction, we may be over-confident about the 100% ASR, . Please see Sec. 4 for detail of experimental settings. Method ASR Proportion of GT Labels in Top-k (smaller is better) Average Rank of GT Labels (larger is better) Top-3 Top-5 Top-10 Top-50 Top-100 C&W9×30 99.9 36.9 50.5 66.3 90.0 95.1 20.4 C&W9×1000 100 71.9 87.0 96.1 99.9 100 2.6 FGSM 80.7 25.5 37.8 52.8 81.2 89.2 44.2 PGD10 100 3.3 6.7 12 34.7 43.9 306.5 MIFGSM10 99.9 0.7 1.9 6.0 22.5 32.3 404.4 especially when some downstream modules may rely on Top-5 predictions in their decision making. But, the three untargeted attack approaches are much better in terms of pushing the GT labels since they are usually move against the GT label explicitly in the optimization, but their perturbation energies are usually much larger. As we shall show, more "robust" attack methods can be developed by harnessing the advantages of the two types of attack methods. In addition, the targeted Top-1 attack setting could limit the flexibility of attacks, and may lead to less rich perturbations. To facilitate explicit control of targeted attacks and enable more "robust" attack methods, one natural solution, which is the focus of this paper, is to develop ordered Top-k targeted attacks which enforce the Top-k predicted labels of an adversarial example to be the k (randomly) selected and ordered labels (k ≥ 1, the GT label is exclusive). In this paper, we present two methods of learning ordered Top-k attacks. The basic idea is to design proper adversarial objective functions that in imperceptible perturbations for any test image through iterative gradient-based back-propagation. First, we extend the vanilla Carlini-Wagner (C&W) method and use it as a strong baseline. Second, we present an adversarial distillation (AD) framework consisting of two components: (i) Computing an adversarial probability distribution for any given ordered Top-k targeted labels. (ii) Learning adversarial examples by minimizing the Kullback-Leibler (KL) divergence between the adversarial distribution and the predicted distribution, together with the perturbation energy penalty. The proposed AD framework can be viewed as applying the network distillation frameworks (; ;) for "the bad" induced by target adversarial distributions. To compute a proper adversarial distribution for any given ordered Top-k targeted labels, the AD framework is motivated by two aspects: (i) The difference between the objective functions used by the C&W method and the three untargeted attack methods (Table 1) respectively. The former maximizes the margin of the logits between the target and the runner-up (either GT or ResNet-50. AD is better than the modified C&W method (CW *). The thickness represents the 2 energy (thinner is better). Please see Sec. 4 for detail of experimental settings. not), while the latter maximizes the cross-entropy between the prediction probabilities (softmax of logits) and the one-hot distribution of the ground-truth. (ii) The label smoothing methods ), which are often used to improve the performance of DNNs by addressing the over-confidence issue in the one-hot vector encoding of labels. More specifically, we explore how to leverage label semantic similarities in computing "smoothed" adversarial distributions, leading to knowledge-oriented attacks. We measure label semantic similarities using the cosine distance between some off-the-shelf word2vec embedding of labels such as the pretrained Glove embedding . Along this direction, another question of interest is further investigated: Are all Top-k targets equally challenging for an attack approach? In experiments, we test Top-k (k = 1, 2, 5, 10) in the ImageNet-1000 val dataset using two popular DNNs trained with clean ImageNet-1000 train dataset, ResNet-50 and DenseNet-121 respectively. Overall, the adversarial distillation approach obtains the best . It reduces the perturbation energy consistently with the same attack success rate on all the four k's, and improve the attack success rate by large margin against the modified C&W method for k = 10 (see Fig. 1). We observe that Top-k targets that are distant from the GT label in terms of either label semantic distance or prediction scores of clean images are actually more difficulty to attack. In summary, not only can ordered Top-k attacks improve the "robustness" of attacks, but also they provide insights on how aggressive adversarial attacks can be (under affordable optimization budgets). Our Contributions. This paper makes three main contributions to the field of learning adversarial attacks: (i) The problem in study is novel. Learning ordered Top-k adversarial attacks is an important problem that reflects the robustness of attacks themselves, but has not been addressed in the literature. (ii) The proposed adversarial distillation framework is effective, especially when k is large (such as k = 5, 10). (iii) The proposed knowledge-oriented adversarial distillation is novel. It worth exploring the existing distillation framework for a novel problem (ordered Top-k adversarial attacks) with some novel modifications (knowledge-oriented target distributions as "teachers"). The growing ubiquity of DNNs in advanced machine learning and AI systems dramatically increases their capabilities, but also increases the potential for new vulnerabilities to attacks. This situation has become critical as many powerful approaches have been developed where imperceptible perturbations to DNN inputs could deceive a well-trained DNN, significantly altering its prediction. Assuming full access to DNNs pretrained with clean images, white-box targeted attacks are powerful ways of investigating the brittleness of DNNs and their sensitivity to non-robust yet well-generalizing features in the data. Distillation. The central idea of our proposed AD method is built on distillation. Network distillation is a powerful training scheme proposed to train a new, usually lightweight model (a.k.a., the student) to mimic another already trained model (a.k.a. the teacher). It takes a functional viewpoint of the knowledge learned by the teacher as the conditional distribution it produces over outputs given an input. It teaches the student to keep up or emulate by adding some regularization terms to the loss in order to encourage the two models to be similar directly based on the distilled knowledge, replacing the training labels. Label smoothing can be treated as a simple hand-crafted knowledge to help improve model performance. Distillation has been exploited to develop defense models to improve model robustness. Our proposed adversarial distillation method utilizes the distillation idea in an opposite direction, leveraging label semantic driven knowledge for learning ordered Top-k attacks and improving attack robustness. Adversarial Attack. For image classification tasks using DNNs, the discovery of the existence of visually-imperceptible adversarial attacks was a big shock in developing DNNs. White-box attacks provide a powerful way of evaluating model brittleness. In a plain and loose explanation, DNNs are universal function approximator and capable of even fitting random labels in large scale classification tasks as ImageNet-1000 . Thus, adversarial attacks are generally learnable provided proper objective functions are given, especially when DNNs are trained with fully differentible back-propagation. Many white-box attack methods focus on norm-ball constrained objective functions (; ; ;). The C&W method investigates 7 different loss functions. The best performing loss function found by the C&W method has been applied in many attack methods and achieved strong (; ; . By introducing momentum in the MIFGSM method and the p gradient projection in the PGD method , they usually achieve better performance in generating adversarial examples. In the meanwhile, some other attack methods such as the StrAttack also investigate different loss functions for better interpretability of attacks. Our proposed method leverages label semantic knowledge in the loss function design for the first time. In this section, we first briefly introduce the white-box attack setting and the widely used C&W method under the Top-1 protocol, to be self-contained. Then we define the ordered Top-k attack formulation. To learn ordered Top-k attacks, we present detail of a modified C&W method as a strong baseline and the proposed AD framework. We focus on classification tasks using DNNs. Denote by (x, y) a pair of a clean input x ∈ X and its ground-truth label y ∈ Y. For example, in the ImageNet-1000 classification task, x represents a RGB image defined in the lattice of 224×224 and we have X R 3×224×224. y is the category label and we have Y {1, · · ·, 1000}. Let f (·; Θ) be a DNN pretrained on clean training data where Θ collects all estimated parameters and is fixed in learning adversarial examples. For notation simplicity, we denote by f (·) a pretrained DNN. The prediction for an input x from f (·) is usually defined using softmax function by, where P ∈ R |Y| represents the estimated confidence/probability vector (P c ≥ 0 and c P c = 1) and z(x) is the logit vector. The predicted label is then inferred byŷ = arg max c∈[1,|Y|] P c. The traditional Top-1 protocol of learning targeted attacks. For an input (x, y), given a target label t = y, we seek to compute some visually-imperceptible perturbation δ(x, t, f) using the pretrained and fixed DNN f (·) under the white-box setting. White-box attacks assume the complete knowledge of the pretrained DNN f, including its parameter values, architecture, training method, etc. The perturbed example is defined by, which is called an adversarial example of x if t =ŷ = arg max c f (x) c and the perturbation δ(x, t, f) is sufficiently small according to some energy metric. The C&W Method . Learning δ(x, t, f) under the Top-1 protocol is posed as a constrained optimization problem , n, where E(·) is defined by a p norm (e.g., the 2 norm) and n the size of the input domain (e.g., the number of pixels). To overcome the difficulty (non-linear and non-convex constraints) of directly solving Eqn. 3, the C&W method expresses it in a different form by designing some loss functions L(x) = L(x + δ) such that the first constraint t = arg max c f (x) c is satisfied if and only if L(x) ≤ 0. The best loss function proposed by the C&W method is defined by the hinge loss, which induces penalties when the logit of the target label is not the maximum among all labels. Then, the learning problem is formulated by, subject to x + δ ∈ n, which can be solved via back-propagation with the constraint satisfied via introducing a tanh layer. For the trade-off parameter λ, a binary search will be performed during the learning (e.g., 9 × 1000). It is straightforward to extend Eqn. 3 for learning ordered Top-k attacks (k ≥ 1). Denote by (t 1, · · ·, t k) the ordered Top-k targets (t i = y). We have, minimize subject to t i = arg max Directly solving Eqn. 6 is a difficulty task and proper loss functions are entailed, similar in spirit to the approximation approaches used in the Top-1 protocol, to ensure the first constraint can be satisfied once the optimization is converged (note that the optimization may fail, i.e., attacks fail). 3.3 LEARNING ORDERED TOP-k ATTACKS 3.3.1 A MODIFIED C&W METHOD We can modify the loss function (Eqn. 4) of the C&W method accordingly to solve Eqn. 6. We have, which covers the vanilla C&W loss (Eqn. 4), i.e., when k = 1, CW (x). The C&W loss function does not care where the underlying GT label will be as long as it is not in the Top-k. On the one hand, it is powerful in terms of attack success rate. On the other hand, the GT label may be very close to the Top-k, leading to over-confident attacks (see Tabel. 1). In addition, it is generic for any given Top-k targets. As we will show, they are less effective if we select the Top-k targets from the sub-set of labels which are least like the ground-truth label in terms of label semantics. To overcome the shortcomings of the C&W loss function and In our adversarial distillation framework, we adopt the view of point proposed in the network distillation method that the full confidence/probability distribution summarizes the knowledge of a trained DNN. We hypothesize that we can leverage the network distillation framework to learn the ordered Top-k attacks by designing a proper adversarial probability distribution across the entire set of labels that satisfies the specification of the given ordered Top-k targets, and facilitates explicit control of placing the GT label, as well as top-down integration of label semantics. Consider a given set of Top-k targets, {t 1, · · ·, t k}, denoted by P AD the adversarial probability distribution in which P AD ti > P AD tj (∀i < j) and P AD ti The space of candidate distributions are huge. We present a simple knowledge-oriented approach to define the adversarial distribution. We first specify the logit distribution and then compute the probability distribution using softmax. Denote by Z the maximum logit (e.g., Z = 10 in our experiments). We define the adversarial logits for the ordered Top-k targets by, z where γ is an empirically chosen decreasing factor (e.g., γ = 0.3 in our experiments). For the remaining categories l / ∈ {t 1, · · ·, t k}, we define the adversarial logit by, where 0 ≤ α < z AD t k is the maximum logit that can be assigned to any j, s(a, b) is the semantic similarity between the label a and label b, and is a small position for numerical consideration (e.g., = 1e-5). We compute s(a, b) using the cosine distance between the Glove embedding vectors of category names and −1 ≤ s(a, b) ≤ 1. Here, when α = 0, we discard the semantic knowledge and treat all the remaining categories equally. Note that our design of P AD is similar in spirit to the label smoothing technique and its variants ) except that we target attack labels and exploit label semantic knowledge. The design choice is still preliminary, although we observe its effectiveness in experiments. We hope this can encourage more sophisticated work to be explored. With the adversarial probability distribution P AD defined above as the target, we use the KL divergence as the loss function in our adversarial distillation framework as done in network distillation and we have, L (k) and then we follow the same optimization scheme as done in the C&W method (Eqn. 5). In this section, we evaluate ordered Top-k attacks with k = 1, 2, 5, 10 in the ImageNet-1000 benchmark using two pretrained DNNs, ResNet-50 and DenseNet-121 from the PyTorch model zoo 1. We implement our method using the AdverTorch toolkit 2. Our source code will be released. Data. In ImageNet-1000 , there are 50, 000 images for validation. To study attacks, we utilize the subset of images for which the predictions of both the ResNet-50 and DenseNet-121 are correct. To reduce the computational demand, we randomly sample a smaller subset, as commonly done in the literature. We first randomly select 500 categories to enlarge the coverage of categories, and then randomly chose 2 images per selected categories, ing in 1000 test images in total. Settings. We follow the protocol used in the C&W method. We only test 2 norm as the energy penalty for perturbations in learning (Eqn. 5). But, we evaluate learned adversarial examples in terms of three norms (1, 2 and ∞). We test two search schema for the trade-off parameter λ in optimization: both use 9 steps of binary search, and 30 and 1000 iterations of optimization are performed for each trial of λ. In practice, computation budget is an important factor and less computationally expensive ones are usually preferred. Only α = 1 is used in Eqn. 9 in experiments for simplicity due to computational demand. We compare the under three scenarios proposed in the C&W method : The Best Case settings test the attack against all incorrect classes, and report the target class(es) that was least difficult to attack. The Worst Case settings test the attack against all incorrect classes, and report the target class(es) that was most difficult to attack. The Average Case settings select the target class(es) uniformly at random among the labels that are not the GT. We first test ordered Top-k attacks using ResNet-50 for the four selected k's. Table. 2 summarizes the quantitative and comparisons. For Top-10 attacks, the proposed AD method obtains significantly better in terms of both ASR and the 2 energy of the added perturbation. For example, the proposed AD method has relative 362.3% ASR improvement over the strong C&W baseline for the worst case setting. For Top-5 attacks, the AD method obtains significantly better when the search budget is relatively low (i.e., 9 × 30). For Top-k (k = 1, 2) attacks, both the C&W method and the AD method can achieve 100% ASR, but the AD method has consistently lower energies of the added perturbation, i.e., finding more effective attacks and richer perturbations. Fig. 2 shows some learned adversarial examples of ordered Top-10 and Top-5 attacks. Table 2: Results and comparisons under the ordered Top-k targeted attack protocol using randomly selected and ordered 10 targets (GT exclusive) in ImageNet using ResNet-50. For Top-1 attacks, we also compare with three state-of-the-art untargeted attack methods, FGSM , PGD and MIFGSM . 10 iterations are used for both PGD and MIFGSM. Intuitively, we understand that they should not be equally difficult. We conduct some experiments to test this hypothesis. In particular, we test whether the label semantic knowledge can help identify the weak spots of different attack methods, and whether the proposed AD method can gain more in those weak spots. We test Top-5 using ResNet-50 3. Table. 3 summarizes the . We observe that for the 9 × 30 search budget, attacks are more challenging if the Top-5 targets are selected from the least-like set in terms of the label semantic similarity (see Eqn. 9), or from the lowest-score set in terms of prediction scores on clean images. To investigate if the observations from ResNets hold for other DNNs, we also test DenseNet-121 in ImageNet-1000. We test two settings: k = 1, 5 4. Overall, we obtain similar . . The proposed AD method has smaller perturbation energies and "cleaner" (lower-entropy) prediction distributions. Note that for Top-10 attacks, the 9 × 30 search scheme does not work (see Table. 2). Table 3: Results of ordered Top-5 targeted attacks with targets being selected based on (Top) label similarity, which uses 5 most-like labels and 5 least-like labels as targets respectively, and (Bottom) prediction score of clean image, which uses 5 highest-score labels and 5-lowest score labels. In both cases, GT labels are exclusive. Top-1 targeted attack protocol using randomly selected and ordered 5 targets (GT exclusive). For Top-1 attacks, we also compare with three state-of-the-art untargeted attack methods, FGSM , PGD and MIFGSM . 10 iterations are used for both PGD and MIFGSM. This paper proposes to extend the traditional Top-1 targeted attack setting to the ordered Top-k setting (k ≥ 1) under the white-box attack protocol. The ordered Top-k targeted attacks can improve the robustness of attacks themselves. To our knowledge, it is the first work studying this ordered Top-k attacks. To learn the ordered Top-k attacks, we present a conceptually simple yet effective adversarial distillation framework motivated by network distillation. We also develop a modified C&W method as the strong baseline for the ordered Top-k targeted attacks. In experiments, the proposed method is tested in ImageNet-1000 using two popular DNNs, ResNet-50 and DenseNet-121, with consistently better obtained. We investigate the effectiveness of label semantic knowledge in designing the adversarial distribution for distilling the ordered Top-k targeted attacks. Discussions. We have shown that the proposed AD method is generally applicable to learn ordered Top-k attacks. But, we note that the two components of the AD framework are in their simplest forms in this paper, and need to be more thoroughly studied: designing more informative adversarial distributions to guide the optimization to learn adversarial examples better and faster, and investigating loss functions other than KL divergence such as the Jensen-Shannon (JS) divergence or the Earth-Mover distance. On the other hand, we observed that the proposed AD method is more effective when computation budget is limited (e.g., using the 9 × 30 search scheme). This leads to the theoretically and computationally interesting question whether different attack methods all will work comparably well if the computation budget is not limited. Of course, in practice, we prefer more powerful ones when only limited computation budget is allowed. Furthermore, we observed that both the modified C&W method and the AD method largely do not work in learning Top-k (k ≥ 20) attacks with the two search schema (9 × 30 and 9 × 1000). We are working on addressing the aforementioned issues to test the Top-k (k ≥ 20) cases, thus providing a thorough empirical answer to the question: how aggressive can adversarial attacks be?' == 'ordered Top-k adversarial attacks' != 'We revisit the simple idea of pruning connections of DNNs through $\ell_1$ regularization achieving state-of-the-art results on multiple datasets with theoretic guarantees.'

Question: Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. Introducing the concept of an optimal representation space, we provide a simple theoretical resolution to this apparent paradox. In addition, we present a straightforward procedure that, without any retraining or architectural modifications, allows deep recurrent models to perform equally well (and sometimes better) when compared to shallow models. To validate our analysis, we conduct a set of consistent empirical evaluations and introduce several new sentence embedding models in the process. Even though this work is presented within the context of natural language processing, the insights are readily applicable to other domains that rely on distributed representations for transfer tasks. Distributed representations have played a pivotal role in the current success of machine learning. In contrast with the symbolic representations of classical AI, distributed representation spaces can encode rich notions of semantic similarity in their distance measures, allowing systems to generalise to novel inputs. Methods to learn these representations have gained significant traction, in particular for modelling words BID30. They have since been successfully applied to many other domains, including images BID15 BID39 and graphs BID25 BID17 BID33.Using unlabelled data to learn effective representations is at the forefront of modern machine learning research. The Natural Language Processing (NLP) community in particular, has invested significant efforts in the construction BID30 BID37 BID10 BID21, evaluation and theoretical analysis BID28 of distributed representations for words. Recently, attention has shifted towards the unsupervised learning of representations for larger pieces of text, such as phrases BID50 BID51, sentences BID22 BID43 BID19 BID7, and entire paragraphs BID27. Some of this work simply sums or averages constituent word vectors to obtain a sentence representation BID32 BID31 BID48 BID7, which is surprisingly effective but naturally cannot leverage any contextual information. Another line of research has relied on a sentence-level distributional hypothesis BID38, originally applied to words BID18, which is an assumption that sentences which occur in similar contexts have a similar meaning. Such models often use an encoder-decoder architecture BID12 to predict the adjacent sentences of any given sentence. Examples of such models include SkipThought, which uses Recurrent Neural Networks (RNNs) for its encoder and decoders, and FastSent BID19, which replaces the RNNs with simpler bagof-words (BOW) versions. Models trained in an unsupervised manner on large text corpora are usually applied to supervised transfer tasks, where the representation for a sentence forms the input to a supervised classification problem, or to unsupervised similarity tasks, where the similarity (typically taken to be the cosine similarity) of two inputs is compared with corresponding human judgements of semantic similarity in order to inform some downstream process, such as information retrieval. Interestingly, some researchers have observed that deep complex models like SkipThought tend to do well on supervised transfer tasks but relatively poorly on unsupervised similarity tasks, whereas for shallow log-linear models like FastSent the opposite is true BID19 BID13. It has been highlighted that this should be addressed by analysing the geometry of the representation space BID6 BID42 BID19, however, to the best of our knowledge it has not been systematically attempted 1.In this work we attempt to address the observed performance gap on unsupervised similarity tasks between representations produced by simple models and those produced by deep complex models. Our main contributions are as follows:• We introduce the concept of an optimal representation space, in which the space has a similarity measure that is optimal with respect to the objective function.• We show that models with log-linear decoders are usually evaluated in their optimal space, while recurrent models are not. This effectively explains the performance gap on unsupervised similarity tasks.• We show that, when evaluated in their optimal space, recurrent models close that gap. We also provide a procedure for extracting this optimal space using the decoder hidden states.• We validate our findings with a series of consistent empirical evaluations utilising a single publicly available codebase. We begin by considering a general problem of learning a conditional probability distribution P model (y | x) over the output symbols y ∈ Y given the input symbols x ∈ X. Definition 1. A space H combined with a similarity measure ρ: H × H → R in which semantically close symbols s i, s j ∈ S have representations h i, h j ∈ H that are close in ρ is called a distributed representation space BID16.In general, a distributed representation of a symbol s is obtained via some function h s = f (s; θ f), parametrised by weights θ f. Distributed representations of the input symbols are typically found as the layer activations of a Deep Neural Network (DNN). One can imagine running all possible x ∈ X through a DNN and using the activations h x of the n th layer as vectors in H x: DISPLAYFORM0 The distributed representation space of the output symbols H y can be obtained via some function h y = g(y; θ g) that does not depend on the input symbol x, e.g. a row of the softmax projection matrix that corresponds to the output y. In practice, although H obtained in such a manner with a reasonable vector similarity ρ (such as cosine or Euclidean distance) forms a distributed representation space, there is no a priori reason why an arbitrary choice of a similarity function would be appropriate given H and the model's objective. There is no analytic guarantee, for arbitrarily chosen H and ρ, that small changes in semantic similarity of symbols correspond to small changes in similarity ρ between their vector representations in H and vice versa. This motivates Definition 2. A space H equipped with a similarity measure ρ such that log P model (y | x) ∝ ρ (h y, h x) is called an optimal representation space. In words, if a model has an optimal representation space, the conditional log-probability of an output symbol y given an input symbol x is proportional to the similarity ρ(h y, h x) between their corresponding vector representations h y, h x ∈ H.For example, consider the following standard classification model DISPLAYFORM0 where u y is the y th row of the output projection matrix U.If H x = {DNN(x) | x ∈ X } and H y = {u y | y ∈ Y}, then H = H x ∪ H y equipped with ρ(h 1, h 2) = h 1 · h 2 (the dot product) is an optimal representation space. Note that if the exponents of Equation FORMULA1 contained Euclidean distance, then we would find log P model (y | x) ∝ ||u y − DNN(x)|| 2. The optimal representation space would then be equipped with Euclidean distance as its optimal distance measure ρ. This easily extends to any other distance measures desired to be induced on the optimal representation space. Let us elaborate on why Definition 2 is a reasonable definition of an optimal space. Let x 1, x 2 ∈ X be the input symbols and y 1, y 2 ∈ Y their corresponding outputs. Using DISPLAYFORM1 to denote that a and b are close under ρ, a reasonable model trained on a subset of (X, Y) will ensure that h x1 ρ ∼ h y1 and h x2 ρ ∼ h y2. If x 1 and x 2 are semantically close and assuming semantically close input symbols have similar outputs, we also have that h x1 ρ ∼ h y2 and h x2 ρ ∼ h y1. Therefore it follows that h x1 ρ ∼ h x2 (and h y1 ρ ∼ h y2). Putting it differently, semantic similarity of input and output symbols translates into closeness of their distributed representations under ρ, in a way that is consistent with the model. Note that any model P model (y | x) parametrised by a continuous function can be approximated by a function in the form of Equation. It follows that any model that produces a probability distribution has an optimal representation space. Also note that the optimal space for the inputs does not necessarily have to come from the final layer before the softmax projection but instead can be constructed from any layer, as we now demonstrate. Let n be the index of the final activation before the softmax projection and let k ∈ {1, . . ., n}. We split the network into three parts: DISPLAYFORM2 where G k contains first k layers, F n contains the remaining n − k layers and U is the softmax projection matrix. Let the space for inputs H x be defined as DISPLAYFORM3 and the space for outputs H y defined as DISPLAYFORM4 where DISPLAYFORM5 is again an optimal representation space. We will show a specific example where this holds in Section 3.3. For the remainder of this paper, we focus on unsupervised models for learning distributed representations of sentences, an area of particular interest in NLP. si consists of words from a pre-defined vocabulary V of size |V |.We transform the corpus into a set of pairs DISPLAYFORM0, where s i ∈ S and c i is a context of s i. The context usually (but not necessarily) contains some number of surrounding sentences of s i, e.g. c i = (s i−1, s i+1).We are interested in modelling the probability of a context c given a sentence s. In general DISPLAYFORM1 One popular way to model P (c | s) for sentence-level data is suggested by the encoder-decoder framework. The encoder E produces a fixed-length vector representation h We first consider encoder-decoder architectures with a log-linear BOW decoder for the context. Let h i = E(s i) be a sentence representation of s i produced by some encoder E. The nature of E is not important for our analysis; for concreteness, the reader can consider a model such as FastSent BID19, where E is a BOW (sum) encoder. In the case of the log-linear BOW decoder, words are conditionally independent of the previously occurring sequence, thus Equation becomes DISPLAYFORM0. where u w ∈ R d is the output word embedding for a word w and h i is the encoder output. (Biases are omitted for brevity.)The objective is to maximise the model probability of contexts c i given sentences s i across the corpus D, which corresponds to finding the Maximum Likelihood Estimator (MLE) for the trainable parameters θ: DISPLAYFORM1 By switching to the negative log-likelihood and inserting the above expression, we arrive at the following optimisation problem: DISPLAYFORM2 Noticing that DISPLAYFORM3 we see that the objective in Equation forces the sentence representation h i to be similar under dot product to its context representation c i, which is simply the sum of the output embeddings of the context words. Simultaneously, output embeddings of words that do not appear in the context of a sentence are forced to be dissimilar to its representation. Figure 1: Unrolling a RNN decoder at inference time. The initial hidden state for the decoder is typically the encoder output, either the recurrent cell final state for a RNN encoder, or the sum of the input word embeddings for a BOW encoder. At the first time step, a learned <GO> token is presented as the input. In subsequent time steps, a probability-weighted sum over word vectors is used. The decoder is then unrolled for a fixed number of steps. The hidden states are then concatenated to produce the unrolled decoder embedding. In the models evaluated in Section 4, this process is performed for the RNN corresponding to the previous and next sentences. The sentence representation is then taken as the concatenation across both RNNs. Using dot ∼ to denote close under dot product, we find that if two sentences s i and s j have similar DISPLAYFORM4 Putting it differently, sentences that occur in related contexts are assigned representations that are similar under the dot product. Hence we see that the encoder output equipped with the dot product constitutes an optimal representation space as defined in Section 2. Another common choice for the context decoder is an RNN decoder DISPLAYFORM0 where h i = E(s i) is the encoder output. The specific structure of E is again not important for our analysis. (When E is also an RNN, this is similar to SkipThought .)The time unrolled states of decoder are converted to probability distributions over the vocabulary, conditional on the sentence s i and all the previously occurring words. Equation becomes DISPLAYFORM1 Similarly to Equation FORMULA11, MLE for the model parameters θ can be found as DISPLAYFORM2 Using ⊕ to denote vector concatenation, we note that DISPLAYFORM3 where the sentence representation h 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10Number of unroll steps Spearman correlation coefficient Figure 2: Performance on the STS tasks depending on the number of unrolled hidden states of the decoders, using dot product as the similarity measure. The top row presents for the RNN encoder and the bottom row for the BOW encoder. Red: Raw encoder output with BOW decoder. Green: Raw encoder output with RNN decoder. Blue: Unrolled RNN decoder output. Independent of the encoder architecture, unrolling even a single state of the decoder always outperforms the raw encoder output with RNN decoder, and almost always outperforms the raw encoder output with BOW decoder for some number of unrolls.the context words. Hence we can come to the same as in the log-linear case, except we have order-sensitive representations as opposed to unordered ones. As before, h D i is forced to be similar to the context c i under dot product, and is made dissimilar to sequences of u w that do not appear in the context. The "transitivity" argument from Section 3.2 remains intact, except the length of decoder hidden state sequences might differ from sentence to sentence. To avoid this problem, we can formally treat them as infinite-dimensional vectors in 2 with only a finite number of initial components occupied by the sequence and the rest set to zero. Alternatively, we can agree on the maximum sequence length, which in practice can be determined from the training corpus. Regardless, the above space of unrolled concatenated decoder states, equipped with dot product, is the optimal representation space for models with recurrent decoders. Consequently, this space could be a much better candidate for unsupervised similarity tasks. We refer to the method of accessing the decoder states at every time step as unrolling the decoder, illustrated in Figure 1. Note that accessing the decoder output does not require re-architecting or retraining the model, yet gives a potential performance boost on unsupervised similarity tasks almost for free. We will demonstrate the effectiveness of this technique empirically in Section 5. We have seen in Section 2 that the optimal representation space for a given model depends on the choice of decoder architecture. To support this theory, we train several encoder-decoder architectures for sentences with the decoder types analysed in Section 3, and evaluate them on downstream tasks using both their optimal space and the standard space of the encoder output as the sentence representations. Models and training. Each model has an encoder for the current sentence, and decoders for the previous and next sentences. As our analysis is independent of encoder type, we train and evaluate models with BOW and RNN encoders, two common choices in the literature for sentence representation learners BID19. The BOW encoder is the sum of word vectors BID19. The RNN encoder and decoders are Gated Recurrent Units (GRUs) BID12. using dot product as the similarity measure. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN encoder: Unrolling the RNN decoders using the concatenation of the decoder hidden states (RNN-concat) dramatically improves the performance across all tasks compared to using the raw encoder output (RNN-RNN), validating the theoretical justification presented in Section 3.3. BOW encoder: Unrolling the RNN decoders improves performance overall, however, the improvement is less drastic than that observed for the RNN encoder, which we discuss further in the main text. Using the notation ENC-DEC, we train RNN-RNN, RNN-BOW, BOW-BOW, and BOW-RNN models. For each encoder-decoder combination, we test several methods of extracting sentence representations to be used in the downstream tasks. First, we use the standard choice of the final output of the encoder as the sentence representation. In addition, for models that have RNN decoders, we unroll between 1 and 10 decoder hidden states. Specifically, when we unroll n decoder hidden states, we take the first n hidden states from each of the decoders and concatenate them in order to get the ing sentence representation. We refer to these representations as *-RNN-concat. All models are trained on the Toronto Books Corpus, a dataset of 70 million ordered sentences from over 7,000 books. The sentences are pre-processed such that tokens are lower case and splittable by space. Evaluation tasks. We use the SentEval tool BID13 to benchmark sentence embeddings on both supervised and unsupervised transfer tasks. The supervised tasks in SentEval include paraphrase identification (MSRP) BID14, movie review sentiment (MR) BID36, product review sentiment (CR), BID20 ), subjectivity (SUBJ) BID35, opinion polarity (MPQA) BID46, and question type (TREC) BID45 BID40. In addition, there are two supervised tasks on the SICK dataset, entailment and relatedness (denoted SICK-E and SICK-R) BID29. For the supervised tasks, SentEval trains a logistic regression model with 10-fold cross-validation using the model's embeddings as features. The unsupervised Semantic Textual Similarity (STS) tasks are STS12-16 BID11 BID2 BID1 BID5, which are scored in the same way as SICK-R but without training a new supervised model; in other words, the embeddings are used to directly compute similarity. We use dot product to compute similarity as indicated by our analysis; and discussion using cosine similarity, which is canonical in the literature, are presented in Appendix B. For more details on all tasks and the evaluation strategy, see BID13.Implementation and hyperparameters. Our goal is to study how different decoder types affect the performance of sentence embeddings on various tasks. To this end, we use identical hyperparameters and architecture for each model (except encoder and decoder types), allowing for a fair headto-head comparison. Specifically, for RNN encoders and decoders we use a single layer GRU with layer normalisation BID8. All the weights (including word embeddings) are initialised uniformly over [−0.1, 0.1] and trained with Adam without weight decay or dropout BID24. Sentence length is clipped or zero-padded to 30 tokens and end-of-sentence tokens are used throughout training and evaluation. Following, we use a vocabulary size of 20k with vocabulary expansion, 620-dimensional word embeddings, and 2400 hidden units in all RNNs. Performance of the unrolled models on the STS tasks is presented in Figure 2. We note that unrolling even a single state of the decoder always improves the performance over the raw encoder output with the RNN decoder, and nearly always does so for the BOW decoder for some number of unrolled hidden states. We observe that the performance tends to peak around 2-3 hidden states and fall off afterwards. In principle, one might expect the peak to be around the average sentence length of the corpus. A possible explanation of this behaviour is the "softmax drifting effect". As there is no context available at inference time, we generate the word embedding for the next time step using the softmax output from the previous time step (see Figure 1). Given that for any sentence, there is no single correct context, the probability distribution over the next words in that context will be multi-modal. This will flatten the softmax and produce inputs for the decoder that diverge from the inputs it expects (i.e. word vectors for the vocabulary). Further work is needed to understand this and other possible causes in detail. Performance across unsupervised similarity tasks is presented in Table 1 and performance across supervised transfer tasks is presented in TAB2. For the unrolled architectures, in these tables we report on the one that performs best on the STS tasks. When the encoder is an RNN, the supervised transfer validate our claims in Section 3.3. The are less conclusive when the encoder is a BOW. We believe this is caused by the simplicity of the BOW encoder forcing its outputs to obey the sentence-level distributional hypothesis irrespective of decoder type, ing in multiple candidates for the optimal representation space, but this should be investigated with a detailed analysis in future work. In addition, see Appendix A for a comparison with the original SkipThought from the literature, and Appendix B for using cosine similarity rather than dot product as the similarity measure in STS tasks, as is the canonical choice. When we look at the performance on supervised transfer in TAB2, combined with the similarity in Table 1, we see that the notion that models cannot be good at both supervised transfer and unsupervised similarity tasks needs refining; for example, RNN-RNN achieves strong performance on supervised transfer, while RNN-RNN-concat achieves strong performance on unsupervised similarity. In general, our indicate that a single model may be able to perform well on different downstream tasks, provided that the representation spaces chosen for each task are allowed to differ. Curiously, the unusual combination of a BOW encoder and concatenation of the RNN decoders leads to the best performance on most benchmarks, even slightly exceeding that of some supervised models on some tasks BID13. This architecture may be worth investigating. In this work, we introduced the concept of an optimal representation space, where semantic similarity directly corresponds to distance in that space, in order to shed light on the performance gap between simple and complex architectures on downstream tasks. In particular, we studied the space of initial hidden states to BOW and RNN decoders (typically the outputs of some encoder) and how that space relates to the training objective of the model. For BOW decoders, the optimal representation space is precisely the initial hidden state of the decoder equipped with dot product, whereas for RNN decoders it is not. Noting that it is precisely these spaces that have been used for BOW and RNN decoders has led us to a simple explanation for the observed performance gap between these architectures, namely that the former has been evaluated in its optimal representation space, whereas the latter has not. Furthermore, we showed that any neural network that outputs a probability distribution has an optimal representation space. Since a RNN does produce a probability distribution, we analysed its objective function which motivated a procedure of unrolling the decoder. This simple method allowed us to extract representations that are provably optimal under dot product, without needing to retrain the model. We then validated our claims by comparing the empirical performance of different architectures across transfer tasks. In general, we observed that unrolling even a single state of the decoder always outperforms the raw encoder output with RNN decoder, and almost always outperforms the raw encoder output with BOW decoder for some number of unrolls. This indicates different vector embeddings can be used for different downstream tasks depending on what type of representation space is most suitable, potentially yielding high performance on a variety of tasks from a single trained model. Although our analysis of decoder architectures was restricted to BOW and RNN, others such as convolutional BID49 and graph BID25 decoders are more appropriate for many tasks. Similarly, although we focus on Euclidean vector spaces, hyperbolic vector spaces BID34, complex-valued vector spaces BID44 and spinor spaces BID23 all have beneficial modelling properties. In each case, although an optimal representation space should exist, it is not clear if the intuitive space and similarity measure is the optimal one. However, there should at least exist a mapping from the intuitive choice of space to the optimal space using a transformation provided by the network itself, as we showed with the RNN decoder. Evaluating in this space should further improve performance of these models. We leave this for future work. Ultimately, a good representation is one that makes a subsequent learning task easier. For unsupervised similarity tasks, this essentially reduces to how well the model separates objects in the chosen representation space, and how appropriately the similarity measure compares objects in that space. Our findings lead us to the following practical advice: i) Use a simple model architecture where the optimal representation space is clear by construction, or ii) use an arbitrarily complex model architecture and analyse the objective function to reveal, for a chosen vector representation, an appropriate similarity metric. We hope that future work will utilise a careful understanding of what similarity means and how it is linked to the objective function, and that our analysis can be applied to help boost the performance of other complex models. 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10Number of unroll steps Spearman correlation coefficient Figure 3: Performance on the STS tasks depending on the number of unrolled hidden states of the decoders, using cosine similarity as the similarity measure. The top row presents for the RNN encoder and the bottom row for the BOW encoder. Red: Raw encoder output with BOW decoder. Green: Raw encoder output with RNN decoder. Blue: Unrolled RNN decoder output. For both RNN and BOW encoders, unrolling the decoder strictly outperforms *-RNN for almost every number of unroll steps, and perform nearly as well as or better than *-BOW.A COMPARISON WITH SKIPTHOUGHT Table 3: Performance of the SkipThought model, with and without layer normalisation BID8, compared against the RNN-RNN model used in our experimental setup. On each task, the highest performing model is highlighted in bold. For SICK-R, we report the Pearson correlation, and for STS14 we report the Pearson/Spearman correlation with human-provided scores. For all other tasks, reported values indicate test accuracy. † indicates taken from BID13. ‡ indicates our from running SentEval on the model downloaded from BID8's publicly available codebase (https://github.com/ryankiros/layer-norm). We attribute the discrepancies in performance to differences in experimental setup or implementation. However, we expect our unrolling procedure to also boost SkipThought's performance on unsupervised similarity tasks, as we show for RNN-RNN in our fair singlecodebase comparisons in the main text. As discussed in Section 3, the objective function is maximising the dot product between the BOW decoder/unrolled RNN-decoder and the context. However, as other researchers in the field and the STS tasks specifically use cosine similarity by default, we present the using cosine similarity in TAB4 and the for different numbers of unrolled hidden decoder states in Figure 3.Although the in TAB4 are consistent with the dot product in Table 1, the overall performance across STS tasks is noticeably lower when dot product is used instead of cosine similarity to determine semantic similarity. Switching from using cosine similarity to dot product transitions from considering only angle between two vectors, to also considering their length. Empirical studies have indicated that the length of a word vector corresponds to how sure of its context the model that produces it is. This is related to how often the model has seen the word, and how many different contexts it appears in (for example, the word vectors for "January" and "February" have similar norms, however, the word vector for "May" is noticeably smaller) BID41. Using the raw encoder output (RNN-RNN) achieves the lowest performance across all tasks. Unrolling the RNN decoders dramatically improves the performance across all tasks compared to using the raw encoder RNN output, validating the theoretical justification presented in Section 3.3. BOW encoder: We do not observe the same uplift in performance from unrolling the RNN encoder compared to the encoder output. This is consistent with our findings when using dot product (see Table 1). A corollary is that longer sentences on average have shorter norms, since they contain more words which, in turn, have appeared in more contexts BID0. During training, the corpus can induce differences in norms in a way that strongly penalises sentences potentially containing multiple contexts, and consequently will disfavour these sentences as similar to other sentences under the dot product. This induces a noise that potentially renders the dot product a less useful metric to choose for STS tasks than cosine similarity, which is unaffected by this issue.using dot product as the similarity measure. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. A practical downside of the unrolling procedure described in Section 3.3 is that concatenating hidden states of the decoder leads to very high dimensional vectors, which might be undesirable due to memory or other practical constraints. An alternative is to instead average the hidden states, which also corresponds to a representation space in which the training objective optimises the dot product as a measure of similarity between a sentence and its context. We refer to this model choice as *-RNN-mean. Results on similarity and transfer tasks for BOW-RNN-mean and RNN-RNN-mean are presented in TAB7 respectively, with for the other models from Section 5 included for completeness. While the strong performance of RNN-RNN-mean relative to RNN-RNN is consistent with our theory, exploring why it is able to outperform RNN-concat experimentally on STS tasks is left to future work.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

By introducing the notion of an optimal representation space, we provide a theoretical argument and experimental validation that an unsupervised model for sentences can perform well on both supervised similarity and unsupervised transfer tasks.

Represent the TLDR for finding the corresponding scientific article!

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.