Question: Represent the scientific article to find a short TLDR:

Question: We investigate the internal representations that a recurrent neural network (RNN) uses while learning to recognize a regular formal language. Specifically, we train a RNN on positive and negative examples from a regular language, and ask if there is a simple decoding function that maps states of this RNN to states of the minimal deterministic finite automaton (MDFA) for the language. Our experiments show that such a decoding function indeed exists, and that it maps states of the RNN not to MDFA states, but to states of an {\em abstraction} obtained by clustering small sets of MDFA states into ``''superstates''. A qualitative analysis reveals that the abstraction often has a simple interpretation. Overall, the suggest a strong structural relationship between internal representations used by RNNs and finite automata, and explain the well-known ability of RNNs to recognize formal grammatical structure. Recurrent neural networks (RNNs) seem "unreasonably" effective at modeling patterns in noisy realworld sequences. In particular, they seem effective at recognizing grammatical structure in sequences, as evidenced by their ability to generate structured data, such as source code (C++, LaTeX, etc.), with few syntactic grammatical errors BID9. The ability of RNNs to recognize formal languages -sets of strings that possess rigorously defined grammatical structure -is less well-studied. Furthermore, there remains little systematic understanding of how RNNs recognize rigorous structure. We aim to explain this internal algorithm of RNNs through comparison to fundamental concepts in formal languages, namely, finite automata and regular languages. In this paper, we propose a new way of understanding how trained RNNs represent grammatical structure, by comparing them to finite automata that solve the same language recognition task. We ask: Can the internal knowledge representations of RNNs trained to recognize formal languages be easily mapped to the states of automata-theoretic models that are traditionally used to define these same formal languages? Specifically, we investigate this question for the class of regular languages, or formal languages accepted by finite automata (FA).In our experiments, RNNs are trained on a dataset of positive and negative examples of strings randomly generated from a given formal language. Next, we ask if there exists a decoding function: an isomorphism that maps the hidden states of the trained RNN to the states of a canonical FA. Since there exist infinitely many FA that accept the same language, we focus on the minimal deterministic finite automaton (MDFA) -the deterministic finite automaton (DFA) with the smallest possible number of states -that perfectly recognizes the language. Our experiments, spanning 500 regular languages, suggest that such a decoding function exists and can be understood in terms of a notion of abstraction that is fundamental in classical system theory. An abstraction A of a machine M (either finite-state, like an FA, or infinite-state, like a RNN) is a machine obtained by clustering some of the states of M into "superstates". Intuitively, an abstraction Figure 1: t-SNE plot (Left) of the hidden states of a RNN trained to recognize a regular language specified by a 6-state DFA (Right). Color denotes DFA state. The trained RNN has abstracted DFA states 1(green) and 2(blue) (each independently model the pattern * ) into a single state. A loses some of the discerning power of the original machine M, and as such recognizes a superset of the language that M recognizes. We observe that the states of a RNN R, trained to recognize a regular language L, commonly exibit this abstraction behavior in practice. These states can be decoded into states of an abstraction A of the MDFA for the language, such that with high probability, A accepts any input string that is accepted by R. Figure 1 shows a t-SNE embedding BID13 of RNN states trained to perform language recognition on strings from the regex [(( {2}+)?)3+]. Although the MDFA has 6 states, we observe the RNN abstracting two states into one. Remarkably, a linear decoding function suffices to achieve maximal decoding accuracy: allowing nonlinearity in the decoder does not lead to significant gain. Also, we find the abstraction has low "coarseness", in the sense that only a few of the MDFA states need be clustered, and a qualitative analysis reveals that the abstractions often have simple interpretations. RNNs have long been known to be excellent at recognizing patterns in text BID10 BID9. Extensive work has been done on exploring the expressive power of RNNs. For example, finite RNNs have been shown to be capable of simulating a universal Turing machine BID16. BID4 showed that the hidden state of a RNN can approximately represent dynamical systems of the same or less dimensional complexity. In particularly similar work, BID19 showed that second order RNNs with linear activation functions are expressively equivalent to weighted finite automata. Recent work has also explored the relationship between RNN internals and DFAs through a variety of methods. Although there have been multiple attempts at having RNNs learn a DFA structure based on input languages generated from DFAs and push down automata BID3 BID5 BID7 BID14 BID17, most work has focused on extracting a DFA from the hidden states of a learned RNN. Early work in this field BID6 demonstrated that grammar rules for regular grammars could indeed be extracted from a learned RNN. Other studies BID18 tried to directly extract a DFA structure from the internal space of the RNN, often by clustering the hidden state activations from input stimuli, noting the transitions from one state to another given a particular new input stimuli. Clustering was done by a series of methods, such as K-Nearest Neighbor BID2, K-means BID11, and Density Based Spatial Clustering of Applications with Noise (DBSCAN) BID2 BID12. Another extraction effort BID1 uses spectral algorithm techniques to extract weighted automata from RNNs. Most recently, BID21 have achieved state-of-the-art accuracy in DFA extraction by utilizing the L* query learning algorithm. Our work is different from these efforts in that we directly relate the RNN to a ground-truth minimal DFA, rather than extracting a machine from the RNN's state space. The closest piece of related work is by BID20. Like our work, this seeks to relate a RNN state with the state of a DFA. However, the RNN in BID20 exactly mimics the DFA; also, the study is carried out in the context of a few specific regular languages that are recognized by automata with 2-3 states. In contrast, our work does not require exact behavioral correspondence between RNNs and DFAs: DFA states are allowed to be abstracted, leading to loss of information. Also, in our approach the mapping from RNN states to FA states can be approximate, and the accuracy of the mapping is evaluated quantitatively. We show that this allows us to establish connections between RNNs and DFAs in the setting of a broad class of regular languages that often demand significantly larger automata (with up to 14 states) than those studied by BID20. We start by introducing some definitions and notation. A formal language is a set of strings over a finite alphabet Σ of input symbols. A Deterministic Finite Automaton (DFA) is a tuple A = (Q, Σ, δ, q 0, F) where Q is a finite set of states, Σ is a finite alphabet, δ: Q × Σ → Q is a deterministic transition function, q 0 ∈ Q is a starting state and F ⊂ Q is a set of accepting states. A reads strings over Σ symbol by symbol, starting from the state q 0 and making state transitions, defined by δ, at each step. It accepts the string if it reaches a final accepting state in F after reading it to the end. The set of strings accepted by a DFA is a special kind of formal language, known as a regular language. A regular language L can be accepted by multiple DFAs; such a DFA A L is minimal if there exists no other DFA A = A L such that A exactly recognizes L and has fewer states than A L. It can be shown that this minimal DFA (MDFA), which we denote by A 0 L, is unique BID8.Abstractions. A Nondeterministic Finite Automaton (NFA) is similar to a DFA, except that the deterministic transition function δ is now a non-deterministic transition relation δ N F A. This means that for a state q in the NFA and a ∈ Σ, we have that δ N F A (q, a) is now a subset of NFA states. For a given regular language L we denote by A n L a Nondeterministic Finite Automaton (NFA) with n states that recognizes a superset of the language L. An abstraction map is a map α: DISPLAYFORM0 DISPLAYFORM1 We define the coarseness of an abstraction A n L, as the number of applications of α on the MDFA required to arrive at A n L. Intuitively, repeated applications of α create NFAs that accept supersets of the language L recognized by the MDFA, and can hence be seen as coarse-grained versions of the original MDFA. The coarsest NFA, given by A DISPLAYFORM2, is a NFA with only one accepting node and it accepts all strings on the alphabet Σ.Given a regular language L, we define R L to be a RNN that is trained to recognize the language L, with a certain threshold accuracy. Each RNN R L will have a corresponding set of hidden states denoted by H. More details about the RNN are provided in §4.1. Note that a RNN can also be viewed as a transition system with 5-tuple R = (H, Σ, δ R, h 0, F R), where H is a set of possible'hidden' states (typically H ⊂ R K), δ R is the transition function of the trained RNN, h 0 is the initial state of the RNN, and F R is the set of accepting RNN states. The key distinction between a DFA and a RNN is that the latter has a continuous state space, endowed with a topology and a metric. Decoding DFA States from RNNs. Inspired by methods in computational neuroscience BID0, we can define a decoding function or decoder f: H → Q 0 as a function from the hidden states of a RNN R L to the states of the corresponding DISPLAYFORM3. We are interested in finding decoding functions that provide an insight into the internal knowledge representation of the RNN, which we quantify via the decoding and transitional accuracy metrics defined below. DISPLAYFORM4 repeatedly n times, and let Q n be the set of states of A n L. We can define an abstraction decoding functionf: DISPLAYFORM5, that is the composition of f with α |n|. The function α |n| is the function obtained by taking n compositions of α with itself. Given a dataset of input strings D ⊂ Σ *, we can define the decoding accuracy of a mapf for an abstraction DISPLAYFORM6 where 1(C) is the boolean indicator function that evaluates to 1 if condition C is true and to 0 otherwise, h t+1 = δ R (h t, a t) and q t+1 = δ 0 (q t, a t). Note in particular, that for decoding abstraction states the condition is only checking if the (t + 1) RNN state is mapped to the (t + 1) NFA state byf, which may be true even if the (t + 1) RNN state is not mapped to the (t + 1) MDFA state by the decoding function f. Therefore a functionf can have a high decoding accuracy even if the underlying f does not preserve transitions. Decoding Abstract State Transitions. We now define an accuracy measure that takes into account how well transitions are preserved by the underlying function f.Intuitively, for a given decoding functionf and NFA A n L, we want to check whether the RNN transition on a is mapped to the abstraction of the MDFA transition on a. We note that in the definition of the decoding function, we take into account only the states at (t + 1) and not the underlying transitions in the original MDFA A 0 L, unlike we do here. More precisely, the transitional accuracy of a mapf for a given RNN and abstraction, with respect to a data-set D, is defined as: DISPLAYFORM7 Our experiments in the next section demonstrate that decoding functions with high decoding and transitional accuracies exist for abstractions with relatively low coarseness. Our goal is to experimentally test the hypothesis that a high accuracy, low coarseness decoder exists from R L to A ∈ L) example strings (see Appendix for details). We then train R L with D on the language recognition task: given an input string x ∈ Σ *, is x ∈ L? Thus, we have two language recognition models corresponding to state transition systems from which state sequences are extracted. Given a length T input string x = (x 1, x 2, x t, ..., x T) ∈ D, let the categorical states generated by A 0 L be denoted by q = (q 0, q 1, q t, ..., q T) and continuous states generated by the R L be h = (h 0, h 1, h t, ..., h T). The recorded state trajectories (q and h) for all input strings x ∈ D are used as inputs into our analysis.. For our experiments, we sample a total of ∼ 500 unique L, and thus perform an analysis of ∼ 500 recognizer MDFAs and ∼ 500 trained recognizer RNNs. As mentioned in the beginning of §4, we must first determine what is a reasonable form for the decoders f andf to ensure high accuracy on the decoding task. FIG4 shows decoding accuracy DISPLAYFORM0 for several different decoding functions f. We test two linear classifiers (Multinomial Logistic Regression and Linear Support Vector Machines (SVM)) and two non-linear classifiers (SVM with a RBF kernel, Multi-Layer Perceptrons with varying layers and hidden unit sizes). In order to evaluate whether accuracy varies significantly amongst all decoders, we use a statistically appropriate F-test. Surprisingly, we find there to be no statistical difference umong our sampled languages: the nonlinear decoders achieve no greater accuracy than the simpler linear decoders. We also observe in our experiments that as the size of the MDFA M increases, the decoding accuracy decreases for all decoders in a similar manner. FIG4 shows this relationship for the multinomial logistic regression classifier. Taken together, these have several implications. First, we find that a highly expressive nonlinear decoder does not yield any increase in decoding accuracy, even as we scale up in MDFA complexity. We can conclude from this finding and our extensive hyperparameter search for each decoder model that the decoder models we chose are expressive enough for the decoding task. Second, we find that decoding accuracy for MDFA states is in general not very high. These two observations suggest linear decoders are sufficient for the decoding task, but also suggests the need for a different interpretation of the internal representation of the trained RNN. Given the information above, how is the hidden state space of the R L organized? One hypothesis that is consistent with the observations above is that the trained RNN reflects a coarse-grained abstraction of the state space Q 0 (Figure 1), rather than the MDFA states themselves. To test this hypothesis, we propose a simple greedy algorithm to find an abstraction mapping α: (a) given an NFA A n L with n unique states in Q n, consider all (n − 1)-partitions of Q n−1 (i.e. two NFA states s, s have merged into a single superstate {s, s}); (b) select the partition with the highest decoding accuracy; (c) Repeat this iterative merging process until only a 2-partition remains. We note that this algorithm does not explicitly take into consideration the transitions between states which are essential to evaluating φf (R L, A n L). Instead, the transitions are taken into account implicitly while learning the decoder f at each iteration of the abstraction algorithm. Decreasing the number of states in a classification trivially increases DISPLAYFORM0. We compare to a baseline where the states abstracted are random to validate our method. We compute the normalized Area Under the Curve (AUC) of a decoder accuracy vs coarseness plot. Higher normalized AUC indicates a more accurate abstraction process. We argue through FIG5 that our method gives a non-trivial increase over the abstraction performance of a random baseline. The abstraction algorithm is greedy in the sense that we may not find the globally optimal partition (i.e. with the highest decoding accuracy and lowest coarseness), but an exhaustive search over all partitions is computationally intractable. The greedy method we have proposed has O(M 2) complexity instead, and in practice gives satisfactory . Despite it being greedy, we note that the ing sequence of clusterings are stable with respect to randomly chosen initial conditions and model parameters. Recognizer RNNs with a different number of hidden units in clustering sequences that are consistent with each other in the critical first few abstractions. Once an abstraction α has been found, we can evaluate whether the learned abstraction decoderf is of high accuracy, and whether the α found is of low coarseness. Results showing the relationship between high decoding accuracy ρf (R L, A n L) as a function of coarseness is presented in FIG6 conditioned on the number of nodes in the original MDFA. As stated in §4.2, as M increases, ρf (R L, A n L) decreases on the MDFA (i.e. n = 0). We attribute this to two factors, as M increases, the decoding problem naturally increases in difficulty, and R L abstracts multiple states of A L into a single state in H as can be seen empirically from Figure 1. We validate the second factor by training a overparameterized non-linear decoder on the decoding task and find no instances where the decoder obtains 0% training error. Alongside the decoding accuracy, we also present transitional accuracy φf (R L, A n L) as a function of coarseness FIG6. Both of these figures showcase that for a given DFA, in general we can find a low coarseness NFA that the hidden state space of R L can be decoded to with high accuracy. FIG5 shows the average ratio of abstractions relative to M needed to decode to 90% accuracy, indicating low coarseness relative to a random baseline. For completeness, we also present decoder and transition accuracy for a nonlinear decoder in FIG7 showing similar as the linear decoder. Our fundamental work shows a large scale analysis of how RNNs R L relate to abstracted NFAs A n L for hundreds of minimal DFAs, most of which are much larger and more complex than DFAs typically used in the literature. By evaluating the transition accuracy between R and A n L we empirically validate our claim. We show that there does exist high accuracy decoders from R to an abstracted NFA A n L. 4.5 INTERPRETING THE RNN HIDDEN STATE SPACE WITH RESPECT TO THE MINIMAL DFA With an established high accuracyf with low coarseness α reveals a unique interpretation of H with respect to A 0 L. Using α and f to relate the two, we uncover an interpretation of how R organizes H with respect to A n L ∀ n ∈ [M]. We can then determine the appropriate level of abstraction the network uses to accomplish the logical language recognition task in relation to the underlying MDFA. We provide two example'real-world' DFAs to illustrate this interpretation and show several interesting patterns. We present in FIG9 the clustering sequences of two regular expressions that have real-world interpretations, namely the SIMPLE EMAILS and DATES languages that recognize simple emails and simple dates respectively. To explain, FIG9 shows the DATES language with its clustering sequence superimposed on the MDFA in the form of a dendrogram. The dendrogram can be read in a top-down fashion, which displays the membership of the MDFA states and the sequence of abstractions up to n = M − 1. A question then arises: How should one pick a correct level of abstraction n?. The answer can be seen in the corresponding accuracies ρf (R L, A n L) in FIG10. As n increases and the number of total NFA states decreases, the linear decoding (LDC) prediction task obviously gets easier (100% accuracy when the number of NFA states Q |Q|−1 is 1), and hence it is important to consider how to choose the number of abstractions in the final partition. We typically set a threshold for ρf (R L, A n L) and select the minimum n required to achieve the threshold accuracy. Consider the first two abstractions of the SIMPLE EMAILS DFA. We notice that both states 2 and 5 represent the pattern matching task [a-d] *, because they are agglomerated by the algorithm. Once two abstractions have been made, the decoder accuracy is at a sufficient point, as seen in FIG10. This suggests that the collection of hidden states for the two states are not linearly separable. One possible and very likely reason for this is the network has learned an abstraction of the pattern [a-d] * and uses the same hidden state space regardless of location in string to recognize this pattern, which has been indicated in past work BID9. This intuitive example demonstrates the RNN's capability to learn and abstract patterns from the DFA. This makes intuitive sense because R L does not have any direct access to A 0 L, only to samples generated from A 0 L. The flexibility of RNNs allows such abstractions to be created easily. The second major pattern that arises can be seen in the dendrogram in the bottom row of FIG9. We notice that, generally, multiple states that represent the same location in the input string get merged (1 and 4, 3 and 6, 0 and 5). The SIMPLE EMAILS dendrogram shows patterns that are location-independent, while the fixed length pattern in the DATES regex shows location-dependent patterns. We also notice that the algorithm tends to agglomerate states that are within close sequential proximity to each other in the DFA, again indicating location-dependent hierarchical priors. Overall, our new interpretation of H reveals some new intuitions, empirically backed by our decoding and transitional accuracy scores, regarding how the RNN R L structures the hidden state space H in the task of language recognition. We find patterns such as these in almost all of the DFA's tested. We provide five additional random DFA's in the appendix FIG4 ) to show the wide variability of the regular expressions we generate/evaluate on. We have studied how RNNs trained to recognize regular formal languages represent knowledge in their hidden state. Specifically, we have asked if this internal representation can be decoded into canonical, minimal DFA that exactly recognizes the language, and can therefore be seen to be the "ground truth". We have shown that a linear function does a remarkably good job at performing such a decoding. Critically, however, this decoder maps states of the RNN not to MDFA states, but to states of an abstraction obtained by clustering small sets of MDFA states into "abstractions". Overall, the suggest a strong structural relationship between internal representations used by RNNs and finite automata, and explain the well-known ability of RNNs to recognize formal grammatical structure. We see our work as a fundamental step in the larger effort to study how neural networks learn formal logical concepts. We intend to explore more complex and richer classes of formal languages, such as context-free languages and recursively enumerable languages, and their neural analogs. In order to generate a wide variety of strings that are both accepted and rejected by the DFA corresponding to a given regex R, we use the Xeger Java library, built atop the dk.brics.automaton library BID15. The Xeger library, given a regular expression, generates strings that are accepted by the regular expression's corresponding DFA. However, there is no standard method to generate examples that would be rejected by the DFA. These rejected examples need to be diverse to properly train an acceptor/rejector model: if the rejected examples are completely different from the accepted examples, the model will not be able to discern between similar input strings, even if one is an accepted string and the other is a rejected string. However, if the rejected examples were too similar to the accepted examples, the model would not be able to make a judgment on a completely new string that does not resemble any input string seen during training. In other words we want the rejected strings to be drawn from two distinct distributions, one similar and one independent compared to the distribution of the accepted strings. In order to achieve this, we generate negative examples in two ways: First, we randomly swap two characters in an accepted example enough times until we no longer have an accepted string. And secondly, we take an accepted string and randomly shuffle the characters, adding it to our dataset if the ing string is indeed rejected. In our experiments we generate 1000 training examples with a 50:50 accept/reject ratio. When applicable we generate strings of varying length capped at some constant, for example with the SIMPLE EMAILS language we generate strings of at most 20 characters.

Represent the short summary for finding the corresponding scientific article (long)

Finite Automata Can be Linearly decoded from Language-Recognizing RNNs using low coarseness abstraction functions and high accuracy decoders.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: Designing accurate and efficient convolutional neural architectures for vast amount of hardware is challenging because hardware designs are complex and diverse. This paper addresses the hardware diversity challenge in Neural Architecture Search (NAS). Unlike previous approaches that apply search algorithms on a small, human-designed search space without considering hardware diversity, we propose HURRICANE that explores the automatic hardware-aware search over a much larger search space and a multistep search scheme in coordinate ascent framework, to generate tailored models for different types of hardware. Extensive experiments on ImageNet show that our algorithm consistently achieves a much lower inference latency with a similar or better accuracy than state-of-the-art NAS methods on three types of hardware. Remarkably, HURRICANE achieves a 76.63% top-1 accuracy on ImageNet with a inference latency of only 16.5 ms for DSP, which is a 3.4% higher accuracy and a 6.35x inference speedup than FBNet-iPhoneX. For VPU, HURRICANE achieves a 0.53% higher top-1 accuracy than Proxyless-mobile with a 1.49x speedup. Even for well-studied mobile CPU, HURRICANE achieves a 1.63% higher top-1 accuracy than FBNet-iPhoneX with a comparable inference latency. HURRICANE also reduces the training time by 54.7% on average compared to SinglePath-Oneshot. Neural Architecture Search (NAS) is a powerful mechanism to automatically generate efficient Convolutional Neural Networks (CNNs) without requiring huge manual efforts of human experts to design good CNN models (; ;). However, most existing NAS methods focus on searching for a single DNN model of high accuracy but pay less attention on the performance of executing the model on hardware, e.g., inference latency or energy cost. Recent NAS methods (; ; b;) start to consider model-inference performance but they use FLOPs 1 to estimate inference latency or only consider the same type of hardware, e.g., smartphones from different manufacturers but all ARM-based. However, the emerging massive smart devices are equipped with very diverse processors, such as CPU, GPU, DSP, FPGA, and various AI accelerators that have fundamentally different hardware designs. Such a big hardware diversity makes FLOPs an improper metric to predict model-inference performance and calls for new trade-offs and designs for NAS to generate efficient models for diverse hardware. To demonstrate it, we conduct an experiment to measure the performance of a set of widely used neural network operators (a.k.a. operations) on three types of mobile processors: Hexagon TM 685 DSP, Snapdragon 845 ARM CPU, and Movidius TM Myriad TM X Vision Processing Unit (VPU). Figure 1 shows the and we make the following key observations. First, from Figure 1 (a), we can see that even the operators have similar FLOPs, the same operator may have very different inference latency on different processors. For example, the latency of operator SEP 5 is nearly 12× higher than that of operator Choice 3 on the ARM CPU, but the difference on the VPU is less than 4×. Therefore, FLOPs is not the right metric to decide the inference latency on different hardware. Second, the relative effectiveness of different operators on different processors is also different. For example, operator SEP 3 has the smallest latency on the DSP, but operator Choice 3 has the 1 In this paper, the definition of F LOP s follows, i.e., the number of multiply-adds. smallest latency on the VPU. Thus, different processors should choose different operators for the best trade-off between model accuracy and inference latency. Furthermore, as shown in Figure 1 (b), the computational complexity and latency of the same operator are also affected by the execution context, such as input feature map shapes, number of channels, etc. Such a context is determined by which layer the operator is placed on. As a , even on the same hardware, optimal operators may change at different layers of the network. In addition, we observe that the existing NAS methods tends to handle all the layers equally. For instance, the uniform sampling in one-shot NAS will give the same sampling opportunities to every layer. However, not all the layers are the same: different layers may have different impacts on inference latency and model accuracy. Indeed, some previous works (D.;) have revealed different behaviors between the earlier layers (close to data input) and the latter layers (close to classification output) in CNN models. The earlier layers extract low-level features from inputs (e.g., edges and colors), are computation intensive and demands more data to converge, while the latter layers capture high-level class-specific features but are less computation intensive. From these findings, we argue that exploring more architecture selections in the latter layers may help find better architectures with the limited sampling budget, and limiting the latency in the earlier layers is critical to search for low-latency models. To this end, it is desirable to explore how to leverage this layer diversity for better architecture sampling in NAS. Motivated by these observations, we argue that there is no one-size-fits-all model for different hardware, and thus propose and develop a novel hardware-aware method, called HURRICANE (Hardware aware one-shot neUral aRchitecture seaRch In Coordinate AsceNt framEwork), to tackle the challenge of hardware diversity in NAS. Different from the existing hardware-aware NAS methods that use a small set of operators (e.g., 6 or 9) manually selected for a specific hardware platform, HURRICANE is initialized with a large-size (32 in our implementation) candidate operators set to cover the diversity of hardware platforms. However, doing so increases the search space by many orders of magnitude and thus leads to unacceptable search and training cost and may even cause non-convergence problem. To reduce the cost, we propose hardware-aware search space reduction at both operator level and layer level. In the operator-level search space reduction, a toolkit is developed to automatically score every layer's candidate operators on target hardware platforms, and choose a sub-set of them with low latency for further utilization. In the layer-level search space reduction, we split the layers into two groups, the earlier group and the latter group according to their locations in the network. Based on a coordinate ascent framework (Appendix A), we propose a multistep search scheme, which searches the complete architecture by a sequence of simpler searching of sub-networks. In each iteration (step), we alternatively fix one group of layers and optimize the other group of layers to maximize the validation accuracy by a one-shot NAS 2. The searching of sub-networks is much easier to complete because of the much smaller size of search space, and the better architectures are reached by a sequence of iterations. This layer-level search space reduction is inspired by the layer diversity mentioned above. We choose most latencyeffective operators for earlier layers and allocate more sampling opportunities to latter layers. As a , we are able to search for models with both low latency and high accuracy. We evaluate the effectiveness of our proposed approach on ImageNet 2012 dataset and a small OUIAdience-Age dataset with the above three mobile hardware platforms (DSP/CPU/VPU). Under all the three platforms, HURRICANE consistently achieves the same level (or better) accuracy with much lower inference latency than state-of-the-art hardware-aware NAS methods. Remarkably, HURRICANE reduces the inference latency by 6.35× on DSP compared to FBNet-iPhoneX and 1.49× On VPU compared to Proxyless-mobile, respectively. Compared to Singlepath-Oneshot, on average HURRICANE reduces the training time by 54.7% on ImageNet. Neural Architecture Search (NAS). first proposed to use reinforcement learning (RL) to search for competitive architectures with low FLOPs. As a full configurable architecture search space grows exponentially, early works; ) search for a cell level structure as the building block and the same cell is reused in all layers. However, many cell structures are very complicated and fragmented, and thus are slow when deployed to a device. Recent methods;;; ) adopt a layer-level hierarchical search space with a back-bone structure allowing different layer structures at different resolution blocks of a network. The goal becomes searching operators for each layer so that the architecture achieves competitive accuracy under given constraints. To search hardware efficient architectures, the search spaces have been built on increasingly more efficient building blocks. MnasNet, ProxylessNAS and Single-path NAS (b) built upon the MobileNetV2 structure (MB k e). FBNet and Singlepath-Oneshot built search space by ShuffleNetV1 and ShuffleNetV2 (Choice k). As these structures are primarily designed for mobile CPU, the efficiency of such manually-designed search space is unknown for other hardware. Hardware aware NAS. Early NAS methods (; adopt hardwareagnostic metric FLOPs to measure the efficiency. However, architecture with lower FLOPs is not necessarily faster (a). Recently, many methods (; b;) adopt direct metrics such as measured latency but only for mobile CPUs. builds a latency prediction model, (b;) profiles every operator's latency, and then the latency metric is viewed as regularization loss. Such design is also not optimized for small mobile models because accuracy changes much more dramatically with latency for small models, as pointed out. One-Shot NAS. Starting from ENAS , weight sharing became popular as it accelerates the search process and makes search cost feasible. Recent one-shot methods encode the search space into an over-parameterized supernet, where each path is a stand-alone model. During the supernet training, architectures are sampled by different proxies (e.g., reinforcement learning, gradient-based) with weights updated. and Fair-NAS observe that such coupled architecture search and weight sharing could be problematic to fairly evaluate the performance of candidate architectures. Our work is built upon Singlepath-Oneshot . It decouples the supernet training and evolution-based architecture search by uniform sampling. In this paper, HURRICANE aims to search the following architectures for a given hardware platform h (any of CPU, DSP, NPU, VPU, etc.) and the latency constant LatC h: HURRICANE can find a architecture a that achieves the maximum accuracy ACC val (a) on the validation set and the inference latency Latency(a, h) is under the constraint LatC h. Without loss of generality, A denotes the network architecture search space. Like other recent works (; ; ;, A is constructed by a back-bone network architecture, n layers of which could be chosen from a set of candidate operators O. denotes the l-th layer of architecture a, and O l is the set of candidate operators specialized for l-th layer according to its context (e.g. input shape, number of channels). Algorithm 1 briefs the hardware aware one-shot neural architecture search in coordinate ascent framework. We employ the same back-bone architecture with but a much bigger collection of candidate operators from the primary blocks of off-the-shelf networks. The set of candidate operators (O) contains up to 32 operators detailed in Section 3.3. Require: h (hardware platform), LatC h (latency constraint), O (candidate operators) Require: I (number of iterations), t, e (hyper parameter) Operator level search space reduction 1: for l ← 1 to n do 2:, e is index of extra exploring operator and e ≥ 4 Layer level search space reduction and coordinate ascent 6: Initialize winning architecture awin with Firstly, we leverage the hardware performance to reduce the search space on operator level. We develop a toolkit to automatically score the candidate operators O l on different hardware platforms. The toolkit profiles the real performances (including but may not limited to inference latency), and give a comprehensive score for every candidate operator. Then we select top 4 or 5 operators (O l) in the non-increasing order of scores for l-th layer. Secondly, we reduce the search space on layer level. At the beginning of coordinate ascent, we setup the winning architecture with the operators of highest scores in every layer. In first iteration, we mark the latter n − t layers as active and the earlier t layers as non-active. The non-active layers are fixed to the corresponding layer structures of current winning architecture, while the active layers will be chosen from the 4 highest scored operators (an extra operator for exploring is added in the last 4 layers) (Line 8 and Line 9). The one-shot NAS method itself is similar to the work , except that we constraint the search space with a hardware latency 3 other than FLOPs. We encode all candidate architectures of search space A into a over parameterized supernet (and its weights W), and sample a specific architecture a (and its weights W (a)). The sampled network is denoted by N (a, W (a)). For every mini batch B in training dataset D tr, the supernet weights W is updated by the gradient (∇L B (N (a, W i (a)))) of the training loss of network N (a, W (a)) . After the supernet is trained (only once), all candidate architectures a inherit weights from the supernet W A, and thus weight W A (a) is used to approximate the optimal weight of architecture a. The weights will not be updated during the process of architecture searching. We adopt evolutionary search to find the winning architecture with the highest validation accuracy (Line 11). After a complete process of one-shot NAS, a new winning architecture would be generated. The new winning architecture would remain the same structure in the non-active layers and update to new operators in the active layers. A new iteration is started with the new winning architecture until the termination test is satisfied. Candidate Operator Pool O. As is known, the computation complexity (e.g., FLOPs) and memory access cost of an architecture are key factors in hardware latency efficiency . To cover the hardware diversity, our initial search space consists of 32 candidate operators (detailed structures are in Appendix Figure 3) that leverage different computation and memory complexity. They are built upon the following 4 basic structures from current efficient models: • SEP: depthwise-separable convolution. Following DARTS, we applied the depthwise-separable convolution twice. We allow choosing kernel size k of 3, 5 or 7 for the depthwise convolution and generate 3 operators SEP k. This rule with different kernel size for depthwise convolution also applies to other operator generation. SEP k has a larger FLOP count than others, but less memory access complexity. • MB: mobile inverted bottleneck convolution in MobileNetV2 . It's widely used in recent one-shot ). We generate 9 operators MB k e based on it, that we allow choosing k of 3, 5, 7 and channel expansion rate e of 1, 3, 6. MB k e has a medium memory access cost due to its shortcut and add operation. Its computation complexity is decided by the kernel size k and expansion rate e. • Choice: basic building block in ShuffleNetV2 . We construct 3 operators Choice k (k=3, 5, 7). Following , we also adds a similar operator ChoiceX. Choice k and ChoiceX have much smaller FLOPs than the others, but the memory complexity is high due to the channel split and concat operation. • SE: squeeze-and-excitation network . To balance the impacts in latency and accuracy, we follow the settings in MobileNetV3. We set the reduction ratio r to 4, and replace the original sigmoid function with a hard version of swish ReLU 6(x+3) 6. We apply SE module to the above operators and generate new 16 operators. The computation complexity of SE is decided by its insert position, while the memory access cost is relatively lower. As a , our global search space consists of n |O| =20 32 (n=20 layers) candidate architectures, which is exponentially larger than current one-shot NAS. For each searchable layer, there are total of 32 operators to choose: SEP k, SEP k SE,MB k e, MB k e SE, Choice k,Choice k SE,ChoiceX, ChoiceX SE, where k=3,5,7, and e=1,3,6. Hardware aware profiling and sorting. As shown in Figure 1(a), the real performances of operators vary significantly on different hardware platforms. Without loss of generality, the scoring function should consider both representation capacity and real hardware performance. There are many methods to approximate the scoring function, such as a customized weighted product of FLOPs and number of parameters (#Params) where α and β are non-negative constants. Construct search space with high scored operators. For each layer, we filter out the top 4 operators with highest scores first. This make the search space reduced to a comparable size with previous works, but specialized for the target platform. Another exploring operator is added to the candidate operator set of the last 4 layers. Only adding to the last layers is because the latency degradations of these layers are acceptable due to their smallest feature size; sophisticated operators could help these layers better capture high-level class-specific patterns. Hyper-parameters. There are two important hyper-parameters in Algorithm 1: the layer grouping boundary t and the number of iterations I. We set t = 8 (only searchable layers counted) according to the natural resolution changes of the supernet (Table 4 in Appendix). Experiment discussed in Appendix C shows the rationality of it. For simplicity, we tested the simplest one-iteration case (I = 1) and adding an extra iteration (I = 2). Diverse hardware platforms. HURRICANE targets three representative mobile hardware that is widely used for CNN deployment: DSP (Qualcomm's Hexagon TM 685 DSP),: CPU (Qualcomm's Snapdragon 845 ARM CPU),: VPU (Intel's Movidius TM Myriad TM X Vision Processing Unit). To make full utilization of these hardware at inference, we use the particular inference engine provided by the hardware vendor. Specifically, DSP and CPU latency are measured by Snapdragon Neural Processing Engine SDK , while VPU latency is measured by Intel OpenVINO TM Toolkit . Operators profiling and sorting. Table 1 lists the top scored operators on different hardware platforms when α = 0. The hardware-aware search space constructed by scoring the operators on target hardware platforms and then selecting the best 4 operators (layer 17-20 has another extra exploring operator with e = 10). We share new important insights from hardware profiling: (i) depthwise convolutions with kernel size k ≤ 3 are well optimized on Hexagon TM 685 DSP. As a , all the operators are of k=3 in search space. (ii) SE module is not supported by the AI accelerator of Myriad TM X VPU, and thus rolled back to relatively slow CPU execution. (iii) Even with complex memory operator, Choice 3 (i.e., ShuffleNetV2 unit) is the most efficient operator on VPU and CPU due to its much smaller FLOPs count. Latency predictor. To avoid measuring the latency of every candidate architecture, we build a latency-prediction model with high accuracy: the RMSE (root mean square error) of prediction model is 0.82ms, 21.84ms, and 0.03ms on DSP, CPU, VPU, respectively, which means an average 4.7%, 4.2%, and 0.08% latency estimated error for DSP, CPU and VPU. It suggests the latency prediction model can be used to replace the expensive direct hardware measurement with little error introduced. More details of latency predictor is in Appendix D. Latency constraints. As discussed in Appendix B, the latency constraint on a given platform should be in a meaningful range. For better comparison with other works, we set the latency constraints to be smaller than the best latency of models from other works, which are 310 ms (CPU), 17 ms (DSP) and 36 ms (VPU). Over-parameterized Supernet and Architecture Search. HURRICANE is built on top of Singlepath-Oneshot (; † : In the form "x(y)", where "x" means the training time reduction and "y" means the accuracy achieved. Table 4 ) consists of stem layers and n=20 searchable layers. Once the supernet training finishes, we perform a 20-iterations evolution search for total 1,000 architectures as Singlepath-Oneshot. For better fairness, the supernet is re-initialized randomly before every iteration. , we randomly split the original training set into two parts: 50,000 images for validation (50 images for each class exactly) and the rest as the training set. The original validation set is used for testing, on which all the evaluation are reported. Training Details. We follow most of the training settings and hyper-parameters used in SinglepathOneshot , with two exceptions: (i) For supernet training, the epochs change with different hardware-aware search spaces (listed in Table 1), and we stop at the same level training loss as Singlepath-Oneshot. (ii) For architecture retraining, we change linear learning rate decay to cosine decay from 0.4 to 0. The batch size is 1,024. Training uses 4 NVIDIA V100 GPUs. Results and Search Cost Analysis. Table 2 summarizes our experiment on ImageNet. HUR-RICANE surpasses state-of-the-art models, both manually and automatically designed: compared to MobileNetV2 (top-1 accuracy 72.0%), HURRICANE improves the accuracy by 2.59% to 4.03% on all target hardware platforms. Compared to models searched automatically, HURRICANE demonstrates that it's essential to leverage hardware diversity in NAS to achieve the best efficiency on different hardware platforms. Specially, compared to the most efficient models searched by NAS, HURRICANE (DSP) reaches a 6.35× inference speedup than FBNet-iPhoneX with 3.43% accuracy improvement, HURRICANE (CPU) achieves 1.39% higher accuracy with 11.7ms latency reduction, HURRICANE (VPU) achieves 0.83% higher accuracy with 3.1ms latency reduction. Remarkably, HURRICANE is the only hardware-aware NAS method that searches the better accuracy with much lower latency on all diverse hardware platforms. To compare the search cost, we report supernet training time reduction compared with SinglepathOneshot instead of exact GPU search days as for two reasons: (i): the GPU search days are highly relevant with the experiment environments (e.g., different GPU hardware) and the code implementation (e.g., ImageNet distributed training). (ii): The primary time cost comes from supernet training in Singlepath-Oneshot, as the evolution search is fast that architectures only perform inference. Table 2 shows that our method reduces an average 54.7% time if executes one step search in coordinate ascent (I = 1), which is almost a 2x training time speedup. In addition, HURRICANE already achieves better classification accuracy than other NAS methods at this step. It demonstrates the effectiveness of exploring more architecture selections in the latter layers. With one more step search, HURRICANE usually search better architectures but takes longer tme. Results suggest our method can still save an average 30.4% time (I = 2). FLOPs (higher accuracy) and the DSP hardware characteristics leverage in search space (lower latency). It also demonstrates again that FLOPs is an improper metric for hardware aware NAS. To further demonstrate the effectiveness HURRICANE, we compare it with recent work on a sequence of tasks. These tasks include the original FLOPs (330M) constrained architecture search in , the hardware latency constrained architecture search for different hardware platforms described in this paper. We do the experiments on ImageNet and also the OUIAdience-Age (OUI) for simplicity. For the fairness, we let the methods use the same search space for a single task. For the FLOPs constrained tasks, we use the original search space of . For those hardware latency constrained tasks, we let the methods use our specialized search space listed in Table 1. Dataset and Training Details. OUI-Adience-Age is a small 8-class dataset consisting of 17,000 face images. We split the images into training and testing test by 8:2 for architecture retraining. For architecture search, we randomly split the training set into two parts: 5,567 images for validation and the rest as the training set for supernet. We adopt the same hyperparameter settings as Singlepath-Oneshot, except that we reduce the initial learning rate from 0.5 to 0.1, and the batch size reduced from 1024 to 64. Supernet trains until converge. For the architecture retraining, we train for 400 epochs and change the original linear learning rate decay to Cyclic decay with a bound. We use 1 NVIDIA Telsa P100 for training. Results and Search Cost Analysis. Table 3 summarizes experiment . The searched models outperform the manual designed light-weight models, such as MobileNetV2 (top-1 acc: 72.00% on ImageNet, 85.67% on OUI-Adience-Age). For every row of the table, the proposed method could achieve not only higher accuracy but also better hardware efficiency for all the tasks. As shown in the column of Singlepath-Oneshot, our hardware-aware search space could also improve the accuracy (0.6%-0.8% on OUI-Adience-Age dataset). To illustrate the cost of supernet training, we listed the number of iterations consumed. As shown in Table 3, in most tasks, only one iteration (I = 1) of HURRICANE could achieve a comparable top-1 accuracy (or even better in some tasks), but the number of training iterations is significantly reduced (50%-77.5%). If the computation budget (e.g. training time) allows, HURRICANE can benefit from another iteration (I = 2). The accuracy is improved by 0.15%-1.06% with an additional cost of only 4.0%-23.3% of training iterations. In this paper, we propose HURRICANE to address the challenge of hardware diversity in NAS. By exploring hardware-aware search space and a multistep search scheme based on coordinate ascent framework, our solution achieves better accuracy and much lower latency on three hardware platforms than state-of-the-art hardware-aware NAS. And the searching cost (searching time) is also significantly reduced. For future work, we plan to support more diverse hardware and speed up more NAS methods. Coordinate descent (CD, or coordinate ascent, CA) are optimization algorithms to find the minimum (or maximum) of a function by successively performing approximate minimization (maximization) along coordinate directions or coordinate hyperplanes. They are iterative methods, in which we optimize along a selected coordinate direction or hyperplane at a time while keeping all others fixed. Though not sophisticated, the CD or CA methods could be surprisingly efficient and scalable in many real world scenarios. As it's extremely difficult to an optimal layersampling rates sequence with respect to different layers' importance for final accuracy, CA algorithm is much easier to achieve competitive with careful implementations. In the body of this paper, we assume that the latency constraint should be in a meaningful range, which means the constraint should neither be too small to achieve nor too great to make the problem degraded to an unbounded optimization for accuracy only. The proposed solution doesn't guarantee the performance in case that the latency constraint is out of the meaningful range. However, this could be patched by • for those constraints too small to achieve, we could reduce the number of layers by adding a skip operator to some of the layers. • for those constraints too big, we could adjust the scoring function to let the algorithm cares more about representation capacity other than latency, or even add additional layers. As illustrated in Section 3, we group the 20 layers into earlier t layers and latter 20-t layers. As the changes of resolution are natural boundary due to the feature map size changes, we split the groups at the second resolution downsampling and set t=8. In a constrained optimization problem, it is often required to check whether an architecture exceeds the latency constraint. To reduce the cost and complexity of connecting hardware frequently, we developed the latency predictor, which consists of multiple independent hardware specific predictors. Each predictor takes the sequence of operators in all the layers as the input and predict the real latency of the whole architecture on the target hardware platform. To build the latency prediction model, we uniformly sample 2,000 candidate architectures from our three hardware-aware search space, where 80% of them are used to build the latency model and the rest are used for test. We encode the architectures into a 84-dimension binary vector, where the binary value indicates the occurrence of a corresponding operator. Different regression models are selected for latency prediction on diverse hardware. Specifically, We build GaussianProcessRegressor with Matern kernel (length scale=1.5, nu=0.35) , Lasso Regression model (alpha=0.01) and Bayesian Ridge Regression Table 4: Supernet architecture. Column-"Block" denotes the block type. "TBS" means layer type needs to be searched. The "stride" column represents the stride of the first block in each repeated group. In our paper, we search the operations for total 20 layers. Table 2.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose HURRICANE to address the challenge of hardware diversity in one-shot neural architecture search

Represent the short summary for finding the corresponding scientific article (long)

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we present Neural Phrase-based Machine Translation (NPMT). Our method explicitly models the phrase structures in output sequences using Sleep-WAke Networks (SWAN), a recently proposed segmentation-based sequence modeling method. To mitigate the monotonic alignment requirement of SWAN, we introduce a new layer to perform (soft) local reordering of input sequences. Different from existing neural machine translation (NMT) approaches, NPMT does not use attention-based decoding mechanisms. Instead, it directly outputs phrases in a sequential order and can decode in linear time. Our experiments show that NPMT achieves superior performances on IWSLT 2014 German-English/English-German and IWSLT 2015 English-Vietnamese machine translation tasks compared with strong NMT baselines. We also observe that our method produces meaningful phrases in output languages. A word can be considered as a basic unit in languages. However, in many cases, we often need a phrase to express a concrete meaning. For example, consider understanding the following sentence, "machine learning is a field of computer science". It may become easier to comprehend if we segment it as " [machine learning] [is] [a field of] [computer science]", where the words in the bracket'[]' are regarded as "phrases". These phrases have their own meanings, and can often be reused in other contexts. The goal of this paper is to explore the use of phrase structures aforementioned for neural networkbased machine translation systems BID22 BID0. To this end, we develop a neural machine translation method that explicitly models phrases in target language sequences. Traditional phrase-based statistical machine translation (SMT) approaches have been shown to consistently outperform word-based ones (; ; BID15 . However, modern neural machine translation (NMT) methods BID22 BID0 do not have an explicit treatment on phrases, but they still work surprisingly well and have been deployed to industrial systems BID31 BID28. The proposed Neural Phrase-based Machine Translation (NPMT) method tries to explore the advantages from both kingdoms. It builds upon Sleep-WAke Networks (SWAN), a segmentation-based sequence modeling technique described in BID25, where segments (or phrases) are automatically discovered given the data. However, SWAN requires monotonic alignments between inputs and outputs. This is often not an appropriate assumption in many language pairs. To mitigate this issue, we introduce a new layer to perform (soft) local reordering on input sequences. Experimental show that NPMT outperforms attention-based NMT baselines in terms of the BLEU score BID19 on IWSLT 2014 German-English/English-German and IWSLT 2015 English-Vietnamese translation tasks. We believe our method is one step towards the full integration of the advantages from neural machine translation and phrase-based SMT. This paper is organized as follows. Section 2 presents the neural phrase-based machine translation model. Section 3 demonstrates the usefulness of our approach on several language pairs. We conclude our work with some discussions in Section 4. We first give an overview of the proposed NPMT architecture and some related work on incorporating phrases into NMT. We then describe the two key building blocks in NPMT: 1) SWAN, and 2) the soft reordering layer which alleviates the monotonic alignment requirement of SWAN. In the context of machine translation, we use "segment" and "phrase" interchangeably. Figure 1(a) shows the overall architecture of NPMT. The input sequence is first turned into embedding representations and then they go through a (soft) reordering layer (described below in Section 2.3). We then pass these "reordered" activations to the bi-directional RNN layers, which are finally fed into the SWAN layer to directly output target language in terms of segments (or phrases). While it is possible to replace bi-directional RNN layers with other layers BID8, in this paper, we have only explored this particular setting to demonstrate our proposed idea. There have been several works that propose different ways to incorporate phrases into attentionbased neural machine translation, such as BID23 BID26; BID6. These approaches typically use predefined phrases (obtained by external methods, e.g., phrase-based SMT) to guide or modify the existing attention-based decoder. The major difference from our approach is that, in NPMT, we do not use attention-based decoding mechanisms, and our phrase structures for the target language are automatically discovered from the training data. Another line of related work is the segment-to-segment neural transduction model (SSNT), which shows promising on a Chinese-to-English translation task under a noisy channel framework BID30. In SSNT, the segments are implicit, and the monotonic alignments between the inputs and outputs are achieved using latent variables. The latent variables are marginalized out during training using dynamic programming. BID25. Symbol $ indicates the end of a segment. Given a sequence of inputs x1,..., x5, which is from the outputs from the bi-directional RNN of FIG0 (a), SWAN emits one particular segmentation of y1:3 = π(a1:5), where DISPLAYFORM0 Here x1 wakes (emitting segment a1) and x4 wakes (emitting segment a4) while x2, x3 and x5 sleep (emitting empty segments a2, a3 and a5 respectively). Here we review the SWAN model proposed in BID25. SWAN defines a probability distribution for the output sequence given an input sequence. It models all valid output segmentations of the output sequence as well as the monotonic alignments between the output segments and the input sequence. Empty segments are allowed in the output segmentations. It does not make any assumption on the lengths of input or output sequence. Assume input sequence for SWAN is x 1:T, which is the outputs from bi-directional RNN of FIG0 (a), and output sequence is y 1:T. Let S y denote the set containing all valid segmentations of y 1:T, with the constraint that the number of segments in a segmentation is the same as the input sequence length, T. Let a t denote a segment or phrase in the target sequence. Empty segments are allowed to ensure that we can correctly align segment a t to input element x t. Otherwise, we might not have a valid alignment for the input and output pair. See FIG1 for an example of the emitted segmentation of y 1:T. The probability of the sequence y 1:T is defined as the sum of the probabilities of all the segmentations in S y {a 1:T : π(a 1:T) = y 1:T }, DISPLAYFORM0 where the p(a t |x t) is the segment probability given input element x t, which is modeled using a recurrent neural network (RNN) with an additional softmax layer. π(·) is the concatenation operator and the symbol $, end of a segment, is ignored in the concatenation operator π(·). (An empty segment, which only contains $ will thus be ignored as well.) SWAN can be also understood via a generative model, 1. For t = 1,..., T: (a) Given an initial state of x t, sample words from RNN until we reach an end of segment symbol $. This gives us a segment a t. 2. Concatenate {a 1, ..., a T} to obtain the output sequence via π(a 1:T) = y 1:T.Since there are more than one way to obtain the same y 1:T using the generative process above, the probability of observing y 1:T is obtained by summing over all possible ways, which is Eq. 1.Note that |S y | is exponentially large, direct summation quickly becomes infeasible when T or T is not small. Instead, BID25 developed an exact dynamic programming algorithm to tackle the computation challenges. 3 The key idea is that although the number of possible segmentations is exponentially large, the number of possible segments is polynomial-O(T 2). In other words, it is possible to first compute all possible segment probabilities, p(a t |x t), ∀a t, x t, and then use dynamic programming to calculate the output sequence probability p(y 1:T |x 1:T) in Eq.. The feasibility of using dynamic programming is due to a property of segmentations-a segmentation of a subsequence is also part of the segmentation of the entire sequence. In practice, a maximum length L for a segment a t is enforced to reduce the computational complexity, since the length of useful segments is often not very long. BID25 also discussed a way to carry over information across segments using a separate RNN, which we will not elaborate here. We refer the readers to the original paper for the algorithmic details. SWAN defines a conditional probability for an output sequence given an input one. It can be used in many sequence-to-sequence tasks. In practice, a sequence encoder like a bi-directional RNN can be used to process the raw input sequence (like speech signals or source language) to obtain x 1:T that is to be passed into SWAN for decoding. For example, BID25 demonstrated the usefulness of SWAN in the context of speech recognition. Greedy decoding for SWAN is straightforward. We first note that p(a t |x t) is modeled as an RNN with an additional softmax layer. Given each p(a t |x t), ∀t ∈ 1,..., T, is independent of each other, we can run the RNN in parallel to produce an output segment (possibly empty) for each p(a t |x t). We then concatenate these output segments to form the greedy decoding of the entire output sequence. The decoding satisfies the non-autoregressive property BID9 and the decoding complexity is O(T L). See BID25 for the algorithmic details of the beam search decoder. We finally note that, in SWAN (thus in NPMT), only output segments are explicit; input segments are implicitly modeled by allowing empty segments in the output. This is conceptually different from the traditional phrase-based SMT where both inputs and outputs are phrases (or segments). We leave the option of exploring explicit input segments as future work. SWAN assumes a monotonic alignment between the output segments and the input elements. For speech recognition experiments in BID25, this is a reasonable assumption. However, for machine translation, this is usually too restrictive. In neural machine translation literature, attention mechanisms were proposed to address alignment problems BID0 BID20 BID24. But it is not clear how to apply a similar attention mechanism to SWAN due to the use of segmentations for output sequences. One may note that in NPMT, a bi-directional RNN encoder for the source language can partially mitigate the alignment issue for SWAN, since it can access every source word. However, from our empirical studies, it is not enough to obtain the best performance. Here we augment SWAN with a reordering layer that does (soft) local reordering of the input sequence. This new model leads to promising on the IWSLT 2014 German-English/English-German, and IWSLT 2015 EnglishVietnamese machine translation tasks. One additional advantage of using SWAN is that since SWAN does not use attention mechanisms, decoding can be done in parallel with linear complexity, as now we remove the need to query the entire input source for every output word BID20 BID9.We now describe the details of the local reordering layer shown in FIG2. Denote the input to the local reordering layer by e 1:T, which is the output from the word embedding layer of FIG0, and the output of this layer by h 1:T, which is fed as inputs to the bi-directional RNN of FIG0. We compute h t as DISPLAYFORM0 where σ(·) is the sigmoid function, and 2τ + 1 is the local reordering window size. weight of e t−τ +i through the gate σ w T i [e t−τ ; . . . ; e t ; . . . ; e t+τ]. The final output h t is a weighted linear combination of the input elements, e t−τ,..., e t,..., e t+τ, in the window followed by a nonlinear transformation by the tanh(·) function. Figure 3(b) illustrates how local reordering works. Here we want to (softly) select an input element from a window given all information available in this window. Suppose we have two adjacent windows, (e 1, e 2, e 3) and (e 2, e 3, e 4). If e 3 gets the largest weight (e 3 is picked) in the first window and e 2 gets the largest weight (e 2 is picked) in the second window, e 2 and e 3 are effectively reordered. Our layer is different from the attention mechanism BID0 BID20 BID24 in following ways. First, we do not have a query to begin with as in standard attention mechanisms. Second, unlike standard attention, which is top-down from a decoder state to encoder states, the reordering operation is bottom-up. Third, the weights {w i} 2τ i=0 capture the relative positions of the input elements, whereas the weights are the same for different queries and encoder hidden states in the attention mechanism (no positional information). The reordering layer performs locally similar to a convolutional layer and the positional information is encoded by a different parameter w i for each relative position i in the window. Fourth, we do not normalize the weights for the input elements e t−τ,..., e t,..., e t+τ. This provides the reordering capability and can potentially turn off everything if needed. Finally, the gate of any position i in the reordering window is determined by all input elements e t−τ,..., e t,..., e t+τ in the window. We provide a visualizing example of the reordering layer gates that performs input swapping in Appendix A.One related work to our proposed reordering layer is the Gated Linear Units (GLU) which can control the information flow of the output of a traditional convolutional layer. But GLU does not have a mechanism to decide which input element from the convolutional window to choose. From our experiments, neither GLU nor traditional convolutional layer helped our NPMT. Another related work to the window size of the reordering layer is the distortion limit in traditional phrase-based statistical machine translation methods BID2. Different window sizes restrict the context of each position to different numbers of neighbors. We provide an empirical comparison of different window sizes in Appendix B. In this section, we evaluate our model on the IWSLT 2014 German-English BID3, IWSLT 2014 English-German, and IWSLT 2015 English-Vietnamese BID4 machine translation tasks. We note that, in this paper, we limit the applications of our model to relatively small datasets to demonstrate the usefulness of our method. We plan to conduct more large scale experiments in future work. MIXER BID21 20.73 21.83 LL BID27 22.53 23.87 BSO BID27 23.83 25.48 LL BID1 25 We evaluate our model on the German-English machine translation track of the IWSLT 2014 evaluation campaign BID3. The data comes from translated TED talks, and the dataset contains roughly 153K training sentences, 7K development sentences, and 7K test sentences. We use the same preprocessing and dataset splits as in Ranzato et al. We report our IWSLT 2014 German-English experiments using one reordering layer with window size 7, two layers of bi-directional GRU encoder (Gated recurrent unit, BID5) with 256 hidden units, and two layers of unidirectional GRU decoder with 512 hidden units. We add dropout with a rate of 0.5 in the GRU layer. We choose GRU since baselines for comparisons were using GRU. The maximum segment length is set to 6. Batch size is set as 32 (per GPU) and the Adam algorithm BID13 ) is used for optimization with an initial learning rate of 0.001. For decoding, we use greedy search and beam search with a beam size of 10. As reported in BID18; BID1, we find that penalizing candidate sentences that are too short was required to obtain the best . We add the middle term of Eq. to encourage longer candidate sentences. All hyperparameters are chosen based on the development set. NPMT takes about 2-3 days to run to convergence (40 epochs) on a machine with four M40 GPUs. The are summarized in Table 1. In addition to previous reported baselines in the literature, we also explored the best hyperparameter using the same model architecture (except the reordering layer) using sequence-to-sequence model with attention as reported as LL * of Table 1.NPMT achieves state-of-the-art on this dataset as far as we know. Compared to the supervised sequence-to-sequence model, LL BID1, NPMT achieves 2.4 BLEU gain in the greedy setting and 2.25 BLEU gain using beam-search. Our are also better than those from the actor-critic based methods in BID1. But we note that our proposed method is orthogonal to the actor-critic method. So it is possible to further improve our using the actor-critic method. We also run the following two experiments to verify the sources of the gain. The first is to add a reordering layer to the original sequence-to-sequence model with attention, which gives us BLEU scores of 25.55 (greedy) and 26.91 (beam search). Since the attention mechanism and reordering layer capture similar information, adding the reordering layer to the sequence-to-sequence model with attention does not improve the performance. The second is to remove the reordering layer from NPMT, which gives us BLEU scores of 27.79 (greedy) and 29.28 (beam search). This shows that the reordering layer and SWAN are both important for the effectiveness of NPMT.. target ground truth there are tens of thousands of machines around the world that make small pieces of dna -30 to 50 letters -in length -and it's a UNK process, so the longer you make the piece, the more errors there are. Table 2: Examples of German-English translation outputs with their segmentations. We label the indexes of the words in the source sentence and we use those indexes to indicate where the output segment is emitted. For example, in greedy decoding , " i word1,..., wordm" denotes i-th word in the source sentence emits words word1,..., wordm during decoding (assuming monotonic alignments). The "•" represents the segment boundary in the target output. See Figure 4 for a visualization of row 1 in this table. In greedy decoding, we can estimate the average segment length 4 for the output. The average segment length is around 1.4-1.6, indicating phrases with more than one word are being decoded. Figure 4 shows an example of the input and decoding with NPMT. We can observe phraselevel translation being captured by the model (e.g., "danke" → "thank you"). The model also knows when to sleep before outputting a phrase (e.g., "das" → "$"). We use the indexes of words in the source sentence to indicate where the output phrases are from. Table 2 shows some sampled exam-ples. We can observe there are many informative segments in the decoding , e.g., "tens of thousands of", "the best thing", "a little", etc. There are also mappings from phrase to phrase, word to phrases, and phrase to word in the examples. Following the analysis, we show the most frequent phrase mappings in Appendix C.We also explore an option of adding a language-model score during beam search as the traditional statistical machine translation does. This option might not make much sense in attention-based approaches, since the decoder itself is usually a neural network language model. In SWAN, however, there is no language models directly involved in the segmentation modeling, 5 and we find it useful to have an external language model during beam search. We use a 4th-order language model trained using the KenLM implementation BID11 for English target training data. So the final beam search score we use is DISPLAYFORM0 where we empirically find that λ 1 = 1.2 and λ 2 = 0.2 give good performance, which are tuned on the development set. The with the external language model are denoted by NPMT+LM in Table 1. If no external language models are used, we set λ 2 = 0. This scoring function is similar to the one for speech recognition in. We also evaluate our model on the opposition direction, English-German, which translates from a more segmented text to a more inflectional one. Following the setup in Section 3.1, we use the same dataset with the opposite source and target languages. We use the same model architecture, optimization algorithm and beam search size as the German-English translation task. NPMT takes about 2-3 days to run to convergence (40 epochs) on a machine with four M40 GPUs. Given there is no previous sequence-to-sequence attention model baseline for this setup, we create a strong one and tune hyperparameters on the development set. The are shown in TAB4. Based on the development set, we set λ 1 = 1 and λ 2 = 0.15 in Eq.. Our model outperforms sequence-to-sequence model with attention by 2.46 BLEU and 2.49 BLEU in greedy and beam search cases. We can also use a 4th-order language model trained using the KenLM implementation for German target training data, which further improves the performance. Some sampled examples are shown in TAB6. Several informative segments/phrases can be found in the decoding , e.g., "some time ago" → "vor enniger zeit". In this section, we evaluate our model on the IWSLT 2015 English to Vietnamese machine translation task. The data is from translated TED talks, and the dataset contains roughly 133K training sentence pairs provided by the IWSLT 2015 Evaluation Campaign BID4. Following the same preprocessing steps in; BID20 We use one reordering layer with window size 7, two layers of bi-directional LSTM (Long shortterm memory, BID12) encoder with 512 hidden units, and three layers of unidirectional LSTM decoder with 512 hidden units. We add dropout with a rate of 0.4 in the LSTM layer. We choose LSTM since baselines for comparisons were using LSTM. The maximum segment length is set to 7. Batch size is set as 48 (per GPU) and the Adam algorithm BID13 is used for optimization with an initial learning rate of 0.001. For decoding, we use greedy decoding and beam search with a beam size of 10. The are shown in TAB8. Based on the development set, we set λ 1 = 0.7 and λ 2 = 0.15 in Eq.. NPMT takes about one day to run to convergence (15 epochs) on a machine with 4 M40 GPUs. Our model outperforms sequence-tosequence model with attention by 1.41 BLEU and 1.59 BLEU in greedy and beam search cases. We also use a 4th-order language model trained using the KenLM implementation for Vietnamese target training data, which further improves the BLEU score. Note that our reordering layer relaxes the monotonic assumption as in BID20 and is able to decode in linear time. Empirically we outperform models with monotonic attention. Table 6 shows some sampled examples. Hard monotonic BID20 23.00 -Luong & Manning FORMULA1 FIG4, we show an example that translates from "can you translate it?" to "können man esübersetzen?", where the mapping between words are as follows: "can → können", "you → man", "translate →übersetzen", "it → es" and "? →?". Note that the example needs to be reordered from "translate it" to "esübersetzen". Each row of FIG4 represents a window of size 7 that is centered at a source sentence word. The values in the matrix represent the gate values for the corresponding words. The gate values will later be multiplied with the embedding e t−τ +i of Eq. and contribute to the hidden vector h t. The y-axis represents the word/phrases emitted from the corresponding position. We can observe that the gates mostly focus on the central word since the first part of the sentence only requires monotonic alignment. Interestingly, the model outputs "$" (empty) when the model has the word "translate" in the center of the window. Then, the model outputs "es" when the model encounters "it". Finally, in the last window (top row), the model not only has a large gate value to the center input "?", but the model also has a relatively large gate value to the word "translate" in order to output the translation "übersetzen?". This shows an example of the reordering effect achieved by using the gating mechanism of the reordering layer. In this section, we examine the effect of window sizes in the reordering layer. Following the setup in Section 3.2, we evaluate the performance of different window sizes on the IWSLT 2014 EnglishGerman translation task. TAB9 summarizes the . We can observe that the performance reaches the peak with a windows size of 7. With a window size of 5, the performance drops 0.88 BLEU in greedy decoding and 0.72 BLEU using beam search. It suggests that the context window is not large enough to properly perform reordering. When the window sizes are 9 and 11, we do not observe further improvements. It might be because the translation between English and German mostly requires local word reordering. Following the examples of Table 2, we analyze the decoding on the test set of the GermanEnglish translation task. Given we do not have explicit input segments in NPMT, we assume input words that emit "$" symbol are within the same group as the next non-'$' word. For example, in Figure 4, input words "das beste" are considered as an input segment. We then can aggregate all the input, output segments (phrases) and sort them based on the frequency. Tables C and C show UNK → the UNK in der → in der UNK → the UNK in diesem → in this und → and und →, and UNK. →. ein UNK → a UNK die welt → the world UNK → UNK das → this is UNK, →, das UNK → the UNK ist es → it's aber → but das, → that's, die → that eine UNK → a UNK ". →. " " → " UNK → a UNK ist. →. in UNK → in UNK ein paar → a few ist → is ich → i think in den → in den UNK → the UNK gibt es → there's der → of es → it was ist, →, wissen sie → you know der welt → the world von → of dies → this is sind. →. in diesem → in this die frage → the question mit → with es → there's, wenn → if dem UNK → the UNK haben wir → we have Table 8: German-English phrase mapping . We show the top 10 input, output phrase mappings in five categories ("One" stands for single word and "Many" stands for multiple words.). In the last column, Many → Many *, we remove the phrases with the "UNK" word as the "UNK" appears often. Phrases with 3 words Phrases with 4 words auf der ganzen → all over the auf der ganzen → a little bit of gibt eine menge → a lot of weiß nicht, was → what's going to be dann hat er→ he doesn't have tun, das wir → we can't do, die man → you can do tat, das ich → i didn't do das können wir → we can do that zu verbessern, die → that can be done Table 9: German-English longer phrase mapping . We show the top 5 input, output phrase mappings for two categories: input and output phrases with three words, and input and output phrases with four words.

Represent the TLDR for finding the corresponding scientific article

Neural phrase-based machine translation with linear decoding time

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.