Question: Represent the scientific article to find a short TLDR

Question: We present a weakly-supervised data augmentation approach to improve Named Entity Recognition (NER) in a challenging domain: extracting biomedical entities (e.g., proteins) from the scientific literature. First, we train a neural NER (NNER) model over a small seed of fully-labeled examples. Second, we use a reference set of entity names (e.g., proteins in UniProt) to identify entity mentions with high precision, but low recall, on an unlabeled corpus. Third, we use the NNER model to assign weak labels to the corpus. Finally, we retrain our NNER model iteratively over the augmented training set, including the seed, the reference-set examples, and the weakly-labeled examples, which in refined labels. We show empirically that this augmented bootstrapping process significantly improves NER performance, and discuss the factors impacting the efficacy of the approach. The increasing wealth of available data fuels numerous machine learning applications. Unfortunately, much of this data is unlabeled, unstructured and noisy. Supervised learning achieves the best task performance, but obtaining training labels is expensive. Crowd-sourcing could provide labels at scale, but may not be feasible for acquiring high-quality labels in technical domains, such as biomedicine that requires expert annotators. In this paper, we explore augmented bootstrapping methods that leverage automatically assigned noisy labels obtained from a large unlabeled corpus. The biomedical literature is a high-impact domain with scarce annotations. Unlocking the knowledge in this data requires machine reading systems that automatically extract important concepts in the text, such as entities and their relations. A critical component of such systems is reliable Named Entity Recognition (NER), which aims to identify parts of the text that refer to a named entity (e.g., a protein). In line with advancements in many domains, most state-of-the-art NER approaches use a deep neural network model that relies on a large labeled training set, which is not usually available in biomedical domains. To address label scarcity, we propose a framework to train any effective neural NER model by leveraging partially labeled data. We do this by creating an augmented training set using a small fully-labeled seed set, and an unlabeled corpus set, which we weakly and automatically label, and then refine its labels via an iterative process. Our main contributions include: An augmented bootstrapping approach combining information from a reference set with iterative refinements of soft labels to improve NER in a challenging domain (biomedicine) where labelling is expensive. A detailed analysis in a controlled setting to study different aspects affecting performance. An analysis of reference-based automated approaches to labeling data, showing that naive labeling decreases performance and how to overcome it. Many effective NER systems assume a fully-supervised setting to train a neural network model BID8 BID9 BID7. Recently, distant supervision has been applied to language-related tasks such as phrase mining BID15, relation extraction BID10, and entity extraction BID5. For NER, automatically generated candidate annotations on an unlabeled dataset using weak labellers. BID14 and BID5 used knowledge bases and linguistic features to tag entities. Our approach combines knowledge extracted from an external reference set with noisy predicted labels and refines them an iteratively. Using a reference set proposed heuristic-based functions to label data with low accuracy. BID15 b) described techniques to automatically tag phrases based on knowledge bases such as MeSH and CTD in the biomedical domain. However, in NER systems with weak supervision, wrongly-labeled entities negatively affects the overall performance BID16. We show that our proposed iterative training technique is able to make the learning process more robust to noisy labels. Our method is closely related to bootstrapping approaches. BID17 introduced the bootstrapping technique by training a tree-based classifier for word-sense disambiguation on labeled seed data and then using it to predict on an unlabeled corpus which further is used for training the model iteratively until convergence. Later BID6 bootstrapped statistical classifiers for NER. BID0 and BID4 applied bootstrapping for language processing, and BID13 for image classification. We propose an augmented bootstrapping technique for the state-of-the-art neural NER model applied to biomedical literature. In contrast to the standard bootstrapping techniques that use hard labels, we leverage and refine soft label values, which may be more suitable for noisy data. More importantly, we further augment the bootstrapping process via a simple domain-independent data annotation scheme based on a reference set, which is in contrast to the hand-crafted domain specific rules or the linguistic or morphological characteristics used in standard bootstrapping approaches. Our main goal in this study is to use easily available external information to leverage unlabeled data and reduce the need for an expensive fully-labeled dataset. We assume to have a small fullyannotated seed dataset D s that has every token tagged by entity type and a larger unlabeled corpus D c. We seek to automatically generate an augmented dataset by partially, and possibly noisily, labeling D c. We show that training a (Neural) NER system over the combined seed and augmented datasets achieves the performance of systems trained with an order of magnitude more labels. We propose an iterative solution to improve NER by labeling the corpus dataset using two complementary sources of information. First, we train a NER model using the small seed dataset D s and use it to label the unlabeled corpus D c, we call this set of labels predicted labels. Second, we use search policies over a reference set to find mentions of entity names in the unlabeled corpus D c, we call these set of labels reference-based labels. We combine the seed, the predicted and the reference labels to retrain the NER model. We use the updated model to iteratively refine the predicted labels portion of the corpus set. FIG0 and Algorithm 1 show the overall process of our method. We use soft scores (between 0 and 1) to label the corpus set, instead of the binary labels produced by the CRF layer used in stateof-the-art NER models. Our aim is to let the model iteratively reinforce the weak signals in the soft scores to improve the label quality. Recent high-performing neural NER (NNER) models BID7 BID9 use Bi-directional LSTM (BiLSTM) layers trained on character and word embeddings. The character embeddings are learned over the training data using a separate BiLSTM layer, and are concatenated with GloVe word embeddings BID11. We use an open-source Tensorflow implementation of this model BID3, which achieves state-of-the-art NER performance on the CoNLL 2003 1 dataset. To produce soft scores for each tag in our experiments, we replace the CRF layer with a softmax layer. Entities found via the reference set receive a score of 1. We show the effectiveness of our approach in a hard NER problem, extracting protein mentions from the biomedical literature, and systematically evaluate the contribution of the different techniques. We use the BioCreative VI Bio-ID dataset BID1, which contains 13,573 annotated figure captions corresponding to 3,658 figures from 570 full length articles from 22 journals, for a total of 102,717 annotations. The Bio-ID dataset is split into a training set of 38,344 sentences, a development set of 4,243 sentences, and a test set with 14,079 sentences. The tokens are tagged using the BIO scheme (Beginning, Inside and Outside of entities). The Bio-ID dataset provides us with a controlled environment where we can evaluate our methods, since it provides ground truth on the labels. The rationale of the following experiments is to simulate our desired data augmentation scenario, which is to search for sentences containing relevant bioentities (e.g., proteins) in a large corpus, such as PubMed Central. We evaluate our three main techniques, namely using a reference set of entity names (i.e., protein names from UniProt), predicting labels for unknown tokens using a NNER system trained in a small fraction of the data, and refining the label predictions by retraining the NNER system iteratively. We focus on protein/gene annotations for simplicity (51,977 mentions with 5,284 distinct entities).Our experimental evaluation appears in Table 1, which shows Precision, Recall and F 1 over the Bio-ID test set for different conditions. Experiments 1 and 2 (rows 1, 2) show of the NNER system trained over the full Bio-ID training dataset, which on the test set achieves F 1 of 82.99% (BiLSTM) and 83.34% (BiLSTM-CRF). This simulates the performance over a large amount of labeled data and is our gold standard upper limit. For the remaining rows, we train a NNER system over a small dataset (3% of the Bio-ID training dataset), which we refer as NNER-3%. We use the NNER-3% model to predict labels for unknown tokens (noisily, since its accuracy is not perfect). Then, we apply different data augmentation techniques over the remaining 97% of the Bio-ID training dataset, which simulates the accessibility of a large unlabeled corpus. Experiment 3 (row 3 in Table 1) shows the for a simple baseline where we train our NNER system over the 3% seed combined with one true protein label per sentence for the remaining 97% of the Bio-ID training dataset, which removes ∼60% of the protein labels. This experiment simulates an augmentation method with perfect precision, but a recall of only 40%. Experiment 4 adds the CRF to the architecture over the same scenario, which on a ∼9 point increase on F 1 to reach ∼58% (although precision suffers). Even in this somehow unrealistic scenario that includes many of the available labels, the overall performance is significantly diminished from the the system trained on 100% of the data (∼25 percentage points below in F 1). Experiments 5 and 6 show the effect of our iterative label refinement method. We first train NNER-3% on the seed data. Then we combine the seed, with the perfect precision (but partial) labels as in experiments 3 and 4, and with the noisy predicted labels for the remaining tokens in the (97% of the) training dataset. Surprisingly, training over only 3% of the data already achieves a good F 1 of 72.91% for the BiLSTM architecture and 76.21% for the BiLSTM+CRF architecture. When we retrain this base system iteratively, the accuracy of the predicted labels increases, which leads to an improvement of ∼3-4 percentage points in F 1 (to 77.58% for the BiLSTM and 79.75% for the BiLSTM+CRF). Thus, the iterative label refinement method reduces the distance to the 100% trained system from 25 to 4 percentage points, which is a substantial improvement. TAB1 shows the evolution of the iterative label refinement procedure. We train NNER-3% (row 0) and use it to predict labels for unknown tokens repeatedly, which yields a jump in performance in the first iteration, since the predicted labels are informative, and then a more gradual improvement as the labels are increasingly refined. Finally, the remaining experiments simulate the more realistic scenario we seek, where we search for sentences in a large corpus to be labeled automatically. In experiment 7, we simply use our reference set to directly search for exact mentions in the corpus. That is, we search in a case sensitive way for protein/gene names from UniProt in the 97% dataset that represents our large corpus. Matching tokens are labeled as true proteins for training. Since we know the true labels, we can compute the precision (=59.23%) and recall (=18.66%) of this selection technique, which is in fact quite poor. Even using our iterative training technique that produced good in the previous experiments, somewhat decreases the performance (from F 1 =72.70 for NNER-3% down to 71.33%). The low quality of the augmented data introduces too much noise to improve performance. To lower the noise, we refined our search procedure to improve the precision. For experiments 8 and 9, we filtered the names of our reference set, since after error analysis we discovered that many protein names were ambiguous. For example, the token ANOVA is a name of Q9UNW protein in UniProt, and a well-known statistical procedure. Thus, we removed all protein names that appear in an English dictionary from our search. More drastically, we also removed protein names of less than 3 characters, to avoid capturing acronyms that may not really be protein mentions. Finally, we also relaxed the matching strategy to be case insensitive and also to allow for partial matches. For example, when searching for TIGAR, we will accept "Flag-tagged-TIGAR". This selection techniques yield an improved precision (=90.20%) and recall (=39.35%) on identifying correct proteins in Bio-ID. We then reconstruct our augmented training dataset that combines the seed, reference-set, and the predicted labels by NNER-3% and iterative refinement. Our method achieves a F 1 of 76.63% for BiLSTM and of 77.70% for BiLSTM+CRF. In summary, through these experiments we show that using a small labeled dataset and our automatic data augmentation procedure, we achieve a performance approaching that of a system trained with over 30 times more labeled data. We proposed a method to improve NER with limited labeled data, which is often the case in technical domains, such as biomedicine. Our method combines bootstrapping and weakly-labeled data augmentation by using a small fully-labeled seed dataset and a large unlabeled corpus, automated labelling using a reference set, and an iterative label refinement process. Our experimental evaluation shows performance equivalent to systems trained with an order of magnitude more labeled data. In future work, we aim to explore additional augmentation methods over other challenging datasets. We plan to apply the findings of these controlled experiments to a much larger in-the-wild scenario where we use all the available labeled data as the seed and operate over a large corpus (e.g., all of PubMed, PubMed Central) to improve state-of-the-art NER performance.

Represent the short summary for finding the corresponding scientific article (long)

Augmented bootstrapping approach combining information from a reference set with iterative refinements of soft labels to improve Name Entity Recognition from biomedical literature.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Quantum machine learning methods have the potential to facilitate learning using extremely large datasets. While the availability of data for training machine learning models is steadily increasing, oftentimes it is much easier to collect feature vectors that to obtain the corresponding labels. One of the approaches for addressing this issue is to use semi-supervised learning, which leverages not only the labeled samples, but also unlabeled feature vectors. Here, we present a quantum machine learning algorithm for training Semi-Supervised Kernel Support Vector Machines. The algorithm uses recent advances in quantum sample-based Hamiltonian simulation to extend the existing Quantum LS-SVM algorithm to handle the semi-supervised term in the loss, while maintaining the same quantum speedup as the Quantum LS-SVM. Data sets used for training machine learning models are becoming increasingly large, leading to continued interest in fast methods for solving large-scale classification problems. One of the approaches being explored is training the predictive model using a quantum algorithm that has access to the training set stored in quantum-accessible memory. In parallel to research on efficient architectures for quantum memory , work on quantum machine learning algorithms and on quantum learning theory is under way (see for example Refs. (; ;) and (Arunachalam & de) for review). An early example of this approach is Quantum LS-SVM (a), which achieves exponential speedup compared to classical LS-SVM algorithm. Quantum LS-SVM uses quadratic least-squares loss and squared-L 2 regularizer, and the optimization problem can be solved using the seminal HHL algorithm for solving quantum linear systems of equations. While progress has been made in quantum algorithms for supervised learning, it has been recently advocated that the focus should shift to unsupervised and semi-supervised setting . In many domains, the most laborious part of assembling a training set is the collection of sample labels. Thus, in many scenarios, in addition to the labeled training set of size m we have access to many more feature vectors with missing labels. One way of utilizing these additional data points to improve the classification model is through semi-supervised learning. In semi-supervised learning, we are given m observations x 1,..., x m drawn from the marginal distribution p(x), where the l (l m) first data points come with labels y 1,..., y l drawn from conditional distribution p(y|x). Semi-supervised learning algorithms exploit the underlying distribution of the data to improve classification accuracy on unseen samples. In the approach considered here, the training samples are connected by a graph that captures their similarity. Here, we introduce a quantum algorithm for semi-supervised training of a kernel support vector machine classification model. We start with the existing Quantum LS-SVM (a), and use techniques from sample-based Hamiltonian simulation to add a semisupervised term based on Laplacian SVM . As is standard in quantum machine learning , the algorithm accesses training points and the adjacency matrix of the graph connecting samples via a quantum oracle. We show that, with respect to the oracle, the proposed algorithm achieves the same quantum speedup as LS-SVM, that is, adding the semisupervised term does not lead to increased computational complexity. Consider a problem where we are aiming to find predictors h(x): X → R that are functions from a RKHS defined by a kernel K. In Semi-Supervised LS-SVMs in RKHS, we are looking for a function h ∈ H that minimizes min h∈H,b∈R where y = (y 1, ..., y m) T, 1 = (1, ..., 1) T, 1 is identity matrix, K is kernel matrix, L is the graph Laplacian matrix, γ is a hyperparameter and α = (α 1, ..., α m) T is the vector of Lagrangian multipliers. Quantum computers are devices which perform computing according to the laws of quantum mechanics, a mathematical framework for describing physical theories, in language of linear algebra. Quantum Systems. Any isolated, closed quantum physical system can be fully described by a unit-norm vector in a complex Hilbert space appropriate for that system; in quantum computing, the space is always finite-dimensional, C d. In quantum mechanics and quantum computing, Dirac notation for linear algebra is commonly used. In Dirac notation, a vector x ∈ C d and its complex conjugate x T, which represents a functional C d → R, are denoted by |x (called ket) and x| (called bra), respectively. We call {|e i} T, |1 = T and α, β ∈ C, |α| 2 + |β| 2, is called a quantum bit, or qubit for short. When both α and β are nonzero, we say |ψ is in a superposition of the computational basis |0 and |1; the two superposition states A composite quantum state of two distinct quantum systems |x 1 ∈ C d1 and |x 2 ∈ C d2 is described as tensor product of quantum states |x 1 ⊗ |x 2 ∈ C d1 ⊗ C d2. Thus, a state of an n-qubit system is a vector in the tensor product space C 2 ⊗n = C 2 ⊗ C 2 ⊗... ⊗ C 2, and is written as i=0 α i |i, where i is expressed using its binary representation; for example for n = 4, we have |2 = |0010 = |0 ⊗ |0 ⊗ |1 ⊗ |0. Transforming and Measuring Quantum States. Quantum operations manipulate quantum states in order to obtain some desired final state. Two types of manipulation of a quantum system are allowed by laws of physics: unitary operators and measurements. Quantum measurement, if done in the computational basis, stochastically transforms the state of the system into one of the computational basis states, based on squared magnitudes of probability amplitudes; that is, will in |0 and |1 with equal chance. Unitary operators are deterministic, invertible, normpreserving linear transforms. A unitary operator U models a transformation of a quantum state |u to |v = U|u. Note that U|u 1 + U|u 2 = U (|u 1 + |u 2), applying a unitary to a superposition of states has the same effect as applying it separately to element of the superposition. In quantum circuit model unitary transformations are referred to as quantum gates -for example, one of the most common gates, the single-qubit Hadamard gate, is a unitary operator represented in the computational basis by the matrix Note that H|0 = |+ and H|1 = |−. Quantum Input Model. Quantum computation typically starts from all qubits in |0 state. To perform computation, access to input data is needed. In quantum computing, input is typically given by a unitary operator that transforms the initial state into the desired input state for the computation -such unitaries are commonly referred to as oracles, and the computational complexity of quantum algorithms is typically measured with access to an oracle as the unit. For problems involving large amounts of input data, such as for quantum machine learning algorithms, an oracle that abstracts random access memory is often assumed. Quantum random access memory (qRAM) uses log N qubits to address any quantum superposition of N memory cell which may contains either quantum or classical information. For example, qRAM allows accessing classical data entries x j i in quantum superposition by a transformation where |x j i is a binary representation up to a given precision. Several approaches for creating quantum RAM are being considered (; ;), but it is still an open challenge, and subtle differences in qRAM architecture may erase any gains in computational complexity of a quantum algorithm. Quantum Linear Systems of Equations. Given an input matrix A ∈ C n×n and a vector b ∈ C n, the goal of linear system of equations problem is finding x ∈ C n such that Ax = b. When A is Hermitian and full rank, the unique solution is x = A −1 b. If A is not a full rank matrix then A −1 is replaced by the Moore-Penrose pseudo-inverse. HHL algorithm introduced an analogous problem in quantum setting: assuming an efficient algorithm for preparing b as a quantum state b = n i=1 b i |i using log n + 1 qubits, the algorithm applies quantum subroutines of phase estimation, controlled rotation, and inverse of phase estimation to obtain the state Intuitively and at the risk of over-simplifying, HHL algorithm works as follows: if A has spec- In general A and A −1 are not unitary (unless all A's eigenvalues have unit magnitude), therefore we are not able to apply A −1 directly on |b. However, since is unitary and has the same eigenvectors as A and A −1, one can implement U and powers of U on a quantum computer by Hamiltonian simulation techniques; clearly for any expected speed-up, one need to enact e iA efficiently. The HHL algorithm uses the phase estimation subroutine to estimate an approximation of λ i up to a small error. The Next step computes a conditional rotation on the approximated value of λ i and an auxiliary qubit |0 and outputs |1. The last step involves the inverse of phase estimation and quantum measurement for getting rid of garbage qubits and outputs our desired state Density Operators. Density operator formalism is an alternative formulation for quantum mechanics that allows probabilistic mixtures of pure states, more generally referred to as mixed states. A mixed state that describes an ensemble {p i, |ψ i} is written as where k i=1 p i = 1 forms a probability distribution and ρ is called density operator, which in a finite-dimensional system, in computational basis, is a semi-definite positive matrix with T r(ρ) = 1. A unitary operator U maps a quantum state expressed as a density operator ρ to UρU †, where U † is the complex conjugate of the operator U. Partial Trace of Composite Quantum System. Consider a two-part quantum system in a state described by tensor product of two density operators ρ ⊗ σ. A partial trace, tracing out the second part of the quantum system, is defined as the linear operator that leaves the first part of the system in a state Tr 2 (ρ ⊗ σ) = ρ tr (σ), where Tr (σ) is the trace of the matrix σ. To obtain Kernel matrix K as a density matrix, quantum LS-SVM (b) relies on partial trace, and on a quantum oracle that can convert, in superposition, each data point where (x i) t refers to the tth feature value in data point x i and assuming the oracle is given x i and y i. Vector of the labels is given in the same fashion as |y = 1 y m i=1 y i |i. For preparation the normalized Kernel matrix K = 1 tr(K) K where K = X T X, we need to prepare a quantum state combining all data points in quantum The normalized Kernel matrix is obtained by discarding the training set state, The approach used above to construct density matrix corresponding to linear kernel matrix can be extended to polynomial kernels (b). LMR Technique for Density Operator Exponentiation. In HHL-based quantum machine learning algorithms, including in the method proposed here, matrix A for the Hamiltonian simulation within the HHL algorithm is based on data. For example, A can contain the kernel matrix K captured in the quantum system as a density matrix. Then, one need to be able to efficiently compute e −iK∆t, where K is scaled by the trace of kernel matrix. Since K is not sparse, a strategy similar to is adapted for the exponentiation of a non-sparse density matrix: where S = i,j |i j| ⊗ |j i| is the swap operator and the facts Tr 1 {S(K ⊗ σ)} = Kσ and Tr 1 {(K ⊗ σ)S} = σK are used. The equation summarizes the LMR technique: approximating e −iK∆t σe iK∆t up to error O(∆t 2) is equivalent to simulating a swap operator S, applying it to the state K ⊗ σ and discarding the first system by taking partial trace operation. Since the swap operator is sparse, its simulation is efficient. Therefore the LMR trick provides an efficient way to approximate exponentiation of a non-sparse density matrix. Quantum LS-SVM. Quantum LS-SVM (b) uses partial trace to construct density operator corresponding to the kernel matrix K. Once the kernel matrix K becomes available as a density operator, the quantum LS-SVM proceeds by applying the HHL algorithm for solving the system of linear equations associated with LS-LSVM, using the LMR technique for performing the density operator exponentiation e −iK∆t where the density matrix K encodes the kernel matrix. 3 QUANTUM SEMI-SUPERVISED LEAST SQUARE SVM. Semi-Supervised Least Square SVM involves solving the following system of linear equations In quantum setting the task is to generate |b, α =Â −1 |0, y, where the normalizedÂ = A T r(A). The linear system differs from the one in LS-SVM in that instead of K, we have K + KLK. While this difference is of little significance for classical solvers, in quantum systems we cannot just multiply and then add the matrices and then apply quantum LS-SVM -we are limited by the unitary nature of quantum transformations. In order to obtain the solution to the quantum Semi-Supervised Least Square SVM, we will use the following steps. First, we will read in the graph information to obtain scaled graph Laplacian matrix as a density operator. Next, we will use polynomial Hermitian exponentiation for computing the matrix inverse (K + KLK) −1. In the semi-supervised model used here, we assume that we have information on the similarity of the training samples, in a form of graph G that uses n edges to connect similar training samples, represented as m vertices. We assume that for each sample, G contains its k most similar other samples, that is, the degree of each vertex is d. To have the graph available as a quantum density operator, we observe that the graph Laplacian L is the Gram matrix of the rows of the m × n graph incidence matrix G I, L = G I G T I. We assume oracle access to the graph adjacency list, allowing us to construct, in superposition, states corresponding to rows of the graph incidence matrix G I That is, state |v i has probability amplitude for each edge |t incident with vertex i, and null probability amplitude for all other edges. In superposition, we prepare a quantum state combining rows of the incidence matrix for all vertices, to obtain The graph Laplacian matrix L, composed of inner products of the rows of G I, is obtained by discarding the second part of the system, For computing the matrix inverse (K + KLK) −1 on a quantum computer that runs our quantum machine algorithm and HHL algorithm as a subroutine, we need to efficiently compute e −i(K+KLK)∆t σe i(K+KLK)∆t. For this purpose we adapt the generalized LMR technique for simulating Hermitian polynomials proposed in to the specific case of e −i(K+KLK)∆t σe i(K+KLK)∆t. Simulation of e −iK∆t follows from the original LMR algorithm, and therefore we focus here only on simulation e −iKLK∆t. The final dynamics (K + KLK) −1 can be obtained by sampling from the two separate output states for e −iKLK∆t and e −iK∆t. Simulating e iKLK∆t e iKLK∆t e iKLK∆t. Let D(H) denote the space of density operators associated with state space H. Let K †, K, L ∈ D(H) be the density operators associated with the kernel matrix and the Laplacian, respectively. We will need two separate systems with the kernel matrix K, to distinguish between them we will denote the first as K † and the second as K; since K is real and symmetric, these are indeed equal. The kernel and Laplacian matrices K †, K, L are not sparse therefore we adapt the generalized LMR technique for simulating Hermitian polynomials for our specific case For adapting the generalized LMR technique to our problem we need to generate a quantum state ρ = |0 0| ⊗ ρ + |1 1| ⊗ ρ with T r(ρ + ρ) = 1, such that where is a controlled partial swap in the forward (+S) and backward direction (−S) in time, and Therefore with one copy of ρ, we obtain the simulation of e −iB∆ up to error O(∆ 2). If we choose the time slice ∆ = δ/t and repeating the above procedure for t 2 /δ times, we are able to simulate e −iBt up to error O(δ) using n = O(t 2 /δ) copies of ρ. Figure 1 shows the quantum circuit for creating ρ = |0 0| ⊗ ρ + |1 1| ⊗ ρ such that T r(ρ + ρ) = 1 and B = ρ − ρ = KLK. The analysis of the steps preformed by the circuit depicted in Fig.1 is as follows. Let P be the cyclic permutation of three copies of H A that operates as P |j 1, j 2, j 3 = |j 3, j 1, j 2. In operator form it can be written as The quantum LS-SVM in (b) offers exponential speedup O(log mp) over the classical time complexity for solving SVM as a quadratic problem, which requires time O(log( −1)poly(p, m)), where is the desired error. The exponential speedup in p occurs as the of fast quantum computing of kernel matrix, and relies on the existence of efficient oracle access to data. The speedup on m is due to applying quantum matrix inversion for solving LS-SVM, which is inherently due to fast algorithm for exponentiation of a ing non-sparse matrix. Our algorithm introduces two additional steps: preparing the Laplacian density matrix, and Hamiltonian simulation for KLK. The first step involves oracle access to a sparse graph adjacency list representation, which is at least as efficient as the oracle access to non-sparse data points. The Hamiltonian simulation involves simulating a sparse conditional partial swap operator, which an efficient strategy for applying e −iKLK∆t in timeÕ(log(m)∆t), where the notationÕ hides more slowly growing factors in the simulation . Considerable effort has been devoted into designing fast classical algorithms for training SVMs. The decomposition-based methods such as SMO are able to efficiently manage problems with large number of features p, but their computational complexities are super-linear in m. Other training strategies (; ;) are linear in m but scale quadratically in p in the worst case. The Pegasos algorithm for non-linear kernel improves the complexity toÕ (m/(λ)), where λ, and are the regularization parameter of SVM and the error of the solution, respectively. Beyond the classical realm, three quantum algorithms for training linear models have been proposed, the quantum LS-SVM that involves L 2 regularizer (a), a recently proposed Quantum Sparse SVM which is limited to a linear kernel , and a quantum training algorithm that solves a maximin problem ing from a maximum -not average -loss over the training set .

represent this text to find a much much longer article about it with say hundreds-thousands of words.

We extend quantum SVMs to semi-supervised setting, to deal with the likely problem of many missing class labels in huge datasets.

Represent the TLDR for finding the corresponding scientific article

We prove that for a large class of functions f there exists an interval certified robust network approximating f up to arbitrary precision.


Question: Represent the scientific article to find a short TLDR.

Question: Deep neural networks have become the state-of-the-art models in numerous machine learning tasks. However, general guidance to network architecture design is still missing. In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress (>50%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10. Deep learning has achieved great success in may machine learning tasks. The end-to-end deep architectures have the ability to effectively extract features relevant to the given labels and achieve state-of-the-art accuracy in various applications BID3 ). Network design is one of the central task in deep learning. Its main objective is to grant the networks with strong generalization power using as few parameters as possible. The first ultra deep convolutional network is the ResNet BID16 which has skip connections to keep feature maps in different layers in the same scale and to avoid gradient vanishing. Structures other than the skip connections of the ResNet were also introduced to avoid gradient vanishing, such as the dense connections BID20, fractal path BID27 and Dirac initialization BID50. Furthermore, there has been a lot of attempts to improve the accuracy of image classifications by modifying the residual blocks of the ResNet. BID49 suggested that we need to double the number of layers of ResNet to achieve a fraction of a percent improvement of accuracy. They proposed a widened architecture that can efficiently improve the accuracy. BID51 pointed out that simply modifying depth or width of ResNet might not be the best way of architecture design. Exploring structural diversity, which is an alternative dimension in network design, may lead to more effective networks. In BID43, BID51, BID47, and BID19, the authors further improved the accuracy of the networks by carefully designing residual blocks via increasing the width of each block, changing the topology of the network and following certain empirical observations. In the literature, the network design is mainly empirical. It remains a mystery whether there is a general principle to guide the design of effective and compact deep networks. Observe that each residual block of ResNet can be written as u n+1 = u n + ∆tf (u n) which is one step of forward Euler discretization (AppendixA.1) of the ordinary differential equation (ODE) u t = f (u) (E, 2017). This suggests that there might be a connection between discrete dynamic systems and deep networks with skip connections. In this work, we will show that many state-of-the-art deep network architectures, such as PolyNet BID51, FractalNet BID27 and RevNet BID12, can be consider as different discretizations of ODEs. From the perspective of this work, the success of these networks is mainly due to their ability to efficiently approximate dynamic systems. On a side note, differential equations is one of the most powerful tools used in low-level computer vision such as image denoising, deblurring, registration and segmentation BID36 BID2 BID4. This may also bring insights on the success of deep neural networks in low-level computer vision. Furthermore, the connection between architectures of deep neural networks and numerical approximations of ODEs enables us to design new and more effective deep architectures by selecting certain discrete approximations of ODEs. As an example, we design a new network structure called linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method in numerical ODEs BID1. This architecture can be applied to any ResNet-like networks. In this paper, we apply the LM-architecture to ResNet and ResNeXt BID47 ) and achieve noticeable improvements on CIFAR and ImageNet with comparable numbers of trainable parameters. We also explain the performance gain using the concept of modified equations from numerical analysis. It is known in the literature that introducing randomness by injecting noise to the forward process can improve generalization of deep residual networks. This includes stochastic drop out of residual blocks BID21 and stochastic shakes of the outputs from different branches of each residual block BID11. In this work we show that any ResNet-like network with noise injection can be interpreted as a discretization of a stochastic dynamic system. This gives a relatively unified explanation to the stochastic learning process using stochastic control. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the proposed LM-architecture. As an example, we introduce stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10. The link between ResNet FIG0 ) and ODEs were first observed by E, where the authors formulated the ODE u t = f (u) as the continuum limit of the ResNet u n+1 = u n + ∆tf (u n). BID31 bridged ResNet with recurrent neural network (RNN), where the latter is known as an approximation of dynamic systems. BID40 and BID30 also regarded ResNet as dynamic systems that are the characteristic lines of a transport equation on the distribution of the data set. Similar observations were also made by BID5; they designed a reversible architecture to grant stability to the dynamic system. On the other hand, many deep network designs were inspired by optimization algorithms, such as the network LISTA BID14 and the ADMM-Net BID48. Optimization algorithms can be regarded as discretizations of various types of ODEs BID18, among which the simplest example is gradient flow. Another set of important examples of dynamic systems is partial differential equations (PDEs), which have been widely used in low-level computer vision tasks such as image restoration. There were some recent attempts on combining deep learning with PDEs for various computer vision tasks, i.e. to balance handcraft modeling and data-driven modeling. BID32 and BID33 proposed to use linear combinations of a series of handcrafted PDE-terms and used optimal control methods to learn the coefficients. Later, BID10 extended their model to handle classification tasks and proposed an learned PDE model (L-PDE). However, for classification tasks, the dynamics (i.e. the trajectories generated by passing data to the network) should be interpreted as the characteristic lines of a PDE on the distribution of the data set. This means that using spatial differential operators in the network is not essential for classification tasks. Furthermore, the discretizations of differential operators in the L-PDE are not trainable, which significantly reduces the network's expressive power and stability. BID28 proposed a feed-forward network in order to learn the optimal nonlinear anisotropic diffusion for image denoising. Unlike the previous work, their network used trainable convolution kernels instead of fixed discretizations of differential operators, and used radio basis functions to approximate the nonlinear diffusivity function. More recently, BID34 designed a network, called PDE-Net, to learn more general evolution PDEs from sequential data. The learned PDE-Net can accurately predict the dynamical behavior of data and has the potential to reveal the underlying PDE model that drives the observed data. In our work, we focus on a different perspective. First of all, we do not require the ODE u t = f (u) associate to any optimization problem, nor do we assume any differential structures in f (u). The optimal f (u) is learned for a given supervised learning task. Secondly, we draw a relatively comprehensive connection between the architectures of popular deep networks and discretization schemes of ODEs. More importantly, we demonstrate that the connection between deep networks and numerical ODEs enables us to design new and more effective deep networks. As an example, we introduce the LM-architecture to ResNet and ResNeXt which improves the accuracy of the original networks. We also note that, our viewpoint enables us to easily explain why ResNet can achieve good accuracy by dropping out some residual blocks after training, whereas dropping off sub-sampling layers often leads to an accuracy drop BID44. This is simply because each residual block is one step of the discretized ODE, and hence, dropping out some residual blocks only amounts to modifying the step size of the discrete dynamic system, while the sub-sampling layer is not a part of the ODE model. Our explanation is similar to the unrolled iterative estimation proposed by BID13, while the difference is that we believe it is the data-driven ODE that does the unrolled iterative estimation. In this section we show that many existing deep neural networks can be consider as different numerical schemes approximating ODEs of the form u t = f (u). Based on such observation, we introduce a new structure, called the linear multi-step architecture (LM-architecture) which is inspired by the well-known linear multi-step method in numerical ODEs. The LM-architecture can be applied to any ResNet-like networks. As an example, we apply it to ResNet and ResNeXt and demonstrate the performance gain of such modification on CIFAR and ImageNet data sets. PolyNet FIG0 ), proposed by BID51, introduced a PolyInception module in each residual block to enhance the expressive power of the network. The PolyInception model includes polynomial compositions that can be described as DISPLAYFORM0 We observe that PolyInception model can be interpreted as an approximation to one step of the backward Euler (implicit) scheme (AppendixA.1): DISPLAYFORM1 Indeed, we can formally rewrite (I − ∆tf) −1 as DISPLAYFORM2 Therefore, the architecture of PolyNet can be viewed as an approximation to the backward Euler scheme solving the ODE u t = f (u). Note that, the implicit scheme allows a larger step size BID1, which in turn allows fewer numbers of residual blocks of the network. This explains why PolyNet is able to reduce depth by increasing width of each residual block to achieve state-of-the-art classification accuracy. FractalNet BID27 FIG0 ) is designed based on self-similarity. It is designed by repeatedly applying a simple expansion rule to generate deep networks whose structural layouts are truncated fractals. We observe that, the macro-structure of FractalNet can be interpreted as the well-known Runge-Kutta scheme in numerical analysis. Recall that the recursive fractal structure of FractalNet can be written as DISPLAYFORM3 For simplicity of presentation, we only demonstrate the FractalNet of order 2 (i.e. c ≤ 2). Then, every block of the FractalNet (of order 2) can be expressed as DISPLAYFORM4, which resembles the Runge-Kutta scheme of order 2 solving the ODE u t = f (u) (see AppendixA.2).RevNet FIG0 ), proposed by BID12, is a reversible network which does not require to store activations during forward propagations. The RevNet can be expressed as the following discrete dynamic system DISPLAYFORM5 RevNet can be interpreted as a simple forward Euler approximation to the following dynamic systeṁ DISPLAYFORM6 Note that reversibility, which means we can simulate the dynamic from the end time to the initial time, is also an important notation in dynamic systems. There were also attempts to design reversible scheme in dynamic system such as BID35. DISPLAYFORM7 We have shown that architectures of some successful deep neural networks can be interpreted as different discrete approximations of dynamic systems. In this section, we proposed a new struc-ture, called linear multi-step structure (LM-architecture), based on the well-known linear multi-step scheme in numerical ODEs (which is briefly recalled in Appendix A.3). The LM-architecture can be written as follows DISPLAYFORM0 where k n ∈ R is a trainable parameter for each layer n. A schematic of the LM-architecture is presented in Figure 2. Note that the midpoint and leapfrog network structures in BID5 are all special case of ours. The LM-architecture is a 2-step method approximating the ODE u t = f (u). Therefore, it can be applied to any ResNet-like networks, including those mentioned in the previous section. As an example, we apply the LM-architecture to ResNet and ResNeXt. We call these new networks the LM-ResNet and LM-ResNeXt. We trained LM-ResNet and LM-ResNeXt on CIFAR BID25 ) and Imagenet BID37, and both achieve improvements over the original ResNet and ResNeXt. Implementation Details. For the data sets CIFAR10 and CIFAR100, we train and test our networks on the training and testing set as originally given by the data set. For ImageNet, our models are trained on the training set with 1.28 million images and evaluated on the validation set with 50k images. On CIFAR, we follow the simple data augmentation in BID28 for training: 4 pixels are padded on each side, and a 32×32 crop is randomly sampled from the padded image or its horizontal flip. For testing, we only evaluate the single view of the original 32×32 image. Note that the data augmentation used by ResNet BID16 BID47 is the same as BID28. On ImageNet, we follow the practice in BID26; BID39. Images are resized with its shorter side randomly sampled in for scale augmentation BID39. The input image is 224×224 randomly cropped from a resized image using the scale and aspect ratio augmentation of BID42. For the experiments of ResNet/LM-ResNet on CIFAR, we adopt the original design of the residual block in BID17, i.e. using a small two-layer neural network as the residual block with bn-relu-conv-bn-reluconv. The residual block of LM-ResNeXt (as well as LM-ResNet164) is the bottleneck structure used by BID47 ) that takes the form 1 × 1, 64 3 × 3, 64 1 × 1, 256. We start our networks with a single 3 × 3 conv layer, followed by 3 residual blocks, global average pooling and a fully-connected classifier. The parameters k n of the LM-architecture are initialized by random sampling from U[−0.1, 0]. We initialize other parameters following the method introduced by BID15. On CIFAR, we use SGD with a mini-batch size of 128, and 256 on ImageNet. During training, we apply a weight decay of 0.0001 for LM-ResNet and 0.0005 for LM-ResNeXt, and momentum of 0.9 on CIFAR. We apply a weight decay of 0.0001 and momentum of 0.9 for both LM-ResNet and LM-ResNeXt on ImageNet. For LM-ResNet on CIFAR10 (CIFAR100), we start with the learning rate of 0.1, divide it by 10 at 80 and 120 epochs and terminate training at 160 epochs. For LM-ResNeXt on CIFAR, we start with the learning rate of 0.1 and divide it by 10 at 150 and 225 epochs, and terminate training at 300 epochs. Figure 2: LM-architecture is an efficient structure that enables ResNet to achieve same level of accuracy with only half of the parameters on CIFAR10. Results. Testing errors of our proposed LM-ResNet/LM-ResNeXt and some other deep networks on CIFAR are presented in TAB1. Figure 2 shows the overall improvements of LM-ResNet over ResNet on CIFAR10 with varied number of layers. We also observe noticeable improvements of both LM-ResNet and LM-ResNeXt on CIFAR100. BID47 claimed that ResNeXt can achieve lower testing error without pre-activation (pre-act). However, our show that LMResNeXt with pre-act achieves lower testing errors even than the original ResNeXt without pre-act. Training and testing curves of LM-ResNeXt are plotted in Figure3. In TAB1, we also present testing errors of FractalNet and DenseNet BID20 on CIFAR 100. We can see that our proposed LM-ResNeXt29 has the best . Comparisons between LM-ResNet and ResNet on ImageNet are presented in TAB2. The LM-ResNet shows improvement over ResNet with comparable number of trainable parameters. Note that the of ResNet on ImageNet are obtained from "https://github.com/KaimingHe/deep-residual-networks". It is worth noticing that the testing error of the 56-layer LM-ResNet is comparable to that of the 110-layer ResNet on CIFAR10; the testing error of the 164-layer LM-ResNet is comparable to that of the 1001-layer ResNet on CI-FAR100; the testing error of the 50-layer LM-ResNet is comparable to that of the 101-layer ResNet on ImageNet. We have similar on LM-ResNeXt and ResNeXt as well. These indicate that the LM-architecture can greatly compress ResNet/ResNeXt without losing much of the performance. We will justify this mathematically at the end of this section using the concept of modified equations from numerical analysis. Explanation on the performance boost via modified equations. Given a numerical scheme approximating a differential equation, its associated modified equation BID45 ) is another differential equation to which the numerical scheme approximates with higher order of accuracy than the original equation. Modified equations are used to describe numerical behaviors of numerical schemes. For example, consider the simple 1-dimensional transport equation u t = cu x. 101, pre-act 22.6 6.4Figure 3: Training and testing curves of ResNext29 (16x64d, pre-act) and and LM-ResNet29 (16x64d, pre-act) on CIFAR100, which shows that the LM-ResNeXt can achieve higher accuracy than ResNeXt. Both the Lax-Friedrichs scheme and Lax-Wendroff scheme approximates the transport equation. However, the associated modified equations of Lax-Friedrichs and Lax-Wendroff are DISPLAYFORM1 respectively, where r = 2∆t ∆x. This shows that the Lax-Friedrichs scheme behaves diffusive, while the Lax-Wendroff scheme behaves dispersive. Consider the forward Euler scheme which is associated to ResNet, DISPLAYFORM2 Thus, the modified equation of forward Euler scheme reads aṡ DISPLAYFORM3 Consider the numerical scheme used in the LM-structure DISPLAYFORM4 Then, the modified equation of the numerical scheme associated to the LM-structure DISPLAYFORM5 Comparing FORMULA11 with, we can see that when k n ≤ 0, the second order termü of is bigger than that of. The termü represents acceleration which leads to acceleration of the convergence of u n when f = −∇g BID41; BID46. When f (u) = L(u) with L being an elliptic operator, the termü introduce dispersion on top of the dissipation, which speeds up the flow of u n. In fact, this is our original motivation of introducing the LM-architecture. Note that when the dynamic is truly a gradient flow, i.e. f = −∇g, the difference equation of the LM-structure has a stability condition −1 ≤ k n ≤ 1. In our experiments, we do observe that most of the coefficients are lying in (−1, 1) FIG1. Moreover, the network is indeed accelerating at the end of the dynamic, for the learned parameters {k n} are negative and close to −1 FIG1 ). Although the original ResNet BID16 did not use dropout, several work BID21 BID11 showed that it is also beneficial to inject noise during training. In this section we show that we can regard such stochastic learning strategy as an approximation to a stochastic dynamic system. We hope the stochastic dynamic system perspective can shed lights on the discovery of a guiding principle on stochastic learning strategies. To demonstrate the advantage of bridging stochastic dynamic system with stochastic learning strategy, we introduce stochastic depth during training of LM-ResNet. Our indicate that the networks with proposed LM-architecture can also greatly benefit from stochastic learning strategies. As an example, we show that the two stochastic learning methods introduced in BID21 and BID11 can be considered as weak approximations of stochastic dynamic systems. Shake-Shake Regularization. introduced a stochastic affine combination of multiple branches in a residual block, which can be expressed as DISPLAYFORM0 where η ∼ U. To find its corresponding stochastic dynamic system, we incorporate the time step size ∆t and consider DISPLAYFORM1 which reduces to the shake-shake regularization when ∆t = 1. The above equation can be rewritten as DISPLAYFORM2 Since the random variable DISPLAYFORM3 2 ), following the discussion in Appendix B, the network of the shake-shake regularization is a weak approximation of the stochastic dynamic system DISPLAYFORM4 where dB t is an N dimensional Brownian motion, 1 N ×1 is an N -dimensional vector whose elements are all 1s, N is the dimension of X and f i (X), and denotes the pointwise product of vectors. Note from that we have alternatives to the original shake-shake regularization if we choose ∆t = 1.Stochastic Depth. BID21 randomly drops out residual blocks during training in order to reduce training time and improve robustness of the learned network. We can write the forward propagation as DISPLAYFORM5 where P(η n = 1) = p n, P(η n = 0) = 1 − p n. By incorporating ∆t, we consider DISPLAYFORM6 which reduces to the original stochastic drop out training when ∆t = 1. The variance of DISPLAYFORM7 is 1. If we further assume that (1 − 2p n) = O(√ ∆t), the condition of Appendix B.2 is satisfied for small ∆t. Then, following the discussion in Appendix B, the network with stochastic drop out can be seen as a weak approximation to the stochastic dynamic system DISPLAYFORM8 Note that the assumption (1 − 2p n) = O(√ ∆t) also suggests that we should set p n closer to 1/2 for deeper blocks of the network, which coincides with the observation made by BID21 Figure 8 ).In general, we can interpret stochastic training procedures as approximations of the following stochastic control problem with running cost DISPLAYFORM9 where L(·) is the loss function, T is the terminal time of the stochastic process, and R is a regularization term. BID17 110,pre-act Orignial 6.37ResNet BID21 In this section, we extend the stochastic depth training strategy to networks with the proposed LMarchitecture. In order to apply the theory of Itô process, we consider the 2nd orderẌ + g(t)Ẋ = f (X) (which is related to the modified equation of the LM-structure) and rewrite it as a 1st order ODE systemẊ = Y,Ẏ = f (X) − g(t)Y. Following a similar argument as in the previous section, we obtain the following stochastic procesṡ DISPLAYFORM10 which can be weakly approximated by DISPLAYFORM11 where P(η n = 1) = p n, P(η n = 0) = 1 − p n. Taking ∆t = 1, we obtain the following stochastic training strategy for LM-architecture DISPLAYFORM12 The above derivation suggests that the stochastic learning for networks using LM-architecture can be implemented simply by randomly dropping out the residual block with probability p. Implementation Details. We test LM-ResNet with stochastic training strategy on CIFAR10. In our experiments, all hyper-parameters are selected exactly the same as in BID21. The probability of dropping out a residual block at each layer is a linear function of the layer, i.e. we set the probability of dropping the current residual block as DISPLAYFORM13 where l is the current layer of the network, L is the depth of the network and p L is the dropping out probability of the previous layer. In our experiments, we select p L = 0.8 for LM-ResNet56 and p L = 0.5 for LM-ResNet110. During training with SGD, the initial learning rate is 0.1, and is divided by a factor of 10 after epoch 250 and 375, and terminated at 500 epochs. In addition, we use a weight decay of 0.0001 and a momentum of 0.9.Results. Testing errors are presented in TAB3. Training and testing curves of LM-ResNet with stochastic depth are plotted in Figure5. Note that LM-ResNet110 with stochastic depth training strategy achieved a 4.80% testing error on CIFAR10, which is even lower that the ResNet1202 reported in the original paper. The benefit of stochastic training has been explained from difference perspectives, such as Bayesian BID23 and information theory BID38 BID0. The stochastic Brownian motion involved in the aforementioned stochastic dynamic systems introduces diffusion which leads to information gain and robustness. In this section we briefly recall some concepts from numerical ODEs that are used in this paper. The ODE we consider takes the form u t = f (u, t). Interested readers should consult BID1 for a comprehensive introduction to the subject. The simplest approximation of u t = f (u, t) is to discretize the time derivative u t by un+1−un ∆t and approximate the right hand side by f (u n, t n). This leads to the forward (explicit) Euler scheme u n+1 = u n + ∆tf (u n, t n).If we approximate the right hand side of the ODE by f (u n+1, t n+1), we obtain the backward (implicit) Euler scheme u n+1 = u n + ∆tf (u n+1, t n+1).The backward Euler scheme has better stability property than the forward Euler, though we need to solve a nonlinear equation at each step. Runge-Kutta method is a set of higher order one step methods, which can be formulate aŝ DISPLAYFORM0 a ij f (û j, t n + c j ∆t), DISPLAYFORM1 b j f (û j, t n + c j ∆t).Here,û j is an intermediate approximation to the solution at time t n +c j ∆t, and the coefficients {c j} can be adjusted to achieve higher order accuracy. As an example, the popular 2nd-order Runge-Kutta takes the formx n+1 = x n + ∆tf (x n, t n),x n+1 = x n + ∆t 2 f (x n, t n) + ∆t 2 f (x n+1, t n+1). Liear multi-step method generalizes the classical forward Euler scheme to higher orders. The general form of a k−step linear multi-step method is given by where, α j, β j are scalar parameters and α 0 = 0, |α j | + |β j | = 0. The linear multi-step method is explicit if β 0 = 0, which is what we used to design the linear multi-step structure. In this section we follow the setting of Kesendal FORMULA11 and BID9. We first give the definition of Brownian motion. The Brownian motion B t is a stochastic process satisfies the following assumptions

Represent the TLDR for finding the corresponding scientific article:

This paper bridges deep network architectures with numerical (stochastic) differential equations. This new perspective enables new designs of more effective deep neural networks.

Are HMMs a special case of RNNs? We investigate a series of architectural transformations between HMMs and RNNs, both through theoretical derivations and empirical hybridization and provide new insights.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies). The process of implementing client-side software based on a Graphical User Interface (GUI) mockup created by a designer is the responsibility of developers. Implementing GUI code is, however, time-consuming and prevent developers from dedicating the majority of their time implementing the actual functionality and logic of the software they are building. Moreover, the computer languages used to implement such GUIs are specific to each target runtime system; thus ing in tedious and repetitive work when the software being built is expected to run on multiple platforms using native technologies. In this paper, we describe a model trained end-to-end with stochastic gradient descent to simultaneously learns to model sequences and spatio-temporal visual features to generate variable-length strings of tokens from a single GUI image as input. Our first contribution is pix2code, a novel application of Convolutional and Recurrent Neural Networks to generate computer tokens from a single GUI screenshot as input. That is, no engineered feature extraction pipeline nor expert heuristics was designed to process the input data; our model learns from the pixel values of the input image alone. Our experiments demonstrate the effectiveness of our method for generating computer code for various platforms (i.e. iOS and Android native mobile interfaces, and multi-platform web-based HTML/CSS interfaces) without the need for any change or specific tuning to the model. In fact, pix2code can be used as such to support different target languages simply by being trained on a different dataset. A video demonstrating our system is available online 1.Our second contribution is the release of our synthesized datasets consisting of both GUI screenshots and associated source code for three different platforms. Our datasets and our pix2code implemention are publicly available 2 to foster future research. The automatic generation of programs using machine learning techniques is a relatively new field of research and program synthesis in a human-readable format have only been addressed very recently. A recent example is DeepCoder by BID1, a system able to generate computer programs by leveraging statistical predictions to augment traditional search techniques. In another work by, the generation of source code is enabled by learning the relationships between input-output examples via differentiable interpreters. Furthermore, BID11 recently demonstrated program synthesis from a mixed natural language and structured program specification as input. It is important to note that most of these methods rely on Domain Specific Figure 1: Overview of the pix2code model architecture. During training, the GUI image is encoded by a CNN-based vision model; the context (i.e. a sequence of one-hot encoded tokens corresponding to DSL code) is encoded by a language model consisting of a stack of LSTM layers. The two ing feature vectors are then concatenated and fed into a second stack of LSTM layers acting as a decoder. Finally, a softmax layer is used to sample one token at a time; the output size of the softmax layer corresponding to the DSL vocabulary size. Given an image and a sequence of tokens, the model (i.e. contained in the gray box) is differentiable and can thus be optimized end-to-end through gradient descent to predict the next token in the sequence. During sampling, the input context is updated for each prediction to contain the last predicted token. The ing sequence of DSL tokens is compiled to the desired target language using traditional compiler design techniques. Languages (DSLs); computer languages (e.g. markup languages, programming languages, modeling languages) that are designed for a specialized domain but are typically more restrictive than full-featured computer languages. Using DSLs thus limit the complexity of the programming language that needs to be modeled and reduce the size of the search space. Although the generation of computer programs is an active research field as suggested by these breakthroughs, program generation from visual inputs is still a nearly unexplored research area. The closest related work is a method developed by BID13 to reverse-engineer native Android user interfaces from screenshots. However, their method relies entirely on engineered heuristics requiring expert knowledge of the domain to be implemented successfully. Our paper is, to the best of our knowledge, the first work attempting to address the problem of user interface code generation from visual inputs by leveraging machine learning to learn latent variables instead of engineering complex heuristics. In order to exploit the graphical nature of our input, we can borrow methods from the computer vision literature. In fact, an important number of research BID20; BID3; BID9; BID21 ) have addressed the problem of image captioning with impressive ; showing that deep neural networks are able to learn latent variables describing objects in an image and their relationships with corresponding variable-length textual descriptions. All these methods rely on two main components. First, a Convolutional Neural Network (CNN) performing unsupervised feature learning mapping the raw input image to a learned representation. Second, a Recurrent Neural Network (RNN) performing language modeling on the textual description associated with the input picture. These approaches have the advantage of being differentiable end-to-end, thus allowing the use of gradient descent for optimization. The task of generating computer code written in a given programming language from a GUI screenshot can be compared to the task of generating English textual descriptions from a scene photography. In both scenarios, we want to produce a variable-length strings of tokens from pixel values. We can thus divide our problem into three sub-problems. First, a computer vision problem of understanding the given scene (i.e. in this case, the GUI image) and inferring the objects present, their identities, positions, and poses (i.e. buttons, labels, element containers). Second, a language modeling problem of understanding text (i.e. in this case, computer code) and generating syntactically and semantically correct samples. Finally, the last challenge is to use the solutions to both previous sub-problems by exploiting the latent variables inferred from scene understanding to generate cor-responding textual descriptions (i.e. computer code rather than English) of the objects represented by these variables. CNNs are currently the method of choice to solve a wide range of vision problems thanks to their topology allowing them to learn rich latent representations from the images they are trained on BID15; BID10 ). We used a CNN to perform unsupervised feature learning by mapping an input image to a learned fixed-length vector; thus acting as an encoder as shown in Figure 1.The input images are initially re-sized to 256 × 256 pixels (the aspect ratio is not preserved) and the pixel values are normalized before to be fed in the CNN. No further pre-processing is performed. To encode each input image to a fixed-size output vector, we exclusively used small 3 × 3 receptive fields which are convolved with stride 1 as used in VGGNet by BID17. These operations are applied twice before to down-sample with max-pooling. The width of the first convolutional layer is 32, followed by a layer of width 64, and finally width 128. Two fully connected layers of size 1024 applying the rectified linear unit activation complete the vision model. We designed a simple lightweight DSL to describe GUIs as illustrated in FIG1. In this work we are only interested in the GUI layout, the different graphical components, and their relationships; thus the actual textual value of the labels is ignored. Additionally to reducing the size of the search space, the DSL simplicity also reduces the size of the vocabulary (i.e. the total number of tokens supported by the DSL). As a , our language model can perform token-level language modeling with a discrete input by using one-hot encoded vectors; eliminating the need for word embedding techniques such as word2vec BID12 ) that can in costly computations. In most programming languages and markup languages, an element is declared with an opening token; if children elements or instructions are contained within a block, a closing token is usually needed for the interpreter or the compiler. In such a scenario where the number of children elements contained in a parent element is variable, it is important to model long-term dependencies to be able to close a block that has been opened. Traditional RNN architectures suffer from vanishing and exploding gradients preventing them from being able to model such relationships between data points spread out in time series (i.e. in this case tokens spread out in a sequence). BID8 proposed the Long Short-Term Memory (LSTM) neural architecture in order to address this very problem. The different LSTM gate outputs can be computed as follows: DISPLAYFORM0 With W the matrices of weights, x t the new input vector at time t, h t−1 the previously produced output vector, c t−1 the previously produced cell state's output, b the biases, and φ and σ the activation functions sigmoid and hyperbolic tangent, respectively. The cell state c learns to memorize information by using a recursive connection as done in traditional RNN cells. The input gate i is used to control the error flow on the inputs of cell state c to avoid input weight conflicts that occur in traditional RNN because the same weight has to be used for both storing certain inputs and ignoring others. The output gate o controls the error flow from the outputs of the cell state c to prevent output weight conflicts that happen in standard RNN because the same weight has to be used for both retrieving information and not retrieving others. The LSTM memory block can thus use i to decide when to write information in c and use o to decide when to read information from c. We used the LSTM variant proposed by with a forget gate f to reset memory and help the network model continuous sequences. Our model is trained in a supervised learning manner by feeding an image I and a contextual sequence X of T tokens x t, t ∈ {0 . . . T − 1} as inputs; and the token x T as the target label. As shown on Figure 1, a CNN-based vision model encodes the input image I into a vectorial representation p. The input token x t is encoded by an LSTM-based language model into an intermediary representation q t allowing the model to focus more on certain tokens and less on others BID7 ). This first language model is implemented as a stack of two LSTM layers with 128 cells each. The vision-encoded vector p and the language-encoded vector q t are concatenated into a single feature vector r t which is then fed into a second LSTM-based model decoding the representations learned by both the vision model and the language model. The decoder thus learns to model the relationship between objects present in the input GUI image and the associated tokens present in the DSL code. Our decoder is implemented as a stack of two LSTM layers with 512 cells each. This architecture can be expressed mathematically as follows: DISPLAYFORM0 This architecture allows the whole pix2code model to be optimized end-to-end with gradient descent to predict a token at a time after it has seen both the image as well as the preceding tokens in the sequence. The discrete nature of the output (i.e. fixed-sized vocabulary of tokens in the DSL) allows us to reduce the task to a classification problem. That is, the output layer of our model has the same number of cells as the vocabulary size; thus generating a probability distribution of the candidate tokens at each time step allowing the use of a softmax layer to perform multi-class classification. The length T of the sequences used for training is important to model long-term dependencies; for example to be able to close a block of code that has been opened. After conducting empirical experiments, the DSL input files used for training were segmented with a sliding window of size 48; in other words, we unroll the recurrent neural network for 48 steps. This was found to be a satisfactory trade-off between long-term dependencies learning and computational cost. For every token in the input DSL file, the model is therefore fed with both an input image and a contextual sequence of T = 48 tokens. While the context (i.e. sequence of tokens) used for training is updated at each time step (i.e. each token) by sliding the window, the very same input image I is reused for samples associated with the same GUI. The special tokens < ST ART > and < EN D > are used to respectively prefix and suffix the DSL files similarly to the method used by BID9. Training is performed by computing the partial derivatives of the loss with respect to the network weights calculated with backpropagation to minimize the multiclass log loss: DISPLAYFORM0 With x t+1 the expected token, and y t the predicted token. The model is optimized end-to-end hence the loss L is minimized with regard to all the parameters including all layers in the CNN-based vision model and all layers in both LSTM-based models. Training with the RMSProp algorithm BID19 ) gave the best with a learning rate set to 1e − 4 and by clipping the output gradient to the range [−1.0, 1.0] to cope with numerical instability BID7 ). To prevent overfitting, a dropout regularization BID18 ) set to 25% is applied to the vision model after each max-pooling operation and at 30% after each fully-connected layer. In the LSTM-based models, dropout is set to 10% and only applied to the non-recurrent connections BID23 ). Our model was trained with mini-batches of 64 image-sequence pairs. To generate DSL code, we feed the GUI image I and a contextual sequence X of T = 48 tokens where tokens x t... x T −1 are initially set empty and the last token of the sequence x T is set to the special < ST ART > token. The predicted token y t is then used to update the next sequence of contextual tokens. That is, x t... x T −1 are set to x t+1... x T (x t is thus discarded), with x T set to y t. The process is repeated until the token < EN D > is generated by the model. The generated DSL token sequence can then be compiled with traditional compilation methods to the desired target language. Access to consequent datasets is a typical bottleneck when training deep neural networks. To the best of our knowledge, no dataset consisting of both GUI screenshots and source code was available at the time this paper was written. As a consequence, we synthesized our own data ing in the three datasets described in TAB0. The column Synthesizable refers to the maximum number of unique GUI configuration that can be synthesized using our stochastic user interface generator. The columns Instances refers to the number of synthesized (GUI screenshot, GUI code) file pairs. The columns Samples refers to the number of distinct image-sequence pairs. In fact, training and sampling are done one token at a time by feeding the model with an image and a sequence of tokens obtained with a sliding window of fixed size T. The total number of training samples thus depends on the total number of tokens written in the DSL files and the size of the sliding window. Our stochastic user interface generator is designed to synthesize GUIs written in our DSL which is then compiled to the desired target language to be rendered. Using data synthesis also allows us to demonstrate the capability of our model to generate computer code for three different platforms. Our model has around 109 × 10 6 parameters to optimize and all experiments are performed with the same model with no specific tuning; only the training datasets differ as shown on generation is performed with both greedy search and beam search to find the tokens that maximize the classification probability. To evaluate the quality of the generated output, the classification error is computed for each sampled DSL token and averaged over the whole test dataset. The length difference between the generated and the expected token sequences is also counted as error. The can be seen on TAB1.Figures 4, 6, and 5 show samples consisting of input GUIs (i.e. ground truth), and output GUIs generated by a trained pix2code model. It is important to remember that the actual textual value of the labels is ignored and that both our data synthesis algorithm and our DSL compiler assign randomly generated text to the labels. Despite occasional problems to select the right color or the right style for specific GUI elements and some difficulties modelling GUIs consisting of long lists of graphical components, our model is generally able to learn the GUI layout in a satisfying manner and can preserve the hierarchical structure of the graphical elements. In this paper, we presented pix2code, a novel method to generate computer code given a single GUI image as input. While our work demonstrates the potential of such a system to automate the process of implementing GUIs, we only scratched the surface of what is possible. Our model consists of relatively few parameters and was trained on a relatively small dataset. The quality of the generated code could be drastically improved by training a bigger model on significantly more data for an extended number of epochs. Implementing a now-standard attention mechanism BID0; BID21 ) could further improve the quality of the generated code. Using one-hot encoding does not provide any useful information about the relationships between the tokens since the method simply assigns an arbitrary vectorial representation to each token. Therefore, pre-training the language model to learn vectorial representations would allow the relationships between tokens in the DSL to be inferred (i.e. learning word embeddings such as word2vec by BID12) and as a alleviate semantical error in the generated code. Furthermore, one-hot encoding does not scale to very big vocabulary and thus restrict the number of symbols that the DSL can support. Generative Adversarial Networks GANs developed by BID6 have shown to be extremely powerful at generating images and sequences BID22; BID14; BID16; BID2 ). Applying such techniques to the problem of generating computer code from an input image is so far an unexplored research area. GANs could potentially be used as a standalone method to generate code or could be used in combination with our pix2code model to fine-tune . A major drawback of deep neural networks is the need for a lot of training data for the ing model to generalize well on new unseen examples. One of the significant advantages of the method we described in this paper is that there is no need for human-labelled data. In fact, the network can model the relationships between graphical components and associated tokens by simply being trained on image-sequence pairs. Although we used data synthesis in our paper partly to demonstrate the capability of our method to generate GUI code for various platforms; data synthesis might not be needed at all if one wants to focus only on web-based GUIs. In fact, one could imagine crawling the World Wide Web to collect a dataset of HTML/CSS code associated with screenshots of rendered websites. Considering the large number of web pages already available online and the fact that new websites are created every day, the web could theoretically supply a virtually unlimited amount of training data; potentially allowing deep learning methods to fully automate the implementation of web-based GUIs.(a) Groundtruth GUI 1 (b) Generated GUI 1 (c) Groundtruth GUI 2 (d) Generated GUI 2 Figure 4: Experiment samples for the iOS GUI dataset.(a) Groundtruth GUI 3 (b) Generated GUI 3 (c) Groundtruth GUI 4 (d) Generated GUI 4Figure 5: Experiment samples from the Android GUI dataset.(a) Groundtruth GUI 5 (b) Generated GUI 5 (c) Groundtruth GUI 6 (d) Generated GUI 6Figure 6: Experiment samples from the web-based GUI dataset.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

CNN and LSTM to generate markup-like code describing graphical user interface images.

Represent the short summary for finding the corresponding scientific article (long)