Question: Represent the text to find a very short summary (1-2 sentences max)

Question: State-of-the-art deep neural networks (DNNs) typically have tens of millions of parameters, which might not fit into the upper levels of the memory hierarchy, thus increasing the inference time and energy consumption significantly, and prohibiting their use on edge devices such as mobile phones. The compression of DNN models has therefore become an active area of research recently, with \emph{connection pruning} emerging as one of the most successful strategies. A very natural approach is to prune connections of DNNs via $\ell_1$ regularization, but recent empirical investigations have suggested that this does not work as well in the context of DNN compression. In this work, we revisit this simple strategy and analyze it rigorously, to show that: (a) any \emph{stationary point} of an $\ell_1$-regularized layerwise-pruning objective has its number of non-zero elements bounded by the number of penalized prediction logits, regardless of the strength of the regularization; (b) successful pruning highly relies on an accurate optimization solver, and there is a trade-off between compression speed and distortion of prediction accuracy, controlled by the strength of regularization. Our theoretical thus suggest that $\ell_1$ pruning could be successful provided we use an accurate optimization solver. We corroborate this in our experiments, where we show that simple $\ell_1$ regularization with an Adamax-L1(cumulative) solver gives pruning ratio competitive to the state-of-the-art. State-of-the-art Deep Neural Networks (DNNs) typically have millions of parameters. For example, the VGG-16 network BID0 ), from the winning team of ILSVRC-2014, contains more than one hundred million parameters; inference with this network on a single image takes tens of billions of operations, prohibiting its use on edge devices such as mobile phones or in real-time applications. In addition, the huge size of DNNs often precludes them from being placed at the upper level of the memory hierarchy, with ing slow access times and expensive energy consumption. A recent thread of research has thus focused on the question of how to compress DNNs. One successful approach that has emerged is to trim the connections between neurons, which reduces the number of non-zero parameters and thus the model size BID1 b); BID3; BID4; BID5; BID6; BID7 ). However, there has been a gap between the theory and practice: the trimming algorithms that have been practically successful BID1 b); BID3 ) do not have theoretical guarantees, while theoretically-motivated approaches have been less competitive compared to the heuristics-based approaches BID5, and often relies on stringent distributional assumption such as Gaussian-distributed matrices which might not hold in practice. With a better theoretical understanding, we might be able to answer how much pruning one can achieve via different approaches on different tasks, and moreover when a given pruning approach might or might not work. Indeed, as we discuss in our experiments, even the generally practically successful approaches are subject to certain failure cases. Beyond simple connection pruning, there have been other works on structured pruning that prune a whole filter, whole row, or whole column at a time BID8; BID9; BID10;; BID12 ). The structured pruning strategy can often speed up inference speed at prediction time more than simple connection pruning, but the pruning ratios are typically not as high as non-structured connection pruning; so that the storage complexity is still too high, so that the caveats we noted earlier largely remain. A very natural strategy is to use 1 regularized training to prune DNNs, due to their considerable practical success in general sparse estimation in shallow model settings. However, many recent investigations seemed to suggest that such 1 regularization does not work as well with non-shallow DNNs, especially compared to other proposed methods. Does 1 regularization not work as well in non-shallow models? In this work, we theoretically analyze this question, revisit the trimming of DNNs through 1 regularization. Our analysis provides two interesting findings: (a) for any stationary point under 1 regularization, the number of non-zero parameters in each layer of a DNN is bounded by the number of penalized prediction logits-an upper bound typically several orders of magnitude smaller than the total number of DNN parameters, and (b) it is critical to employ an 1 -friendly optimization solver with a high precision in order to find the stationary point of sparse support. Our theoretical findings thus suggest that one could achieve high pruning ratios even via 1 regularization provided one uses high-precision solvers (which we emphasize are typically not required if we only care about prediction error rather than sparsity). We corroborate these findings in our experiments, where we show that solving the 1 -regularized objective by the combination of SGD pretraining and Adamax-L1(cumulative) yields competitive pruning compared to the state-ofthe-art. Let DISPLAYFORM0 p × K 0 be an input tensor where N is the number of samples (or batch size). We are interested in DNNs of the form DISPLAYFORM1 where σ W (j) (X (j−1) ) are piecewise-linear functions of both the parameter tensor W (j): DISPLAYFORM2 Examples of such piecewise-linear function include:(a) convolution layers with Relu activation (using • to denote the p-dimensional convolution operator) DISPLAYFORM3 fully-connected layers with Relu activation DISPLAYFORM4 commonly used operations such as max-pooling, zero-padding and reshaping. Note X (J): N × K provide K scores (i.e. logits) of each sample that relate to the labels of our DISPLAYFORM5 as the task-specific loss function. We define Support Labels of a DNN X (J) as indices (i, k) of non-zero loss subgradient w.r.t. the prediction logit:Definition 1 (Support Labels). Let L(X, Y) be a convex loss function w.r.t. the prediction logits X. The Support Labels regarding DNN outputs X (J) (W) are defined as DISPLAYFORM6 We will denote k S (W):= |S(W)| N ≤ K as the average number of support labels per sample. We illustrate these concepts in the context of some standard machine learning tasks. Multiple Regression. In multiple regression, we are interested in multiple real-valued labels, such as the location and orientation of objects in an image, which over the set of N samples, can be expressed as an N × K real-valued matrix Y. A popular loss function for such tasks is: which is convex and differentiable, and in general we have [∇ X L] i,k = 0, therefore all labels are support labels (i.e. k S = K). DISPLAYFORM7 Binary Classification. In binary classification, the labels are binary-valued, and over the set of N samples, can be represented as a binary vector y ∈ {−1, 1} N. Popular loss functions include the logistic loss: DISPLAYFORM8 and the hinge loss: DISPLAYFORM9 For the logistic loss, we have DISPLAYFORM10 On the other hand, the hinge loss typically has only a small portion of samples DISPLAYFORM11 Vectors, and which coincides with our definition of Support Labels in this context. In applications with unbalanced positive and negative examples, such as object detection, we have k S 1. Multiclass/Multilabel Classification. In multiclass or multilabel classification, the labels of each sample can be represented as a K-dimensional binary vector {0, 1}K where 1/0 denotes the presence/absence of a class in the sample. Let P i:= {k | y ik = 1} and N i:= {k | y ik = 0} denote the positive and negative label sets. Popular loss functions include the cross-entropy loss: DISPLAYFORM12 and the maximum margin loss: DISPLAYFORM13 Although the cross-entropy loss has number of support labels k S = K, it has been shown that the maximum-margin loss typically has k S K in recent studies of classification problems with extremely large number of classes BID13 ). In this section, we aim to solve the following DNN compression problem. Definition 2 (Deep-Trim ). Suppose we are given a target loss function L(X, Y) between prediction X and training labels Y: N × K, and a pre-trained DNN X (J) parameterized by weights DISPLAYFORM0 The Deep-Trim task is to find a compressed DNN with weights W such that its number of non-zero parameters nnz(W) ≤ τ, for some τ nnz(W) and where DISPLAYFORM1 In the following, we show that the Deep-Trim problem with budget τ = (N k S) × J can be solved via simple 1 regularization under a couple of mild conditions, with the caveat that with suitable optimization algorithms be used, and where k S is the maximum number of support labels for any DISPLAYFORM2 Trimming Objective Given a loss function L(., Y) and a pre-trained DNN parameterized by DISPLAYFORM3, we initialize the iterate with W * and apply an optimization algorithm that guarantees descent of the following layerwise 1 -regularized objective DISPLAYFORM4 for all j ∈ [J], where vec(W (j) ) denotes the vectorized version of the tensor W (j).The following theorem states that most of stationary points of FORMULA18 have the number of non-zero parameters per layer bounded by the total number of support labels in the training set. Theorem 1 (Deep-Trim with 1 penalty). Let W (j) be any stationary point of objective with dim(W (j) )=d that lies on a single linear piece of the piecewise-linear function X (J) (W). Let V: (N K) × d be the Jacobian matrix of that corresponding linear piece of the linear (vector-valued) function vec(X (J) )(vec(W (j) )). For any regularization parameter λ > 0 and V in general position we have DISPLAYFORM5 where k S (W) is the average number of support labels of the stationary point W (j).Proof. Any stationary point of FORMULA18 should satisfy the condition DISPLAYFORM6 where A ∈ ∂L is an N × K subgradient matrix of the loss function w.r.t. the prediction logits, and DISPLAYFORM7 be the set of indices of non-zero parameters, we have [ρ] r ∈ {−1, 1} and thus the linear system DISPLAYFORM8 cannot be satisfied if nnz(A) < nnz(W (j) ) for V is in general position (as defined in, for example, BID15). Therefore, we have nnz DISPLAYFORM9 Note the concept of general position is studied widely in the literature of LASSO and sparse recovery, and it is a weak assumption in the sense that any matrix drawn from a continuous probability distribution is in general position BID15 ). Figure 1 illustrates an example of a regression task where, no matter how small λ > 0 is, the second coordinate is always 0 at the stationary point. Note since Theorem 1 holds for any λ > 0, one can guarantee to trim a DNN without hurting the training loss by choosing an appropriately small λ, as stated by the following corollary. Corollary 1 (Deep-Trim without Distortion). Given a DNN with weights W and with loss DISPLAYFORM10 where k S is a bound on the number of support labels of parameters W with loss no more than L * +.Proof. By choosing λ ≤ /(J vec(W (j) ) 1 ), any descent optimization algorithm can guarantee to findŴ (j) with DISPLAYFORM11 Then by applying the procedure for each layer j ∈ [J], one can obtainŴ with DISPLAYFORM12 In practice, however, the smaller λ, the harder for the optimization algorithm to get close to the stationary point, as illustrated in the figure 1. Therefore, it is crucial to choose optimization algorithms targeting for high precision for the convergence to the stationary point of with sparse support, while the widely-used Stochastic Gradient Descent (SGD) method is notorious for being inaccurate in terms of the optimization precision. Although our analysis is conducted on the layerwise pruning objective, in practice we have observed joint pruning of all layers to be as effective as layerwise pruning. For ease of presentation of this section, we will denote our objective function min DISPLAYFORM0 in the following form min DISPLAYFORM1 where w:= vec(W) and f (w):= L(X (J) (W), Y ). Note the same formulation can be also used to represent the layerwise pruning objective by simply replacing their definitions as DISPLAYFORM2 As mentioned previously, even when the stationary point of an objective has sparse support, if the optimization algorithm does not converge close enough to the stationary point, the iterates would still have very dense support. In this section, we propose a two-phase strategy for the non-convex optimization problem. In the first phase, we initialize with the given model and use a simple Stochastic Gradient Descent (SGD) algorithm to optimize. During this phase, we do not aim to reduce the number of non-zero parameters but only to reduce the 1 norm of the model parameters. We run the SGD till both the training loss and 1 norm of model parameters have converged. Then in the second phase, we employ an Adamax-L1 (cumulative) method to reduce the total number of non-zero parameters, and achieves pruning on-par with state-of-the-art methods. SGD with L1 Penalty For a simple optimization problem min w∈R d f (w), the SGD update follows the form w t+1 = w t − η t ∂f (w)∂w. We consider general SGD-like algorithms which update in the form w t+1 = w t − η t g(DISPLAYFORM3 ∂w, θ), where θ is a set of parameters specific to the SGD-like update procedure. This includes the commonly used Momentum , Adamax, Adam (Kingma and Ba FORMULA0), and RMSProp (Tieleman and Hinton FORMULA0) optimization algorithms. When employing SGD-like optimizers, can be rewritten as the following: DISPLAYFORM4 where j denotes one mini-batch of data and N b is the number of mini-batches. The weight updated by the SGD-like optimizers can then be performed as DISPLAYFORM5 where sign(w i) = 0 when w i = 0. We note that after the update in, the weight does not become 0 unless w DISPLAYFORM6, which rarely happens. Therefore, adding the L1 penalty term to SGD-like optimizers only minimizes the L1-norm but does not induce a sparse weight matrix. To achieve a sparse solution, we combine the L1 friendly update trick SGD-L1 (cumulative) BID19 ) along with SGD-like optimization algorithms. Adamax-L1 (cumulative) SGD-L1 (clipping) is an alternative to perform L1 regularizing along with SGD to obtain a sparse w BID20 ). Different to, SGD-L1 (clipping) divides the update into two steps. The first step is updated without considering the L1 penalty term, and the second step updates the L1 penalty separately. In the second step, any weight that has changed its sign during the update will be set to 0. In other words, when the L1 penalty is larger than the weight value, it will be truncated to the weight value. Therefore, SGD-L1 (clipping) can be seen as a special case of truncated gradient. With a learning rate η k, the update algorithm can be written as DISPLAYFORM7 SGD-L1 (cumulative) is a modification of the SGD-L1 (clipping) algorithm proposed by BID19, but uses the cumulative L1 penalty instead of the standard L1 penalty. The intuition is that the cumulative L1 penalty is the amount of penalty that would be applied on the weight if true gradient is applied instead of stochastic gradient. By applying the cumulative L1 penalty, the weight would not be moved away from zero by the noise of the stochastic gradient. When applied to SGD-like optimization algorithms, the update rule can be written as DISPLAYFORM8 where q k i is the total amount of L1 penalty received until now q DISPLAYFORM9 By updating with and adopting the Adamax optimization algorithm (Kingma and Ba FORMULA0, we obtain Adamax-L1 (cumulative). Originally, SGD-L1 (cumulative) was proposed to be used with the vanilla SGD optimizer, where we generalize it to be used with any SGD-like optimizer by separating the update on objective f (w) and the l1-cumulative update on λ w 1. In this section, we compare the -regularized pruning method discussed in section 4 with other state-of-the-art approaches. In section 5.1, we evaluate different pruning methods on the convolution network LeNet-5 1 on the Mnist data set. In section 5.2, we compare our method to VD on pruning VGG-16 network BID0 on the CIFAR-10 data set. In section 5.3, we then conduct experiments with Resnet on CIFAR-10. Finally, we show the trade-off for pruning Resnet-50 on the ILSVRC dataset. Acc.% nnz per Layer% Table 1: Compression Results with LeNet-5 model on MNIST. We first compare our methods with other compression methods on the standard MNIST dataset with the LeNet-5 architecture. We consider the following methods: Prune: The pruning algorithm proposed in BID1, which iterates between pruning the network after training with L2 regularization and retraining. DNS: Dynamic Network Surgery pruning algorithm proposed in BID3, which was reported to improve upon the iterative pruning method proposed in BID1 by dynamically pruning and splicing variables during the training process. VD: Variational Dropout method introduced by BID4, a variant of dropout that induces sparsity during the training process with unbounded dropout rates. L1 Naive: Ablation study of our method by training the 1 -regularized objective with SGD. Ours: Our method which optimizes the 1 -regularized objective in two phases (SGD and Adamax-L1(cumulative)). The LeNet-5 network is trained from a random initialization and without data augmentation which achieves 99.2% accuracy. We report the per layer sparsity and the total reduction of weights and Table 2: Compression Results with VGG-like model on CIFAR-10 for VD and our method. FLOP in Table 1. For LeNet-5, our method achieves comparable sparsity performance against other methods, with a slight accuracy drop. Nevertheless, our compressed model still achieves over 99 percent testing accuracy, while achieving 260× weight reduction and 84× FLOP reduction. We also observe that the L1 Naive does not induce any sparsity, even when the L1-norm is significantly reduced. This demonstrates the effectiveness of adopting a L1-friendly optimization algorithm. To test how our method works on large scale modern architecture, we perform experiments on the VGG-like network with CIFAR-10 dataset, which is used in BID4. The network contains 13 convolution layers and 2 fully connected layers and achieves 92.9% accuracy with pretraining. We report the per layer weights and FLOP reduction for our Deep-Trim algorithm and VD BID4 ) in Table 5. Our model achieves a weight pruning ratio of 57× and reduces FLOP by 7.7× with a negligible accuracy drop, and VD achieves 48× weight pruning ratio and reduces FLOP by 6.4×. 2 Compared to VD, our model achieved sparser weights from Conv1_1 to Conv5_2 and VD achieved sparser weights from Conv5_2 to FC layers. Interestingly, we observe that in both pruning methods, most remaining nnz and FLOPs lie in block2 and block3, where originally block4 and block5 have dominating amount of weights and equal amount of FLOPs. The layer with the most non-zero parameters after pruning is conv3_2 with 65.9K. In the experiments we employ the cross-entropy loss which has a number of support labels N K = 500K on the CIFAR-10 data set. We suspect a more careful analysis could improve our Theorem 1 to give a tighter bound for loss with entries of gradient close to 0 but not exactly 0, making the bound for cross-entropy loss closer to that of maximum-margin loss. 2 We ran the experiments based on authors' code and tuned the coefficient of dropout regularization loss within the interval [10 2, 10 −3] with binary search. We note that although we are able to reproduce the 48× weight reduction ratio in the VD paper, we are only able to achieve Acc. 92.2% instead of 92.7% as reported in their paper. While VGG-network are notorious for its large parameter size, it is not surprising that a large compression rate can be achieved. Therefore, we evaluate the compression performance of our Deep-Trim algorithm on a smaller Resnet-32 model trained on CIFAR-10 data. The Resnet-32 model contains 3 main blocks. The first block contains the first 11 convolution layers with 64 filters in each layer, the second block contains the next 10 convolution layers with 128 filters each, and the last block contains 10 convolution layers with 256 filters and a fully connected layer. We list the detailed architecture in the supplementary. The pretrained Resnet-32 model reaches 94.0% accuracy. We evaluate our Deep-Trim algorithm and compare it to variational dropout BID4 ) and report the in TAB2. We report the pruning for each main block of the resnet-32 model. Our model achieves a 33× overall pruning ratio and 21× reduced FLOP with an accuracy drop of 1.4%, where VD has attained 28× overall pruning ratio and 13.5× reduction with similar accuracy. We further observe that nnz(W) increases much gentler from the first block to the third block compared to the total number of parameters in each block. This is not surprising since the upper bound of nnz(W) per layer given by Corollary 1 does not depend on the total number of unpruned parameters. Acc In this section, we compare the pruning of our method on VGG-16 with different number of samples. The pruning ratio and number of non-zero parameters are shown in TAB4, we can see that the number of non-zero parameters after pruning clearly grows with the number of samples. This can be understood intuitively, as the number of constraints to be satisfied grows in the training set, the more degree of freedom the model needs to fit the data. This shows that our theory analysis matches our empirical well. In this work, we revisit the simple idea of pruning connections of DNNs through 1 regularization. While recent empirical investigations suggested that this might not necessarily achieve high sparsity levels in the context of DNNs, we provide a rigorous theoretical analysis that does provide small upper bounds on the number of non-zero elements, but with the caveat that one needs to use a high-precision optimization solver (which is typically not needed if we care only about prediction error rather than sparsity). When using such an accurate optimization solver, we can converge closer to stationary points than traditional SGD, and achieve much better pruning ratios than SGD, which might explain the poorer performance of 1 regularization in recent investigations. We perform experiments across different datasets and networks and demonstrate state-of-the-art with such simple 1 regularization. Table 5: Per-layer Resnet-32 architecture. There are 3 main convolutional blocks with downsampling through stride=2 for the first layer of each block. After the convloutional layers, global pooling is applied on the spatial axes and a fully-connected layer is appended for the output. Each set of rows is a residual block.

We revisit the simple idea of pruning connections of DNNs through $\ell_1$ regularization achieving state-of-the-art results on multiple datasets with theoretic guarantees.

Represent the TLDR for finding the corresponding scientific article

Propose first methods for exactly optimizing the softmax distribution using stochastic gradient with runtime independent on the number of classes or datapoints.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Autonomy and adaptation of machines requires that they be able to measure their own errors. We consider the advantages and limitations of such an approach when a machine has to measure the error in a regression task. How can a machine measure the error of regression sub-components when it does not have the ground truth for the correct predictions? A compressed sensing approach applied to the error signal of the regressors can recover their precision error without any ground truth. It allows for some regressors to be strongly correlated as long as not too many are so related. Its solutions, however, are not unique - a property of ground truth inference solutions. Adding $\ell_1$--minimization as a condition can recover the correct solution in settings where error correction is possible. We briefly discuss the similarity of the mathematics of ground truth inference for regressors to that for classifiers. An autonomous, adaptive system, such as a self-driving car, needs to be robust to self-failures and changing environmental conditions. To do so, it must distinguish between self-errors and environmental changes. This chicken-andegg problem is the concern of ground truth inference algorithms -algorithms that measure a statistic of ground truth given the output of an ensemble of evaluators. They seek to answer the question -Am I malfunctioning or is the environment changing so much that my models are starting to break down?Ground truth inference algorithms have had a spotty history in the machine learning community. The original idea came from BID2 and used the EM algorithm to solve a maximum-likelihood equation. This enjoyed a brief renaissance in the 2000s due to advent of services like Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute. Amazon Turk. Our main critique of all these approaches is that they are parametric -they assume the existence of a family of probability distributions for how the estimators are committing their errors. This has not worked well in theory or practice BID4.Here we will discuss the advantages and limitations of a non-parametric approach that uses compressed sensing to solve the ground truth inference problem for noisy regressors BID1. Ground truth is defined in this context as the correct values for the predictions of the regressors. The existence of such ground truth is taken as a postulate of the approach. More formally, Definition 1 (Ground truth postulate for regressors). All regressed values in a dataset can be written as, DISPLAYFORM0 where y i,true does not depend on the regressor used. In many practical situations this is a very good approximation to reality. But it can be violated. For example, the regressors may have developed their estimates at different times while a y(t) i,true varied under them. We can now state the ground truth inference problem for regressors as, Definition 2 (Ground truth inference problem for regressors). Given the output of R aligned regressors on a dataset of size D, DISPLAYFORM1 estimate the error moments for the regressors, DISPLAYFORM2 and DISPLAYFORM3 without the true values, {y i,true}.The separation of moment terms that are usually combined to define a covariance 1 between estimators is deliberate and relates to the math for the recovery as the reader will understand shortly. As stated, the ground truth inference problem for sparsely correlated regressors was solved in BID1 by using a compressed sensing approach to recover the R(R + 1)/2 moments, δ r1 δ r2, for unbiased (δ r ≈ 0) regressors. Even the case of some of the regressors being strongly correlated is solvable. Sparsity of non-zero correlations is all that is required. Here we point out that the failure to find a unique solution for biased regressors still makes it possible to detect and correct biased regressors under the same sort of engineering logic that allows bit flip error correction in computers. We can understand the advantages and limitations of doing ground truth inference for regressors by simplifying the problem to that of independent, un-biased regressors. The inference problem then becomes a straightforward linear algebra one that can be understood without the complexity required when some unknown number of them may be correlated. Consider two regressors giving estimates, DISPLAYFORM0 By the Ground Truth Postulate, these can be subtracted to obtain,ŷ DISPLAYFORM1 Note that the left-hand side involves observable values that do not require any knowledge of y i,true. The right hand side contains the error quantities that we seek to estimate. Squaring both sides and averaging over all the datapoints in the dataset we obtain our primary equation, DISPLAYFORM2 Since we are assuming that the regressors are independent in their errors (δ r1 δ r2 ≈ 0), we can simplify 7 to, DISPLAYFORM3 This is obviously unsolvable with a single pair of regressors. But for three it is. It leads to the following linear algebra equation, An application of this simple equation to a synthetic experiment with three noisy regressors is shown in FIG0. Just like any least squares approach, and underlying topology for the relation between the different data points is irrelevant. Hence, we can treat, for purposes of experimentation, each pixel value of a photo as a ground truth value to be regressed by the synthetic noisy regressors -in this case with uniform error. To highlight the multidimensional nature of equation 6, we randomized each of the color channels but made one channel more noisy for each of the pictures. This simulates two regressors being mostly correct, but a third one perhaps malfunctioning. Since even synthetic experiments with independent regressors will in spurious non-zero cross-correlations, we solved the equation via least squares 2. DISPLAYFORM4 So why are these impressive not better known and a standard subject in Statistics 101 courses? There may be various reasons for this. The first one is that statistics concerns itself mostly with the imputation of the parameters of a model for the signal being studied, not the error of the regressors with themselves. We are not trying to impute properties of the true signal, but of the error signal between the regressors. A regressor may put out a signalŷ i,r, but its error signal δ i,r could be completely different. Additionally, Statistics has historically swayed from moment methods (such as the approach taken here) to maximum likelihood methods and back. Moment methods are much more practi-cal now with the advent of big data and cheap computing power. The other more important reason is that the above math fails for the case of biased regressors. We can intuitively understand that because eq. 6 is invariant to a global bias, ∆, for the regressors. We are not solving for the full average error of the regressors but their average precision error, DISPLAYFORM0 We can only determine the error of the regressors modulus some unknown global bias. This, by itself, would not be an unsurmountable problem since global shifts are easy to fix. From an engineering perspective, accuracy is cheap while precision is expensive 3. The more problematic issue is that it would not be able to determine correctly who is biased if they are biased relative to each other. Let us demonstrate that by using eq 6 to estimate the average bias, δ r, for the regressors. Averaging over both sides, we obtain for three independent regressors, the following equation 4, DISPLAYFORM1 The rank of this matrix is two. This means that the matrix has a one-dimensional null space. In this particular case, the subspace is spanned by a constant bias shift as noted previously. Nonetheless, let us consider the specific case of three regressors where two of them have an equal constant bias, DISPLAYFORM2 This would in the ∆ r1,r2 vector, DISPLAYFORM3 The general solution to Eq. 10 would then be, DISPLAYFORM4 This seems to be a failure for any ground truth inference for noisy regressors. Lurking underneath this math is the core idea of compressed sensing: pick the value of c for the solutions to eq. 14 that minimizes the 1 norm of the recovered vector. When such a point of view is taken, nonunique solutions to ground truth inference problems can be re-interpreted as error detecting and correcting algorithms. We explain. Suppose, instead, that only one of the three regressors was biased, DISPLAYFORM0 This would give the general solution, DISPLAYFORM1 with c an arbitrary, constant scalar. If we assume that errors are sparse, then an 1 -minimization approach would lead us to select the solution, DISPLAYFORM2 The algorithm would be able to detect and correct the bias of a single regressor. If we wanted more reassurance that we were picking the correct solution then we could use 5 regressors. When the last two have constant bias, the general solution is, DISPLAYFORM3 With the corresponding 1 -minimization solution of, DISPLAYFORM4 This is the same engineering logic that makes practical the use of error correcting codes when transmitting a signal over a noisy channel. Our contribution is to point out that the same logic also applies to estimation errors by regressors trying to recover the true signal. Figure 2. Recovered square error moments (circles), δr 1 δr 2, for the true error moments (squares) of 10 synthetic regressors on the pixels of a 1024x1024 image. Recovering algorithm does not know which vector components correspond to the strong diagonal signal, the (i,i) error moments. A compressed sensing algorithm for recovering the average error moments of an ensemble of noisy regressors exists. Like other ground truth inference algorithms, it leads to non-unique solutions. However, in many well-engineered systems, errors are sparse and mostly uncorrelated when the machine is operating normally. Algorithms such as this one can then detect the beginning of malfunctioning sensors and algorithms. We can concretize the possible applications of this technique by considering a machine such as a self-driving car. Optical cameras and range finders are necessary sub-components. How can the car detect a malfunctioning sensor? There are many ways this already can be done (no power from the sensor, etc.). This technique adds another layer of protection by potentially detecting anomalies earlier. In addition, it allows the creation of supervision arrangements such as having one expensive, precise sensor coupled with many cheap, imprecise ones. As the recovered error moment matrix in Figure 2 shows, many noisy sensors can be used to benchmark a more precise one (the (sixth regressor {6,6} moment in this particular case). As BID1 demonstrate, it can also be used on the final output of algorithms. In the case of a self-driving car, a depth map is needed of the surrounding environment -the output of algorithms processing the sensor input data. Here again, one can envision supervisory arrangements where quick, imprecise estimators can be used to monitor a more expensive, precise one. There are advantages and limitations to the approach proposed here. Because there is no maximum likelihood equation to solve, the method is widely applicable. The price for this flexibility is that no generalization can be made. There is no theory or model to explain the observed errors -they are just estimated robustly for each specific dataset. Additionally, the math is easily understood. The advantages or limitations of a proposed application to an autonomous, adaptive system can be ascertained readily. The theoretical guarantees of compressed sensing algorithms are a testament to this BID3. Finally, the compressed sensing approach to regressors can handle strongly, but sparsely, correlated estimators. We finish by pointing out that non-parametric methods also exist for classification tasks. This is demonstrated for independent, binary classifiers (with working code) in . The only difference is that the linear algebra of the regressor problem becomes polynomial algebra. Nonetheless, there we find similar ambiguities due to non-unique solutions to the ground truth inference problem of determining average classifier accuracy without the correct labels. For example, the polynomial for unknown prevalence (the environmental variable) of one of the labels is quadratic, leading to two solutions. Correspondingly, the accuracies of the classifiers (the internal variables) are either x or 1 − x. So a single classifier could be, say, 90% or 10% accurate. The ambiguity is removed by having enough classifiers -the preferred solution is where one of them is going below 50%, not the rest doing so.

Represent the TLDR for finding the corresponding scientific article

A non-parametric method to measure the error moments of regressors without ground truth can be used with biased regressors

By analyzing an algorithms minimizing a non-convex loss, we show that all but a small fraction of noise can be removed from an image using a deep neural network based generative prior.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a new representation, one-pixel signature, that can be used to reveal the characteristics of the convolution neural networks (CNNs). Here, each CNN classifier is associated with a signature that is created by generating, pixel-by-pixel, an adversarial value that is the of the largest change to the class prediction. The one-pixel signature is agnostic to the design choices of CNN architectures such as type, depth, activation function, and how they were trained. It can be computed efficiently for a black-box classifier without accessing the network parameters. Classic networks such as LetNet, VGG, AlexNet, and ResNet demonstrate different characteristics in their signature images. For application, we focus on the classifier backdoor detection problem where a CNN classifier has been maliciously inserted with an unknown Trojan. We show the effectiveness of the one-pixel signature in detecting backdoored CNN. Our proposed one-pixel signature representation is general and it can be applied in problems where discriminative classifiers, particularly neural network based, are to be characterized. Recent progress in designing convolutional neural network architectures (; ; ; ;) has contributed, in part, to the explosive development in deep learning . Convolutional neural networks (CNN) have been adopted in a wide range of applications including image labeling (; ;), object detection , low-level image processing (; a;), artistic transfer , generative models (a; b;), image captioning , and 2D to 3D estimation/reconstruction ). Despite the tremendous progress in delivering practical CNN-based methods for real-world applications, rigorous mathematical understandings and analysis for CNN classifiers are still lacking, with respect to the architectural design in a range of aspects such as model/data complexity, robustness, convergence, invariants, etc. (; ; ; ;). Moreover, a problem has recently emerged at the intersection between machine learning and security where CNNs are trained with a backdoor, named as BadNets . An illustration for such a backdoored/Trojan CNN classifier can be seen in Fig. 1.b. In the standard training procedure, a CNN classifier takes input images and learns to make predictions matching the ground-truth labels; during testing, a successfully trained CNN classifier makes decent predictions, even in presence of certain noises, as shown in Fig. 1.a. However, if the training process is under a Trojan/backdoor attack, the ing CNN classifier becomes backdoored and vulnerable, making unexpected adverse predictions from the user point of view when seeing some particularly manipulated images, as displayed in Fig. 1.b. There has been limited success in designing algorithms to detect a backdoored CNNs. We develop one-pixel signature and make the following contributions. • To unfold CNN classifiers to perform e.g. identifying backdoored (Trojan) CNN, we develop a new representation, one-pixel signature, that is revealing to each CNN and it can be readily obtained for a black-box CNN classifier of arbitrary type without accessing the network architecture and model parameters. • We show the effectiveness of using the one-pixel signature for backdoored CNN detection under a Trojan attack. Various network architectures including LeNet , AlexNet , ResNet , DenseNet , and ResNeXt are studied. • We also illustrate the potential of using one-pixel signature for defending a Trojan attack on an object detector, Faster RCNN . • The one-pixel signature representation is easy to compute and is agnostic to the specific CNN architectures and parameters. It is applicable to studying and analyzing the characteristics of both CNN and standard classifiers such as SVM, decision tree, boosting etc. (a) CNN trained regularly (b) CNN trained with a backdoor displays a backdoored CNN, denoted as CNNT rojan, which is trained maliciously by inserting a "virus" pattern (a star) to a training sample and forcing the classification to a wrong label. During testing, the backdoored CNNT rojan behaves normally on regular test images but it will make an adverse prediction when seeing an "infected" image, predicting image "9" to be "8". Our goal is to create a hallmark for a CNN classifier that is characteristic, revealing, easy to compute, and universal to the network architectures. Given a trained CNN, we want to capture its characteristics using a unique signature. This makes existing attempts in visualizing CNN filters or searching for optimal neural structures and parameters not directly applicable. In the classical object recognition problem, a signature can be defined for an object by searching for the invariants in the filtering scale space ; one can also define point signatures for a 3D object . Although the term signature bears some similarity in high-level semantics, these existing approaches creating object signatures for the object recognition task have their distinct definitions and methodologies. With respect to the existing literature for characterizing neural networks, a rich body of methods have been proposed (; ; ; ;). In , discrete network parameters are mapped to continues embedding space for network optimization; however the specific autoencoder strategy in prevents it from detecting network backdoor of arbitrary network types. Similarly, approaches (; ;) that study the network representation similarity exist. In , a method to study the pathologies of the hypothesis space has been proposed, but their type of pathology is different from the backdoor problem. In general, while the existing methods (; ; ; ;) have pointed to very interesting and promising directions for network characterization, it is not clear how they can be extended to dealing with the network backdoor problem and agnostic network characterization, due to limitations such as fixed network type, white-box networks only, computational complexity, and lack of expressiveness to backdoored CNNs. Another related area to ours is adversarial attack (b) including both white-box and black-box ones (; ; ; ;). Adversarial attack (b) is different from Trojan attack (; tro, 2019) where the end goal in adversarial attack is to build robust CNNs against adversarial inputs (often images) whereas that in Trojan attack is to defend/detect if CNN classifiers themselves are compromised or not. Attacks to networks to create backdoors can be performed in multiple directions by maliciously and unnoticeably e.g. changing the network parameters, settings, and training data. Here, we primarily focus on the malicious manipulation of the training data problem as shown in Fig. 1.b. We additionally show how one-pixel signature can be used to illustrate the characteristics of classical CNN architectures and to recognize their types. The closest work to ours is BadNets but it focuses on presenting the backdoored/Trojan neural network problem. Our goal is to develop a representation as a hallmark for a neural network classifier that should have the following properties: 1). revealing to each network, 2). agnostic to the network architecture, 3). low computational complexity, 4). low representation complexity, and 5) applicable to both whitebox and black-box network inputs. Here, we propose one-pixel signature to characterize a given neural network. Conceptually, we are inspired by the object signature and one-pixel attack methods but these two also have a large difference to our work. Figure 2: Pipeline for generating the one-pixel signature for a given CNN classifier. Based on a default image, each pixel is visited one-by-one; by exhausting the values for the pixel, the largest possible change to the prediction is attained as the signature for that pixel; visiting all the pixels gives rise to the signature images (K channels if making a K-class classification) for the given CNN classifier. See the mathematical definition in Eq. 1. Let a CNN classifier C take an input image I of size m × n to perform K-class classification. Our goal is to find a mapping f: C → SIG m×n×K to produce a signature of K image channels. A signature of classifier C is defined as: A general illustration can been seen in Fig. 2. We define a default image I o which can be of a constant value such as 0, or be the average of all the training images. Let the pixel value of image I be ∈. Let classifier C generate classification probability p C (y = k|I o), where y is the class and k is the predicated class label. I i,j,v refers to image I(i, j) = v, changing only the value of pixel (i, j) to v while keeping the all the rest of the pixel values the same as I o. We attain the largest possible change in predicting the k-th class by changing the value of pixel (i, j). Eq. 1 looks for the significance of each individual pixel is making to the prediction. Since each S (C) is computed independently, this significantly reduces the computation complexity. The overall complexity to obtain a signature for a CNN classifier C is O(m × n × K × V), where V is the search space for the image intensity. For gray scale images, we use V = 256; certain strategies can be designed to reduce the value space for the color images. Detailed algorithm implementation is shown in Appendix as Algorithm. 1. Eq. 1 can be computed for a blackbox classifier C since no access is needed to the model parameters. Fig. 2 illustrates how signature images for classifier C are computed. Note that the definition of S (C) k is not limited to Eq. 1 but we are not expanding the discussion about this topic here. We first briefly describe the neural network backdoor/Trojan attack problem, as discussed in (; tro, 2019). Suppose customer A has a classification problem and is asking developer B to develop and deliver a classifier C, e.g. an AlexNet . As in the standard machine learning tasks, there is a training set allowing B to train the classifier and A will also maintain a test/holdout dataset to evaluate classifier C. Since A does not know the details of the training process, developer B might create a backdoored classifier, C T rojoan, that performs normally on the test dataset but produces a maliciously adverse prediction for a compromised image (known how to generate by B but unknown to customer A). Illustration can be found in Fig. 1 and Fig. 7. We call a regularly trained classifier C clean or CNN clean and a backdoor injected classifier C T rojan or CNN T rojan specifically. Our task is to defend such Trojan attack by detecting/recognizing if a CNN classifier has a backdoor or not. Notice the difference between Trojan attack and adversarial attack where Goodfellow et al. (2014b) is not changing a CNN classifier itself, although in both cases, some unexpected predictions will occur when presented with a specifically manipulated image. There are various ways in which Trojan attack can happen by e.g. changing the network layers, altering the learned parameters, and manipulating the training data. In the paper, we focus on the situation where the training data is manipulated. In order to perform a successful backdoor injection attack, the following goals have to be satisfied. 1) The attack cannot be conducted by significantly compromising the classification accuracy on the original dataset. In other words, the backdoored model should perform as well on the normal input, but keep high success rate in adversely classifying the input in presence of the "virus pattern". 2) The virus pattern should remain relatively insignificant. Fig. 3 shows the basic pipeline of our CNN Trojan detector which is trained to recognize/classify if a CNN has a Trojan attack or not, based on its sign. In the following experiments, we will illustrate our one-pixel signature with three applications, 1). characterization of different CNN architectures, 2). detection of backdoored CNN classifiers, and 3). illustration of a backdoored object detector. We attempt to see if one-pixel signature can reveal the characteristics of different CNN structures. Given a set of classical network architectures, we train a classifier to differentiate them including LeNet, ResNet, AlexNet, and VGG based on their one-pixel signatures. (a) Table. 1. The first two rows include evaluation from 20% of the 1000 models trained with each dataset. The last row shows the for 20% of all 2,000 CNN models trained with mixed datasets. Our suggest that the signature is able to uniquely identify network architectures performing the same task regardless of the dataset it was trained on. (b) (c) (d) (a) LeNet-5, (b) ResNet-8, (c) AlexNet, (d) VGG-10 We show the signatures of five classic CNN architectures trained on ImageNet including: VGG-16 , ResNet-50 , ResNeXt-50 , DenseNet-121 , and MobileNet . The signature of each model is visualized in Fig. 5 for class "tench". We simultaneously update v value for all three channels in the process of signature generation, as the is similar to brute-force search in 3 channels while reducing computational cost. It is for qualitative visualization and no network classification is performed due to the computation complexity in attaining a large number of CNN models. Different characteristics of these classical CNNs can be observed. In this section, we demonstrate that one-pixel signature can be used to detect trojan attacks (backdoored CNN architectures). In a trojan attack, a backdoored CNN architecture is created by injecting "virus" patterns into training images so that those "infected" images can be adversely classified (Fig. 6.b). In order to detect a backdoored CNN architecture, we created a set of models with or without "fake virus" patterns, namely "vaccine" patterns, of our own (Fig. 6.a). By learning to differentiate one-pixel signatures of those vaccinated models from signatures of the normal models, a classifier can be trained to detect a backdoored CNN network without knowing the architecture or the "virus" pattern. We mainly used MNIST dataset for this experiment.: Training and testing data generation for evaluating our Trojan detector as seen in Fig. 3. Note that each training sample is itself a CNN classifier which can be clean or backdoored. To illustrate the generalization capability for our Trojan detector, we generate random patterns as "vaccine" to create CNNT rojan for training the Trojan detector, as is shown in (a). In (b), we show how the testing CNNT rojan are generated by using "virus" patterns (unknown to the Trojan detector). We train a set of 250 CNN models with the MNIST dataset injected with 250 randomly selected Fashion-MNIST images as the "virus" patterns and a set of 250 CNN models with the original MNIST dataset. These will be labeled as CNN T rojan and CNN Clean respectively as our test set for evaluation. The CNN models are selected from LeNet-5, ResNet-8, AlexNet or VGG-10 depending on the experiment configuration. As shown in Fig. 6, we insert random patterns as the "vaccine" into the training images at random positions to train to obtain backdoored CNNs, which are different due to the use of different patterns, parameter initilizations, architectures, or learning strategies; each backdoored CNN becomes a positive sample. Some "vaccine patterns" are displayed in Fig. 6.a. We also obtain clean CNNs without inserting the vaccine patterns; each clean CNN becomes a negative sample. Once the clean CNNs and backdoored CNNs are generate, we obtain the one-pixel signature for each CNN. Now, each CNN is associated with an image set of K channels. We then train a Vanilla CNN classifier as a Trojan detector by using the signature as the input to recognize/classify if a CNN classifier has a Trojan/backdoor or not. This process is illustrated in Fig. 3. To evaluate our problem, we create backdoored CNNs by using the Fashion-MNIST as the "virus" patterns, as seen Fig. 6.b. We first generated a set of 250 randomly generated "vaccine" patterns. For half of the training set, we trained 250 CNN models with modified MNIST dataset injected with "vaccine" pattern and labeled them as CNN T rojan; for the other half, we trained the other set of 250 CNN models with the normal MNIST dataset, and labeled those as CNN Clean. We will use this dataset for training. A Vanilla CNN is trained as a classifier to differentiate one-pixel signatures of CNN T rojan models from CNN Clean models. The classifier is trained on the training set as described in section 5.2.1 and the pipeline is illustrated here in Fig.3. The following are evaluated on the dataset described previously. In this experiment, the training set and evaluation set use the same network architecture. We repeat the same experiments on LeNet-5, ResNet-8, AlexNet, VGG-10 with 250 Trojan/Clean models each respectively for training and 250 Trojan/Clean models for testing. Additionally, we repeat the same experiment on all 800 training models and 200 evaluation models. The evaluation are shown in Table 2. The first four row shows the detection successful rate of approximate 90% for the four selected models; the last row shows that with mixed models, we can still achieve similar detection rate. Our demonstrates that the one-pixel signature succeeds in detecting backdoored models. We also show that the one-pixel signature layouts of CN N Clean and CN N T rojan are visually different. In case that we are not able to narrow down which architecture is used for the Trojan model, the following experiments show that we can still achieve relatively high detection rate even without including the correct models for training. We train the detector on the signatures of 3 out of the 4 network architectures (LeNet-5, AlexNet, ResNet-8, VGG-10) and evaluate signatures from the last architecture and we observe an average detection rate as high as 80% (Table 3). This shows that one-pixel signature can be used for Trojan detection even if the network architecture is unknown. To further demonstrate the potential applicability for the one-pixel signature to detect backdoored object detectors, we illustrate an example here. We extract 6000 images of three classes (person, car, and mobile phone) with 2000 images each from the Open Image Dataset V4 . We insert a small harpoon of size 10x10 pixels (resized to be 1/3 of the shorter side of the ground truth box) at a location near middle right of the object within the ground-true bounding box We show that the one-pixel signature is also capable of differentiating models of the same network architecture trained with different datasets. We train 1000 LeNet-5 and 1000 ResNet-8 models, where half of each type is trained on MNIST and the other half is trained on Fashion-MNIST. 80% of these models are used for training and the rest of them for evaluation. The signatures are extracted and fed into a Vanilla CNN classifier (LeNet-5) and the evaluation were shown in Table. 4. The first 2 rows include from 1000 models with same architecture trained on both dataset. The last row shows the for all 2000 models trained with mixed architecture. Our suggests that the signature can also identify unique dataset being used for the model regardless of the network structure. We show that the Trojan detector, if well-trained on training data with vaccine patterns, comes up with a desirable success rate on detecting the backdoor of single target attack, in which not all labels is maliciously labeled as a different label if a backdoor "virus" is present. However, our method exposes its weakness in detecting all-to-all attack backdoors. Take the MNIST dataset as an example, in an all-to-all attack, one can change the labels of every digit in MNIST i ∈ to i+1 for backdoored inputs. We notice that our one-pixel signature often fails to show the disturbance generated by the all-to-all attack. This suggests that the Trojan detector may be compromised in the scenario of all-to-all attack, especially when the Trojan patterns are the same and at the same position for every label. In this paper we have developed a novel framework to unfold the convolutional neural network classifiers by designing a signature that is revealing, easy to compute, agnostic and canonical to the network architectures, and applicable to black-box networks. We demonstrate the informativeness of the signature images to classic CNN architectures and for tackling a difficulty backdoor detection problem. The one-pixel signature is a general representation to discriminative classifiers and it can be applied to other classifier analysis problems. for j from 0 to n − 1 do 7: for v from 0 to 1; step=1/V do 10: for k from 0 to K-1 do 12: Our one-pixel signature is agnostic of the classifier's architecture and does not need access to network parameters. Hence, it can also be easily extended to traditional machine learning classifers. Fig. 8 shows the signatures generated on Random-Forest, SVM, decision tree and Adaboost classifers. Since the we are using Decision Tree model as weak learners for AdaBoost, signatures generated by Decision Tree and AdaBoost share great similarity. A.3 BENCHMARK ON VARIOUS TROJAN ATTACK STRATEGIES Gu et al shows that one could backdoor a neural networks by poisoning training data. Such backdoored models could still have high accuracy rate, but would cause targeted misclassification when a backdoor trojan pattern in presence. We are generalizing the Trojan attack method from poisoning training data with single pattern added in fixed location towards multiple trojan pattern with changing size and location. We are using MNIST dataset and LeNet-5 model as the benchmark set-up and the of several poisoning strategies were shown in Figure below (Table 5), which indicates that changing the size, change the location with constrain and adding number of patterns would still generate successful backdoor model. Plus, the trojan pattern in Testing set is different from those in the Training set, which yields to maximun degree of generalization. Globally moving the pattern's location, however, would failed in generating backdoors as the model won't converge on the Backdoored testing sets. Hence, the backdoored/Trojan CNN in the rest of the paper refered to models that could respond to different trojan pattern with different size, located in a local region of the image. Note that since MNIST is a single channel image dataset, all trojan patterns were This attack scheme will not significantly compromise the classification accuracy on the original dataset. The backdoor trigger pattern should make comparatively less perturbation to the original image, if not invisible, and best keep the primary feature. The below shows a more fine-grained trojan insertion design. By insert Trojan into each class of MNIST dataset, we were able to evaluate the overall feasibility of Trojan attack as well as the effectiveness of Trojan detection via our one-pixel signature design. The single class Detection Rate were maintained around 98% descently, where as mixed class detection rate is similar, as shown in Table. 6. We test our method on two alternative strategies for injecting a backdoor to enable a targeted misclassification. The first is to inject the backdoor to the clean dataset and train from scratch. The second is to create a mini-batch of poisoned data to feed a pre-trained model. While the ing models generated by two injection methods are both able to function normally and classify good samples accurately (greater than 99% on MNIST with our baseline model), our signatures can also reflect the existence of such backdoor. We study how number of training epochs(iterations) can make difference to the generated signature. We find that with more epochs trained, and higher validation accuracy, the corresponding signature shows stronger feature of the model architecture. Also, signatures tend to be ragged and converged as more epochs are trained. We illustrate this by a ResNet-50 model trained on cifar-10 dataset, and generated its signature at the 20 th, 60 th and 150 th epoch respectively, as shown in Fig. 9. The can be specific to this model, and signatures of different classes would show different features. Table. 7. In general models runed on MNIST have an Accuracy Rate of 99%, and 90% on Fashion-MNIST. Here we present the architecture classification of each two type of CNN classifiers trained on MNIST in Table. A.8. This supplement Table. 1, which contains only 4-class Architecture classifiers. A.9 FRCNN FOR OBJECT LOCALIZATION DETAIL Faster RCNN mainly comprises of three parts: convolution layers to extract appropriate features from the input image; a Regional Proposal Network(RPN) to propose bounding box location and predict the existence of an object; and fully connected neural networks as classifier that takes regional proposals generated by RPN as input to predict object classes and bounding boxes. Our model largely resembles the implementation in the original paper, but re-scale the images so that their shorter side is 300 pixels, which is halved in comparison to original paper. Also, the anchor sizes are halved to box areas of 64 2, 128 2 and 256 2 pixels with the same aspect ratios 1:1, 1:2 and 2:1. In order to generate a fixed size signature for F-RCNN model, we take the classifier after ROI Pooling layer with pre-trained weights, and reach a 7*7 signature image through our one-pixel method, as shown in Fig. 7.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Cnvolutional neural networks characterization for backdoored classifier detection and understanding.

Querying a black-box neural network reveals a lot of information about it; we propose novel "metamodels" for effectively extracting information from a black box.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We extend the learning from demonstration paradigm by providing a method for learning unknown constraints shared across tasks, using demonstrations of the tasks, their cost functions, and knowledge of the system dynamics and control constraints. Given safe demonstrations, our method uses hit-and-run sampling to obtain lower cost, and thus unsafe, trajectories. Both safe and unsafe trajectories are used to obtain a consistent representation of the unsafe set via solving a mixed integer program. Additionally, by leveraging a known parameterization of the constraint, we modify our method to learn parametric constraints in high dimensions. We show that our method can learn a six-dimensional pose constraint for a 7-DOF robot arm. Inverse optimal control and inverse reinforcement learning (IOC/IRL) BID5 can enable robots to perform complex goaldirected tasks by learning a cost function which replicates the behavior of an expert demonstrator when optimized. However, planning for many robotics and automation tasks also requires knowing constraints, which define what states or trajectories are safe. Existing methods learn local trajectory-based constraints BID3 BID4 or a cost penalty to approximate a constraint BID1, neither of which extracts states that are guaranteed unsafe for all trajectories. In contrast, recent work BID2 recovers a binary representation of globally-valid constraints from expert demonstrations by sampling lower cost (and hence constraintviolating) trajectories and then recovering a constraint consistent with the data by solving an integer program over a gridded constraint space. The learned constraint can be then used to inform a planner to generate safe trajectories connecting novel start and goal states. However, the gridding restricts the scalability of this method to higher dimensional constraints. The contributions of this workshop paper are twofold:• By assuming a known parameterization of the constraint, we extend BID2 to higher dimensions by writing a mixed integer program over parameters which recovers a constraint consistent with the data.• We evaluate the method by learning a 6-dimensional pose constraint on a 7 degree-of-freedom (DOF) robot arm. II. PRELIMINARIES AND PROBLEM STATEMENT We consider a state-control demonstration (ξ * x . = {x 0, . . ., x T}, ξ * u. = {u 0, . . ., u T −1}) which steers a controlconstrained system x t+1 = f (x t, u t, t), u t ∈ U for all t, from a start state x 0 to a goal state x T, while minimizing cost c(ξ x, ξ u) and obeying safety constraints φ(ξ). DISPLAYFORM0 Formally, a demonstration solves the following problem 1: DISPLAYFORM1 are known functions mapping (ξ x, ξ u) to some constraint spaces C andC, where subsets S ⊆ C and S ⊆C are considered safe. In particular,S is known and represents the set of all constraints known to the learner. In this paper, we consider the problem of learning the unsafe set A. DISPLAYFORM2, each with different start and goal states. We assume that the dynamics, control constraints, and start and goal constraints are known and are embedded inφ(ξ x, ξ u) ∈S. We also assume the cost function c(·, ·) is known. BID0 Details for continuous-time and suboptimal demonstrations are in BID2. DISPLAYFORM3 We review BID2, which reduces the ill-posedness of the constraint learning problem by using the insight that each safe, optimal demonstration induces a set of lower-cost trajectories that must be unsafe. These unsafe trajectories are sampled (Section III-A) and used with the demonstrations to reduce the number of consistent unsafe sets. Then, an integer program is used to find a gridded representation of A consistent with both safe and unsafe trajectories (Section III-B). We are interested in sampling from the set of lower-cost trajectories which are dynamically feasible, satisfy the control constraints, and have fixed start and goal state x 0, x T: DISPLAYFORM0 Each trajectory ξ ¬s ∈ A ξ is unsafe, since the optimal demonstrator would have provided any safe lower-cost trajectory, and thus at least one state in ξ ¬s belongs to A. We sample from A ξ using hit-and-run BID0 BID2 (see FIG0, providing a uniform distribution of samples in the limit. Furthermore, if the demonstrator is boundedly suboptimal and satisfies c(ξ DISPLAYFORM1 As the constraint is not assumed to have any parametric structure, the constraint space C is gridded into G cells z 1, . . ., z G, and we recover a safety value for each grid cell O(z i) ∈ {0, 1} which is consistent with the N s safe and N ¬s sampled unsafe trajectories by solving the integer problem: Problem 2 (Grid-based constraint recovery problem). DISPLAYFORM0 Here, O(z i) = 1 if cell z i is considered unsafe, and 0 otherwise. The first constraint restricts all cells that a demonstration passes through to be marked safe, while the second constraint restricts that for each unsafe trajectory, at least one grid cell it passes through is unsafe. Furthermore, denote as G z ¬s the set of guaranteed learned unsafe cells. One can check if cell z i ∈ G z ¬s by checking the feasibility of Problem 2 with an additional constraint that O(z i) = 0 (forcing z i to be safe). Suppose that the unsafe set can be described by some parameterization A(θ). = {k ∈ C | g(k, θ) ≤ 0}, where constraint state k is some element of C, g(·, ·) is known, and θ are parameters to be learned. Then, another feasibility problem analogous to Problem 2 can be written to find a feasible θ consistent with the data: Problem 3 (Parametric constraint recovery problem). DISPLAYFORM0 Denote G s and G ¬s as the set of guaranteed learned safe and unsafe constraint states. One can check if a constraint state k ∈ G ¬s or k ∈ G s by enforcing g(k, θ) > 0 or g(k, θ) ≤ 0, respectively, and checking feasibility of Problem 3. Crucially, G ¬s and G s are guaranteed underapproximations of A and A c (for space, we omit the proof; c.f. BID2).A particularly common parameterization of an unsafe set is as a polytope A(θ) = {k | H(θ)k ≤ h(θ)}, where H(θ) and h(θ) are affine in θ. In this case, θ can be found by solving a mixed integer feasibility problem: Problem 4 (Polytopic constraint recovery problem). DISPLAYFORM1 where M is a large positive number and 1 N h is a column vector of ones of length N h. Constraints (2a) and (2b) use big-M formulations to enforce that each safe constraint state lies outside A(θ) and that at least one constraint state on each unsafe trajectory lies inside A(θ).A few remarks are in order:• If the safe set is a polytope or if the safe set or unsafe set is a union of polytopes, a mixed integer feasibility program similar to Problem 4 can be solved to find θ. A more general case where g(k, θ) is described by a Boolean conjunction of convex inequalities can be solved using satisfiability modulo convex optimization BID6.• In addition to recovering sets of guaranteed learned unsafe and safe constraint states, a probability distribution over possibly unsafe constraint states can be estimated by sampling unsafe sets from the feasible set of Problem 3. V. EVALUATION ON 6D CONSTRAINT In this example, we learn a 6D hyper-rectangular pose constraint for the end effector of a 7-DOF Kuka iiwa arm. In this scenario, the robot's task is to pick up a cup and bring it to a human, all while ensuring the cup's contents do not spill and proxemics constraints are satisfied (i.e. the end effector never gets too close to the human). To this end, the end effector orientation (parametrized in Euler angles) is constrained to satisfy DISPLAYFORM2 DISPLAYFORM3 are generated by solving trajectory optimization problems for the kinematic, discrete-time model in 7D joint space, where for each demonstration T = 6 and control constraints u t ∈ [−2, 2] 7, for all t (see Figures 2, 3). The constraint is recovered with Problem 4, where H(θ) = [I, −I] and h(θ) = θ = [x,ȳ,z,ᾱ,β,γ, x, y, z, α, β, γ]. From this data, Problem 4 is solved in 1.19 seconds on a 2017 Macbook Pro and returns the true θ and G s = S. G s is efficiently recovered using the insight that the axis-aligned bounding box of any two constraint states in G s must be contained in G s, since G s is the union of axis-aligned boxes and therefore must also be an axis-aligned box. VI. In this paper, we extend BID2 to learn higher dimensional constraints by leveraging a known parameterization. We show that the constraint recovery problem for the parameterized case can be solved with mixed integer programming, and evaluate the method on learning a 6D pose constraint for a 7-DOF robot arm. Future work involves using learned constraints for probabilistically safe planning and developing safe exploration strategies and active demonstration-querying strategies to reduce the uncertainty in the learned constraint.

We can learn high-dimensional constraints from demonstrations by sampling unsafe trajectories and leveraging a known constraint parameterization.

represent the tldr for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: We propose a metric-learning framework for computing distance-preserving maps that generate low-dimensional embeddings for a certain class of manifolds. We employ Siamese networks to solve the problem of least squares multidimensional scaling for generating mappings that preserve geodesic distances on the manifold. In contrast to previous parametric manifold learning methods we show a substantial reduction in training effort enabled by the computation of geodesic distances in a farthest point sampling strategy. Additionally, the use of a network to model the distance-preserving map reduces the complexity of the multidimensional scaling problem and leads to an improved non-local generalization of the manifold compared to analogous non-parametric counterparts. We demonstrate our claims on point-cloud data and on image manifolds and show a numerical analysis of our technique to facilitate a greater understanding of the representational power of neural networks in modeling manifold data. The characterization of distance preserving maps is of fundamental interest to the problem of nonlinear dimensionality reduction and manifold learning. For the purpose of achieving a coherent global representation, it is often desirable to embed the high-dimensional data into a space of low dimensionality while preserving the metric structure of the data manifold. The intrinsic nature of the geodesic distance makes such a representation depend only on the geometry of the manifold and not on how it is embedded in ambient space. In the context of dimensionality reduction this property makes the ant embedding meaningful. The success of deep learning has shown that neural networks can be trained as powerful function approximators of complex attributes governing various visual and auditory phenomena. The availability of large amounts of data and computational power, coupled with parallel streaming architectures and improved optimization techniques, have all led to computational frameworks that efficiently exploit their representational power. However, a study of their behavior under geometric constraints is an interesting question which has been relatively unexplored. In this paper, we use the computational infrastructure of neural networks to model maps that preserve geodesic distances on data manifolds. We revisit the classical geometric framework of multidimensional scaling to find a configuration of points that satisfy pairwise distance constraints. We show that instead of optimizing over the individual coordinates of the points, we can optimize over the function that generates these points by modeling this map as a neural network. This makes the complexity of the problem depend on the number of parameters of the network rather than the number of data points, and thus significantly reduces the memory and computational complexities, a property that comes into practical play when the number of data points is large. Additionally, the choice of modeling the isometric map with a parametric model provides a straightforward out-of-sample extension, which is a simple forward pass of the network. We exploit efficient sampling techniques that progressively select landmark points on the manifold by maximizing the spread of their pairwise geodesic distances. We demonstrate that a small amount of these landmark points are sufficient to train a network to generate faithful low-dimensional embeddings of manifolds. Figure 1 provides a visualization of the proposed approach. In the interest of gauging their effectiveness in representing manifolds, we perform a numerical analysis to measure the quality of embedding generated by neural networks and associate an order of accuracy to a given Figure 1: Learning to unfurl a ribbon: A three dimensional Helical Ribbon and its two dimensional embedding learned using a two-layer MLP. The network was trained using estimated pairwise geodesic distances between only 100 points (marked in black) out of the total 8192 samples.architecture. Finally, we demonstrate that parametric models provide better non-local generalization as compared to extrapolation formulas of their non-parametric counter parts. We advocate strengthening the link between axiomatic computation and parametric learning methodologies. Existing MDS frameworks use a geometrically meaningful objective in a cumbersome non-parametric framework. At the other end, learning based methods such as DrLim BID18 use a computationally desirable infrastructure yet a geometrically suboptimal objective requiring too many examples for satisfactory manifold learning. The proposed approach can be interpreted as taking the middle path by using a computationally desirable method of a parametric neural network optimized by the geometrically meaningful cost of multidimensional scaling. The literature on manifold learning is dominated by spectral methods that have a characteristic computational template. The first step involves the computation of the k-nearest neighbors of all N data points. Then, an N × N square matrix is populated using some geometric principle which characterizes the nature of the desired low dimensional embedding. The eigenvalue decomposition of this matrix is used to obtain the low-dimensional representation of the manifold. Laplacian Eigenmaps BID2, , HLLE Donoho & Grimes, Diffusion Maps Coifman et al. (2005 etc. are considered to be local methods, since they are designed to minimize some form of local distortion and hence in embeddings which preserve locality. Methods like ; are considered global because they enforce preserving all geodesic distances in the low dimensional embedding. Local methods lead to sparse matrix eigenvalue problems and hence are computationally advantageous. However, global methods are more robust to noise and achieve globally coherent embeddings, in contrast to the local methods which can sometimes lead to excessively clustered . All spectral techniques are non-parametric in nature and hence do not characterize the map that generates them. Therefore, the computational burden of large spectral decompositions becomes a major drawback when the number of data-points is large. BID3 and BID13 address this issue by providing formulas for out-of-sample extensions to the spectral algorithms. However, these interpolating formulations are computationally inefficient and exhibit poor non-local generalization of the manifold BID4 . Multidimensional scaling (henceforth MDS) is a classical problem in geometry processing and data science and a powerful tool for obtaining a global picture of data when only pairwise distances or dissimilarity information is available. The core idea of MDS is to find an embedding X = x 1, x 2, x 3...x N such that the pairwise distances measured in the embedding space are faithful to the desired distances DISPLAYFORM0 i,j=N i,j=1 as much as possible. There are two prominent versions of MDS: Classical Scaling and Least Squares Scaling. Classical Scaling is based on the observation that the double centering of a pairwise squared distance matrix gives an inner-product matrix which can be factored to obtain the desired embedding. Therefore, if DISPLAYFORM1 T is the centering matrix, classical scaling minimizes the Strain of the embedding configuration X given by DISPLAYFORM2 and is computed conveniently using the eigen-decomposition of the N × N matrix − 1 2 HD s H. At the other end, least squares scaling is based on minimizing the misfits between the pairwise distances of DISPLAYFORM3 i,j=N i,j=1 measured by the Stress function DISPLAYFORM4 In the context of manifold learning and dimensionality reduction, the MDS framework is enabled by estimating all pairwise geodesic distances with a shortest path algorithm like and using the minimizers of Equations 1 and 2 to generate global embeddings by preserving metric properties of the manifold. FORMULA2 and BID5, that used spectral representations of the Laplace-Beltrami Operator of the manifold. BID24 shows a least squares scaling technique which can overcome holes and non-convex boundaries of the manifold. All these algorithms are non-parametric and few out-of-sample extensions have been suggested BID13 to generalize them to new samples. Examining the ability of neural networks to represent data manifolds has received considerable interest and has been studied from multiple perspectives. From the viewpoint of unsupervised parametric manifold learning, one notable approach is based on the metric-learning arrangement of the Siamese configuration BID18 BID6; BID9. Similarly, the parametric version of the Stochastic Neighborhood Embedding van der is another example of using a neural network to generate a parametric map that is trained to preserve local structure BID21. However, these techniques demand an extensive training effort requiring large number of training examples in order to generate satisfactory embeddings. BID30 use a parametric network to learn a classifier which enforces a manifold criterion, requiring nearby points to have similar representations. BID1 have argued that neural-networks can efficiently represent manifolds as monotonic chain of linear segments by providing an architectural construction and analysis. However, they do not address the manifold learning problem and their experiments are based on supervised settings where the ground-truth embedding is known a priori. BID17; BID22 BID10 use neural networks specifically for solving the out-of-sample extension problem for manifold learning. However, their procedure involves training a network to follow a pre-computed non-parametric embedding rather than adopting an entirely unsupervised approach thereby inheriting some of the deficiencies of the non-parametric methods. It is advantageous to adopt a parametric approach to non-linear dimensionality reduction and replace the computational block of the matrix construction and eigenvalue decomposition with a straightforward parametric computation. Directly characterizing the non-linear map provides a simple out-ofsample extension which is a plain forward pass of the network. More importantly, it is expected that the reduced and tightly controlled parameters would lead to an improved non-local generalization of the manifold BID4.In BID18 (henceforth DrLim), it was proposed to use Siamese Networks for manifold learning using the popular hinge-embedding criterion (Equation FORMULA7) as a loss function. A Siamese configuration (Figure 2) comprises of two identical networks that process two separate units of data to achieve output pairs that are compared in a loss function. The contrastive training comprises of DISPLAYFORM0 Figure 2: Siamese Configuration constructing pairs {X DISPLAYFORM1 where λ (k) ∈ {0, 1} is a label for indicating a positive (a neighbor) or negative pair (not a neighbor) by building a nearest neighbor graph from the manifold data. DISPLAYFORM2 Training with the loss in Equation means that at any given update step, a negative pair contributes to the training only when their pairwise distance is less than µ. This leads to a hard-negative sampling problem where the quality of embedding depends on the selection of negative examples in order to prevent excessive clustering of the neighbors. This typically requires an extensive training effort with a huge amount of training data (30000 positive and approximately 17.9 million negatives for a total of 6000 data samples as reported in BID18). We propose to incorporate the ideas of Least Squares Scaling into the computational infrastructure of the Siamese configuration as shown in Figure 2. For every k th pair, we estimate the geodesic distance using a shortest path algorithm and train the network to preserve these distances by minimizing the Stress function DISPLAYFORM0 The advantage of adopting the loss in Equation over Equation FORMULA7 is that every pair of training data contributes to the learning process thereby eliminating the negative sampling problem. More importantly, it facilitates the use of efficient manifold sampling techniques like the Farthest Point Sampling strategy that make it possible to train with much fewer pairs of examples. The farthest point sampling strategy BID7 (also referred to as the MinMax strategy in BID13) is a method to pick landmarks amongst the points of a discretely sampled manifold such that under certain conditions, these samples uniformly cover the manifold much as possible. Starting from a random selection, the landmarks are chosen one at a time such that, each new selection from the unused samples has the largest geodesic distance to the set of the selected sample points. FIG1 provides a visualization of this sampling mechanism. We train the network by minimizing the loss in Equation by computing the pairwise geodesic distances of the landmarks. Therefore, the pre-training computational effort is confined to computing the pairwise geodesic distances of only the landmark points. The proposed geometric manifold learning algorithm can be summarized in two steps, Step1: Compute the nearest-neighbor graph from the manifold data and obtain a set of landmarkpoints and their corresponding pairwise graph/geodesic distances using Dijkstra's algorithm and the Farthest Point Strategy. Step2: Form a dataset of landmark pairs with corresponding geodesic distances {X DISPLAYFORM1} and train network in Siamese configuration using the least-squares MDS loss in Equation FORMULA8 4 EXPERIMENTS 4.1 3D POINT CLOUD DATA Our first set of experiments are based on point-cloud manifolds, e.g., like the Swiss Roll, S-Curve and the Helical Ribbon. We use a multilayer perceptron (MLP) with the PReLU as the non-linear activation function, given by DISPLAYFORM2 where a is a learnable parameter. The networks are trained using the ADAM optimizer with constants (β 1, β 2) = (0.95, 0.99) and a learning rate of 0.01 for 1000 iterations. We run each optimization 5 times with random initialization to ensure convergence. All experiments are implemented in python using the PyTorch framework BID23. We used the scikit-learn machine learning library for the nearest-neighbor and scipy-sparse for Dijkstra's shortest path algorithms. FIG1 show the of our method on the Helical Ribbon and S-Curve respectively with varying number of training samples (in black) out of a total of 8172 data points. The number of landmarks dictates the approximation quality of the low-dimensional embedding generated by the network. Training with too few samples will in inadequate generalization which can be inferred from the corrugations of the unfurled in the first two parts of FIG1 and increasing the number of landmarks expectedly improves the quality of the embedding. We compute the Stress function (Equation FORMULA4) of the entire point configuration to measure the quality of the MDS fit. FIG2 shows the decay in the Stress as a function of the number of training points (or Landmarks) of a 2-Layer MLP.The natural next question to ask is how many landmarks? how many layers? and how many hidden nodes per layer? We observe that these questions relate to an analogous setup in numerical methods for differential equations. For a given numerical technique, the accuracy of the solution depends on the resolution of the spatial grid over which the solution is estimated. Therefore, numerical methods are ranked by an assessment of the order of accuracy their solutions observe. This can be obtained by assuming that the relationship between the approximation error E and the resolution of the grid h is given by DISPLAYFORM3 where P is the order of accuracy of the technique and C is some constant. Therefore, P is obtained by computing the slope of the line obtained by charting log(E) vs log(h), log(E) = log(C) + P log(h).We extend the same principle to evaluate network architectures (in place of numerical algorithms) for estimating the quality of isometric maps. We use the generalized Stress in Equation as the error function for Equation. We assume that due to the 2-approximate property of the farthest point strategy BID19 the sampling is approximately uniform and hence h ∝ DISPLAYFORM4 where K is the number of landmarks. By varying the number of layers and the number of nodes per layer we associate an order of accuracy to each architecture using Equation FORMULA12. FIG2 shows the of our experiment. It shows that a single layer MLP has the capacity for modeling functions upto the first order of accuracy. Adding an additional layer increases the representational power by moving to a second order . Adding more layers does not provide any substantive gain arguably due to a larger likelihood of over-fitting as seen in the considerably noisier estimates (in green). Therefore, a two layer MLP with 70 hidden nodes per layer can be construed as a good architecture for approximating the isometric map of the S-Curve of FIG1 with 200 landmarks. We extend the parametric MDS framework to image articulation manifolds where each sample point is a binary image governed by the modulation of a few parameters. We specifically deal with image manifolds that are isometric to Euclidean space BID16, that is, the geodesic distance between any two sample points is equal to the euclidean distance between their articulation parameters. In the context of the main discussion of this paper, which is metric preserving properties of manifolds, we find that such datasets provide an appropriate test-bed for evaluating metric preserving algorithms. We construct a horizon articulation manifold where each image contains two distinct regions separated by a horizon which is modulated by a linear combination of two fixed sinusoidal basis elements. See Figure 5. DISPLAYFORM0 Thus, each sample has an intrinsic dimensionality of two -the articulation parameters (α 1, α 2) which govern how the sinusoids representing the horizon are mixed. We sample the articulation The proposed method shows maximum fidelity to the ground truth shown in Figure 5 (b) Visualizing the outputs of some filters trained using our method. The 1 st column shows the input images. The filters act to exaggerate (2 nd column) and suppress (3 rd column) a governing frequency in Equation.parameters from a 2D uniform distribution DISPLAYFORM1 We generate 1000 images of the horizon articulation manifold of size 100 × 100. The network architecture comprises of two convolution layers each with kernel sizes 12 and 9, number of kernels 15 and 2 respectively along with a stride of 3 and followed by a fully connected layer mapping the image to a two dimensional entity. We train using the ADAM optimizer BID20 with a learning rate of 0.01 and parameters (β 1, β 2) = (0.95, 0.99). We train the network using 50 Landmark points. Figure 6a shows the comparison between our method and other non-parametric counterparts along with the parametric approach of DrLim. Training with the least squares loss of Equation FORMULA8 shows high fidelity to the ground truth of Figure 5. Except for Isomap and our proposed method, all other methods show some form of distortion indicating a suboptimal metric preservation property. Training with the articulation manifold in Figure5 provides an opportunity to get more detail in understanding the parametric action of the neural networks. The 2 nd and 3 rd columns in Figure 6b show the outputs of some of the filters in the first layer of the architecture trained on the manifold in Figure 5. The distance preserving loss facilitates the learning of image filters which separate the underlying frequencies governing the non-linear manifold, thereby providing a visual validation of the parametric map. We compare our parametric multi-dimensional scaling approach to its direct non-parametric competitor: Landmark-Isomap De BID13. The main idea of Landmark-Isomap is to perform classical scaling on the inter-geodesic distance matrix of only the landmarks and to estimate the embeddings of the remaining points using an interpolating formula (also mentioned in BID3). The formula uses the estimated geodesic distance of each new point to the selected Landmarks in order to estimate its low dimensional embedding. We use the image articulation manifold dataset to provide a quantitative and visual comparison between the two methods. Both the methods are imputed with the same set of landmarks for evaluation. In the first experiment, we generate two independent horizon articulation datasets each containing 1000 samples generated using Equations FORMULA14 and FORMULA15 for training and testing. We then successively train both algorithms on the training dataset with varying number of Landmark points and then use the examples from the test data-set to evaluate performance. FIG4 (a) shows that low dimensional embedding of Landmark-Isomap admits smaller stress values (hence better metric preservation) for the training dataset, but behaves poorly on unseen examples. On the other hand, despite larger stress values for training, the network shows better generalizability. In order to visualize non-local generalization properties, we repeat the previous experiment with a minor modification. We train both algorithms on horizon articulation manifolds with parameters sampled from 0, 0.75] ) and visualize the outputs on test datasets with parameters sampled from DISPLAYFORM0 DISPLAYFORM1 thereby isolating a part of the manifold during training. As shown in FIG4, the output of Landmark-Isomap shows a clustered due to the lack of non-local data in the geodesic distance calculations for the interpolation. In contrast, the network clearly shows a better generalization property. Finally we test our method on a more realistic dataset where the constraint of being isometric to a low dimensional euclidean space is not necessarily strict. We generate 1369 images obtained by smoothly varying the azimuth and elevation of the camera. = 936396 pairs. Our integrated approach yields an improved in a considerably smaller training time (geodesic distances between only 600 2 = 179700 pairs). We used the same 600 landmarks from Isomap and the same architecture of DrLim for generating the embedding in Figure 8c. In the interest of obtaining a better understanding of neural network behavior, we advocate using learning methodologies for solving geometric problems with data by allowing a limited infusion of axiomatic computation to the learning process. In this paper we demonstrate such a scheme by combining parametric modeling with neural networks and the geometric framework of multidimensional scaling. The of this union leads to reduction in training effort and improved local and nonlocal generalization abilities. As future work, we intend to further explore methods that leverage learning methodologies for improving the largely axiomatic setups of numerical algorithms.

Represent the short summary for finding the corresponding scientific article (long)

Parametric Manifold Learning with Neural Networks in a Geometric Framework

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose nearest neighbor overlap, a procedure which quantifies similarity between embedders in a task-agnostic manner, and use it to compare 21 sentence embedders.