Question: Represent the article for finding a single-sentence TLDR

Question: In many applications labeled data is not readily available, and needs to be collected via pain-staking human supervision. We propose a rule-exemplar model for collecting human supervision to combine the scalability of rules with the quality of instance labels. The supervision is coupled such that it is both natural for humans and synergistic for learning. We propose a training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables. Empirical evaluation on five different tasks shows that our algorithm is more accurate than several existing methods of learning from a mix of clean and noisy supervision, and the coupled rule-exemplar supervision is effective in denoising rules. With the ever-increasing reach of machine learning, a common hurdle to new adoptions is the lack of labeled data and the pain-staking process involved in collecting it via human supervision. Over the years, several strategies have evolved for reducing the tedium of collecting human supervision. On the one hand are methods like active learning and crowd-consensus learning that seek to reduce the cost of supervision in the form of per-instance labels. On the other hand is the rich history of rule-based methods where humans code-up their supervision as labeling rules. There is growing interest in learning from such scalable, albiet noisy, supervision (; ; ; ;). However, clean task-specific instance labels continue to be critical for reliable even when fine-tuning models pre-trained on indirect supervision . In this paper we propose a unique blend of cheap coarse-grained supervision in the form of rules and expensive fine-grained supervision in the form of labeled instances. Instead of supervising rules and instance labels independently, we propose that each labeling rule be attached with exemplars of where the rule correctly'fires'. Thus, the rule can be treated as a noisy generalization of those exemplars. Often rules are coded up only after inspecting data. As a human inspects instances, he labels them, and then generalizes them to rules. Thus, humans provide paired supervision of rules and exemplars demonstrating correct deployment of that rule. We explain further with two illustrative applications. Our examples below are from the text domain because rules have been traditionally used in many NLP tasks, but our learning algorithm is agnostic to how rules are expressed. Sentiment Classification Consider an instance I highly recommend this modest priced cellular phone that a human inspects for a sentiment labeling task. After labeling it as positive, he can easily generalize it to a rule Contains'highly recommend' → positive label. This rule generalizes to several more instances, thereby eliminating the need of per-instance labeling on those. However, the label assigned by this rule on unseen instances may not be as reliable as the explicit label on this specific exemplar it generalized. For example, it misfires on I would highly recommend this phone if it weren't for their poor service. Slot-filling Consider a slot-filling task on restaurant reviews over labels like cuisine, location, and time. When an annotator sees an instance like: what chinese restaurants in this city have good reviews?, after labeling token chinese as cuisine, he generalizes it to a rule: (. * ese|. * ian|mexican) restaurants → (cuisine) restaurants. This rule matches hundreds of instances in the unlabeled set, but could wrongly label a phrase like these restaurants. We present in Section 3 other applications where such supervision is natural. Our focus in this paper is developing algorithms for training models under such coupled rule-exemplar supervision. Our main challenge is that the labels induced by the rules are more noisy than instance-level supervised labels because humans tend to over generalize as we saw in the illustrations above. Learning with noisy labels with or without additional clean data has been a problem of long-standing interest in ML (; ; b; ;). However, we seek to design algorithms that better capture rule-specific noise with the help of exemplars around which we have supervision that the rule fired correctly. We associate a latent random variable on whether a rule correctly'covers' an instance, and jointly learn the distribution among the label and all cover variables. This way we simultaneously train the classifier with corrected rule-label examples, and restrict over-generalized rules. In summary our contributions in this paper are as follows: Our contributions We propose the paradigm of supervision in the form of rules generalizing labeled exemplars that is natural in several applications. We design a training method that simultaneously denoises over-generalized rules via latent coverage variables, and trains a classification model with a soft implication loss that we introduce. Through experiments on five tasks spanning question classification, spam detection, sequence labeling, and record classification we show that our proposed paradigm of supervision enables an effective synergy between rule-level and instance-level supervision. We compare our algorithm to several recent frameworks for learning with noisy supervision and constraints, and show much better with our method. We first formally describe the problem of learning from rules generalizing examplars on a classification task. Let X denote the space of instances and Y = {1, . . ., K} denote the space of class labels. Let the set of labeled examples be L = {(x 1, 1, e 1),..., (x n, n, e n)} where x i ∈ X is an instance, i ∈ Y is its user-provided label, and e i ∈ {R 1, . . ., R m, ∅} denotes that x i is an exemplar for rule e i. Some labeled instances may not be generalized to rules and for them e i = ∅. Also, a rule can have more than one exemplar associated with it. Each rule R j could be a blackbox function R j: x → {j, ∅} that takes as input an instance x ∈ X and assigns it either label j or no-label. When the ith labeled instance is an exemplar for rule R j (that is, e i = R j), the label of the instance i should be j. Additionally, we have a different set of unlabeled instances U = {x n+1, . . ., x N}. The cover set H j of rule R j is the set of all instances in U ∪ L for which R j assigns a noisy label j. An instance may be covered by more than one rule or no rule at all, and the labels provided by these rules may be conflicting. Our goal is to train a classification model P θ (y|x) using L and U to maximize accuracy on unseen test instances. A baseline solution is to use R j to noisily label the covered U instances using majority or other consensus method of resolving conflicts. We then train P θ (y|x) on the noisy labels using existing algorithms for learning from noisy and clean labels (; b). However, we expect to be able to do better by learning the systematic pattern of noise in rules along with the classifier P θ (y|x). Our noise model on R j A basic premise of our learning paradigm is that the noise induced by a rule R j is due to over-generalizing the exemplar(s) seen when creating the rule. And, there exists a smaller neighborhood closer to the exemplar(s) where the noise is zero. We model this phenomenon by associating a latent Bernoulli random variable r ji for each instance x i in the stated cover set H j of each rule R j. When r ji = 1, rule R j has not over-generalized on x i, and there is no noise in the label j that R j assigns to x i. When r ji = 0 we flag an over-generalization, and abstain from labeling x i as j suspecting it to be too noisy. We call r ji s as the latent coverage variables. We propose to learn the distribution of r j using another network with parameters φ that outputs the probability P jφ (r j |x) that r j = 1. We then seek to jointly learn P θ (y|x) and P jφ (r j |x) to model the distribution over the true label y and true coverage r j for each rule j and each x in H j. Thus P jφ plays the role of restricting a rule R j so that r j is not necessarily 1 for all instances in its cover set H j An example We make our discussion concrete with an example. Figure 1 shows a two-dimensional X space with labeled points L denoted as red crosses and blue circles and unlabeled points as dots, and the true labels as color of the region. We show two rule-exemplar pairs: (x 1, y 1 = red, R 1), (x 2, y 2 = blue, R 2). Clearly, both rules R 1, R 2 have over-generalized to the wrong region. If we train a classifier with many examples in H 1 ∪ H 2 wrongly labeled by rules, then even with a noise tolerant loss function like , the classifier P θ (y|x) might be misled. In contrast, what we hope to achieve is to learn the P jφ (r j |x) distribution using the limited labeled data and the overlap among the rules such that Pr(r j |x) predicts a value of 0 for examples wrongly covered. Such examples are then excluded from training P θ. The dashed boundaries indicate the revised boundaries of R j s that we can hope to learn based on consensus on the labeled data and the set of rules. Even after such restriction, R j s are useful for training the classifier because of the unlabeled points inside the dashed regions that get added to the labeled set. 2.1 HOW WE JOINTLY LEARN P θ AND P jφ In general we will be provided with several rules with arbitrary overlap in the set of labeled L and unlabeled examples U that they cover. Intuitively, we want the label distribution P θ (y|x) to correctly restrict the coverage distribution P jφ (r j |x), which in turn can provide clean labels to instances in U that can be used to train P θ (y|x). We have two types of supervision in our setting. First, individually for each of the networks we have ground truth values of y and r j for some instances. For the P θ (y|x) distribution, supervision on y is provided by the human labeled data L, and we use these to define the usual log-likelihood as one term in our training objective: For learning the distribution P jφ (r j |x) over the coverage variables, the only sure-shot labeled data is that r ji = 1 for any x i that is an exemplar of rule R j and r ji = 0 for any x i ∈ H j whose label i is different from j. For other labeled instances x i covered with rules R j with agreeing labels, that is i = j we do not strictly require that r ji = 1. In the example above the corrected dashed red boundary excludes a red labeled point to reduce its noise on other points. However, if the number of labeled exemplars are too few, we regularize the networks towards more rule firings, by adding a noise tolerant r ji = 1 loss on the instances with agreeing labels. We use the generalized cross entropy loss of. Note for other instances x i in R j's cover H j, value of r ji is unknown and latent. The second type of supervision is on the relationship between r ji and y i for each Figure 2: Negative implication loss A rule R j imposes a causal constraint that when r ji = 1, the label y i has to be j. r ji = 1 =⇒ y i = j ∀x i ∈ H j We convert this hard constraint into a (log) probability of the constraint being satisfied under the P θ (y|x) and P jφ (r j |x) distributions as: Figure 2 shows a surface plot of the above log probability as a function of P θ (j |x) (shown as axis P(y) in figure) and P jφ (r j = 1|x) (shown Table 1 : Statistics of datasets and their rules. %Cover is fraction of instances in U covered by at least one rule. Precision refers to micro precision of rules. Conflict denotes the fraction of instances covered by conflicting rules among all the covered instances. Avg |Hj| is average cover size of a rule in U . Rules Per Instance is average number of rules covering an instance in U . as axis P(r) in figure) for a single rule. Observe that likelihood drops sharply as P (r j |x) is close to 1 but P (y = j |x) is close to zero. For all other values of these probabilities the log-likelihood is flat and close to zero. Specifically, when P jφ predicts low values of r j for a x, the P θ (y|x) surface is flat, effectively withdrawing the (x, j) supervision from training the classifier P θ. Thus maximizing this likelihood provides a soft enforcement of the constraint without any other unwanted biases. We call this the negative implication loss. We do not need to explicitly model the conflict among rules, that is when an x i is covered by two rules R j and R k of differing labels (j = k), then both r ji and r ki cannot be 1. This is because the constraint among pairs (y i, r ji) and (y i, r ki) as stated in Equation 3 subsumes this one. During training we then seek to maximize the log of the above probability along with normal data likelihood terms. Putting the terms in Equations 1, 2 and 4 together our final training objective is: We refer to our training loss as a denoised rule-label implication loss or ImplyLoss for short. The LL(φ) term seeks to denoise rule coverage which then influence the y distribution via the implication loss. We explored several other methods of enforcing the constraint among y and r j in the training of the P θ and P jφ networks. Our method ImplyLoss consistently performed the best among several methods we tried including the recent posterior regularization method of enforcing soft constraints and co-training . Network Architecture Our network has three modules. A shared embedding layer that provides the feature representation of the input. When labeled data is scarce, this will typically be a pre-trained layer from a related task. We describe the embedding module for each task individually in the experiments section. A classification network that models P θ (y|x) with parameters θ. The embedding of an input x is passed through multiple non-linear layers with ReLU activation, a last linear layer followed by Softmax to output a distribution over the class labels. A rule network that models P jφ (r j = 1|x) whose parameters φ are shared across all rules. The input to the network is rule-specific and concatenates the embedding of the input instance x, and a a one-hot encoding of the rule id'j'. The inputs are transformed through multiple layers of ReLU before passing through a Sigmoid activation which outputs the probability P jφ (r j = 1|x). We compare our training algorithms against simple baselines, existing error-tolerant learning algorithms, and existing constraint-based learning in deep networks. We evaluate across five datasets spanning three task types: text classification, sequence labeling, and record classification. We augment the datasets with rules, that we obtained manually in three cases, from pre-existing public sources in one case, and automatically in another. Table 1 presents statistics summarizing the datasets and rules. A brief description of each appears below. Question Classification : This is a TREC-6 dataset to classify a question to one of six categories: {Abbreviation, Entity, Description, Human, Location, Numeric-value}. The training set has 5452 instances which are split as 68 for L, 500 for validation, and the remaining as U. Each example in L is generalized as a rule represented by a regular expression. E.g. After labeling How do you throw a housewarming party? as Description we define a rule (how|How|what|What)(does|do|to|can). * → Description. More rules in Table 4 of supplementary. Although, creating such 68 generalised rules required 90 minutes, the generalizations cover 4637 instances in U, almost two orders of magnitude more instances than in L! On an average each of our rule covered 124 instances (|H j | column in Table 1). But the precision of labels assigned by rules was only 63.8%, and 22.5% of covered instances had an inter-rule conflict. This clearly demonstrates the noise in the rule labelings. Accuracy is used as the performance metric. MIT-R 1 : This is a slot-filling task on sentences about restaurant search and the task is to label each token as one of {Location, Hours, Amenity, Price, Cuisine, Dish, Restaurant Name, Rating, Other}. The training data is randomly split into 200 sentences (1690 tokens) as L, 500 sentences (4k tokens) as validation and remaining 6.9k sentences (64.9k tokens) as U. We manually generalize 15 examples in L. E.g. After inspecting the sentence where can i get the highest rated burger within ten miles and labeling highest rated as Rating, we provide the rule:. * (highly|high|good|top|highest)(rate|rating|rated). * → Rating to the matched positions. More examples in Table 7 of supplementary. Although, creating 15 generalizing rules took 45 minutes of annotator effort, the rules covered roughly 9k tokens in U. F1 metric is used for evaluation on the default test set of 14.2k tokens over 1.5k sentences. : This dataset contains 5.5k text messages labeled as spam/not-spam, out of which 500 were held out for validation and 500 for testing. We manually generalized 69 exemplars to rules. Remaining examples go in the U set. The rules here check for presence of keywords or phrases in the SMS. * guaranteed gift. * → spam. A rule covers 31 examples on an average and has a precision of 97.3%. However, in this case only 40% of the unlabeled set is covered by a rule. We report F1 here since class is skewed. More examples in Table 5 of supplementary. Youtube Spam Classification : Here the task is to classify comments on YouTube videos as Spam or Not-Spam. We obtain this from Snorkel's Github page 2, which provides 10 labeling functions which we use as rules, an unlabeled train set which we use as U, a labeled dev set to guide the creation of their labeling functions which we use as L, and labeled test and validation sets which we use in the same roles. Their labeling functions have a large coverage (258 on average), and a precision of 78.6%. Census Income : This UCI dataset is extracted from the 1994 U.S. census. It lists a total of 13 features of an individual such as age, education level, marital status, country of origin etc. The primary task on it is binary classification -whether a person earns more than $50K or not. The train data consists of 32563 records. We choose 83 random data points as L, 10k points as U and 5561 points as validation data. For this case we created the rules synthetically as follows: We hold out disjoint 16k random points from the training dataset as a proxy for human knowledge and extract a PART decision list from it as our set of rules. We retain only those rules which fire on L. Network Architecture Since our labeled data is small we depend on pre-trained resources. As the embedding layer we use a pretrained ELMO network where 1024 dimensional contextual token embeddings serve as representations of tokens in the MIT-R sentences, and their average serve as representation for sentences in Question and SMS dataset. Parameters of the embedding network are held fixed during training. For sentences in the YouTube dataset, we use Snorkel's 2 architecture of a simple bag-of-words feature representation marking the frequent uni-grams and bi-grams present in a sentence using a few-hot vector. For the Census dataset categorical features are represented as one hot vectors, while real valued features are simply normalized. For MIT-R, Question and SMS both classification and rule-weight network contain two 512 dimensional hidden layers with ReLU activation. For Census, both the networks contain two 256 dimensional hidden layers with ReLU activation. For YouTube, the classifier network is a simple logistic regression like in Snorkel's code. The rule network has one 32-dimensional hidden layer with ReLU activation. Each reported number is obtained by averaging over five random initializations. Whenever a method involved hyper-parameters to weigh the relative contribution of various terms in the objective, we used a validation dataset to tune the value of the hyper-parameter. Hyperparameters used are provided in Section C of supplementary. In Table 2 we compare our method with the following alternatives on each of the five datasets: Majority: that predicts via majority vote among the rules that cover an instance. This baseline indicates the stand-alone quality of rules, no network is learned here. Ties are broken arbitrarily for class-balanced datasets or by using a default class. Table 2, shows that the accuracy of majority is quite poor indicating either poor precision or poor coverage of the rule sets 3. Only-L: Here we train the classifier P θ (y|x) only on the labeled data L using the standard crossentropy loss (Equation 1). Rule generalisations are not utilized at all in this case. We observe in Table 2 that even with the really small labeled set we used for each dataset, the accuracy of a classifier learned with clean labeled data is much higher than noisy majority labels of rules. We consider this method as our baseline and report the gains on remaining methods. L+Umaj: Next we train the classifier on L along with U maj obtained by labeling instances in U with the majority label among the rules applicable to the instance. The row corresponding to L+Umaj in Table 2 provides the gains of this method over Only-L. We observe gains with the noisily labeled U in four out of the five cases. Noise-tolerant: Since labels in U maj are noisy, we next use's noise tolerant generalized cross entropy loss on them with regular cross-entropy loss on the clean L as follows: Parameter q ∈ controls the noise tolerance which we tune as a hyper-parameter. We observe that in all cases the above objective improves beyond Only-L validating that noise-tolerant loss functions are useful for learning from noisy labels on U maj. Learning to Reweight (L2R) (b): is a recent method for training with a mix of clean and noisy labeled data. They train the classifier by meta-learning to re-weight the loss on the noisily labelled instances (U maj) with the help of the clean examples (L). This method shows huge variance in its accuracy gains over Only-L across datasets and is worse in two of the cases. All the above methods employ no extra parameters to denoise or weight individual rules. We next compare with a number of methods that do. L+Usnorkel: This method replaces Majority-based consensus with Snorkel's generative model that assigns weights to rules and labels examples in U. Thereafter we use the same approach as in L+Umaj with just Snorkel's soft-labels instead of Majority on U. The are mixed and we do not get any consistent gains over Only-L and over L+Umaj. We also compare with using noise-tolerant loss on U labeled by Snorkel (Eqn:6) which we call SnorkelNoise-Tolerant. We observe more consistent improvements then, but these are not much better than Noise-Tolerant on U maj. We next compared with a method that simultaneously learns two sets of networks P θ and P jφ like in ours but with different loss function and training schedule. Posterior Regularization (PR): This method proposed in also treats rules as softconstraints and has been used for training neural networks for structured outputs. They use's posterior regularization framework to train the two networks in a teacher-student setup. We adapt the same framework and get a procedure as follows: The student proposes a distribution over y and r j s using current P θ and P jφ, the teacher uses the constraint in Eq 3 to revise the distributions so as to minimize the probability of violations, the student updates parameters θ and φ to minimize KL distance with the revised distribution. The detailed formulation appear in the Section A of supplementary. We find that this method is worse than Only-L in two cases and worse than the noise-tolerant method that does not train extra φ parameters. Overall our approach of training with denoised rule-label implication loss provides much better accuracy than all the above eight methods and we get consistent gains over Only-L on all datasets. On the question dataset we get 11.9 points gains over Only-L whereas the best gains by existing method was 0.5. A useful property of our method compared to the PR method above is that the training process is simple and fits into the batch stochastic gradient training template. In contrast, PR requires special alternating computations. We next perform a number of diagnostics experiments to explain the reasons for the superior performance of our method. Diagnostics: Effectiveness of learning true coverage via P jφ An important part of our method is the rule-specific denoising learned via the P jφ network. In the chart alongside we plot the original precision of each rule on the test data, and the precision after suppressing those rule labelings where P jφ (r j |x) predicts 0 instead of 1. Observe now that the precision is more than 91% on all datasets. For the Question dataset, the precision jumped from 64% to 98%. The percentage of labelings suppressed (shown by the dashed line) is higher on datasets with noisier rules (e.g. compare Question and SMS). This shows that P jφ is able to denoise rules by capturing the distribution of the latent true coverage variables with the limited LL(φ) loss and indirectly via the implication loss. We next evaluate the importance of the exemplar-rule pairs in learning the P jφ and P θ networks. The exemplars of a rule give an interesting new form of supervision about an instance where a labeling rule must fire. To evaluate the importance of this supervision, we exclude the r j = 1 likelihood on rule-exemplar pairs from LL(φ), that is, the first term in Equation 2 is dropped. In the table below we see that performance of ImplyLoss drops when the exemplar-rule supervision is removed. Interestingly, even after this drop, the performance of ImplyLoss surpasses most of the methods in Table 2 indicating that even without exemplar-rule pairs our training objective is effective at learning from rules and labeled instances. Effect of increasing labeled data L We increase L while keeping the number of rules fixed on the Question dataset. In the attached plot we see the accuracy of our method (ImplyLoss) against Only-L and Posterior Reg. We observe the expected trend that the gap between the method narrows as labeled data increases. Learning from noisily labeled data has been extensively studied in settings like crowdsourcing. One category of these algorithms upper-bound the loss function to make it robust to noise. These include methods like MAE , Generalized Cross Entropy (CE) , and Ramp loss . Most of these assume that noise is independent of the input given the true label. In our model noise is systematic and instance-dependent. A second category assume that a small clean dataset is available along with noisily labeled data. This is also true in our case, and we compared with a state of the art method in that category Ren et al. (2018b) that chooses a descent direction that aligns with a clean validation set using meta-learning. Others in this category include:'s method of iteratively selecting examples with smallest loss, and's method of learning a separate network to transform noisy labels to cleaned ones which are used to impose a cross-entropy loss on P θ (y|x). In contrast, we perform rule-specific cleaning via latent coverage variables and a flexible implication loss which withdraws y supervision when P jφ (r ji |x) assumes low values. Another way of relating clean and noisy labels is via an instance-independent confusion matrix learned jointly with the classifier (; ; b; a). These works assume that the confusion matrix is instance independent, which does not hold for our case. uses confidence from the classifier to eliminate noise but they need to ensure that the network does not memorize noise. Our learning setup also has the advantage of extracting confidence from a different network. There is growing interest in integrating logical rules with labeled examples for training networks, specifically for structured outputs (; ; ; ; a).; convert rules on output nodes of network, to (almost differentiable) loss functions during training. The primary difference of these methods from ours is that they assume that rules are correct whereas we assume them to be noisy. Accordingly, we simultaneously correct the rules and use them to improve the classifier, whereas they use the rules as-is to train the network outputs. A well-known framework for working with soft rules is posterior regularization which is used in to train deep structured output networks while harnessing logic rules. works only with noisy rules treating them as black-box labeling functions and assigns a linear weight to each rule based on an agreement objective. Our learning model is more powerful that attempts to learn a non-linear network to restrict rule boundaries rather than just weight their outputs. We presented a comparison with both these approaches in the experimental section, and showed superior performance. To the best of our knowledge, our proposed paradigm of coupled rule-exemplar supervision is novel, and our proposed training algorithm is able to harness them in ways not possible by existing frameworks for learning from rules or noisy supervision. We proposed a new rule-exemplar model for collecting human supervision to combine the scalability of top-level rules with the quality of instance-level labels. We show that such supervision is natural since humans typically inspect examples to code rules. Furthermore, such coupled examples provide supervision on correct firing of rules which help to denoise rules. We propose to train the classifier while jointly denoising rules via latent coverage variables imposing a soft-implication constraint on the true label. Empirically on five datasets we show that our training algorithm that performs rule-specific denoising is better than generic noise-tolerant learning. In future we plan to deploy this framework on other applications where human supervision is a scarce resource. We model a joint distribution Q(y, r 1, . . ., r n |x) to capture the interaction among the label random variable y and coverage random variables r 1,..., r n of any instance x. We use r to compactly represent r 1,..., r n. Strictly speaking, when a rule R j does not cover x, the r j is not a random variable and its value is pinned to 0 but we use this fixed-tuple notation for clarity. The random variables r j and y impose a constraint on the joint distribution Q: for a x ∈ H j when r j = 1, the label y cannot be anything other than j. r j = 1 =⇒ y = j ∀x ∈ H j We can convert this into a soft constraint on the marginals of the distribution Q by stating the probability of y = j Q(y, r j = 1|x) should be small. The singleton marginals of Q along the y and r j variables are tied to the P θ and P jφ (r j |x) we seek to learn. A network with parameters θ models the classifier P θ (y|x), and a separate network with φ variables (shared across all rules) learns the P jφ (r j |x) distribution. The marginals of joint Q should match these trained marginals and we use a KL term for that: We call the combined KL term succinctly as KL(Q, P θ) + KL(Q, P φ). Further the P θ and P jφ distributions should maximize the log-likelihood on their respective labeled data as provided in Equation 1 and Equation 2 respectively. Putting all the above objectives together with hyper-parameters α > 0, λ > 0 we get our final objective as: We show in Section A.1 that this gives rise to the solution for Q in terms of P θ, P jφ and alternately for P θ, P jφ in terms of Q as follows. where δ(y = j ∧ r j = 1) is an indicator function that is 1 when the constraint inside holds, else it is 0. Computing marginals of the above using straight-forward message passing techniques we get: Thereafter, we solve for θ and φ in terms of a given Q as Here, γ = 1 α. This gives rise to an alternating optimization algorithm as in the posterior regularization framework of. We initialize θ and φ randomly. Then in a loop, we perform the following two steps alternatively much like the EM algorithm . Here we compute marginals Q(y|x) and Q(r j |x) from current P θ and P jφ using Equations 12 and 13 respectively for each x in a batch. This computation is straight-forward and does not require any neural optimization. We can interpret the Q(y|x) as a small correction of the P θ (y|x) so as to align better with the constraints imposed by the rules in Equation 3. Likewise Q(r j |x) is an improvement of current P jφ s in the constraint preserving direction. For example, the expected r j values might be reduced for an instance if its probability of y being j is small. Parameter update step: We next reoptimize the θ and φ parameters to match the corrected Q distribution as shown in Equation 14. This is solved using standard stochastic gradient techniques. The Q terms can just be viewed as weights at this stage which multiply the loss or label likelihood. A pseudocode of our overall training algorithm is described in Algorithm 1. Input: L, U Initialize parameters θ, φ randomly for a random training batch from U ∪ L do Obtain P θ (y|x) from the classification network. Obtain P jφ (r j |x) j∈[n] from the rule-weight network. Calculate Q(y|x) using Eqn 12 and Q(r j |x) j∈[n] using Eqn 13. Update θ and φ by taking a step in the direction to minimize the loss in Eqn 14. end for Output: θ, φ Treat each Q(y, r) as an optimization variable with the constraint that y,r Q(y, r) = 1. We express this constraint with a Langrangian multiplier η in the objective. Also, define a distribution It is easy to verify that the KL terms in our objective 10 can be collapsed as KL(Q; P θ,φ). The rewritten objective (call it F (Q, θ, φ) ) is now: Next we solve for ∂F ∂Q(y,r) = 0 after expressing the marginals in their expanded forms: e.g. Q(y, r j |x) = r1,...,rj−1,rj+1,...,rn Q(y, r 1, . . ., r n |x). This gives us ∂F ∂Q(y, r) = log Q(y, r) − log P θ,φ (y, r|x) Equating it to zero and substituting for P θ,φ we get the solution for Q(y, r) in Equation 11. The proof for the optimal P θ and P jφ while keeping Q fixed in Equation 15 is easy and we skip here. We provide a list of rules for each task type. Great News! Call FREEFONE 08006344447 to claim your guaranteedå£1000 CASH orå£2000 gift. cuisine1a= ['italian','american', 'japanese','spanish','mexican', 'chinese','vietnamese','vegan'] cuisine1b= ['bistro','delis'] cuisine2= ['barbecue','halal', 'vegetarian','bakery'] can you find me some chinese food For all the experiments we use a learning rate of 1e-4, batch-size of 32 and a dropout of 0.8 (keep probability). All the models were trained for a maximum of 100 epochs. We use early stopping using a validation set. We provide a list of hyperparameters used in our experiments. Table 9: Meta-learning rate of Learning to Reweight method (L2R) for various datasets

represent the tldr for finding the corresponding scientific article

Examples:

"We study the sparsity-inducing bias of deep models, caused by their learning dynamics." == "A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditions on depth and initialization for which this phenomenon arises in deep linear models. Our main theoretical contribution is a dynamical depth separation , proving that while shallow models can exhibit incremental learning dynamics, they require the initialization to be exponentially small for these dynamics to present themselves. However, once the model becomes deeper, the dependence becomes polynomial and incremental learning can arise in more natural settings. We complement our theoretical findings by experimenting with deep matrix sensing, quadratic neural networks and with binary classification using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning. Neural networks have led to a breakthrough in modern machine learning, allowing us to efficiently learn highly expressive models that still generalize to unseen data. The theoretical reasons for this success are still unclear, as the generalization capabilities of neural networks defy the classic statistical learning theory bounds. Since these bounds, which depend solely on the capacity of the learned model, are unable to account for the success of neural networks, we must examine additional properties of the learning process. One such property is the optimization algorithm -while neural networks can express a multitude of possible ERM solutions for a given training set, gradient-based methods with the right initialization may be implicitly biased towards certain solutions which generalize. A possible way such an implicit bias may present itself, is if gradient-based methods were to search the hypothesis space for possible solutions of gradually increasing complexity. This would suggest that while the hypothesis space itself is extremely complex, our search strategy favors the simplest solutions and thus generalizes. One of the leading along these lines has been by , deriving an analytical solution for the gradient flow dynamics of deep linear networks and showing that for such models, the singular values converge at different rates, with larger values converging first. At the limit of infinitesimal initialization of the deep linear network, show these dynamics exhibit a behavior of "incremental learning" -the singular values of the model are learned separately, one at a time. Our work generalizes these to small but finite initialization scales. Incremental learning dynamics have also been explored in gradient descent applied to matrix completion and sensing with a factorized parameterization (, ,). When initialized with small Gaussian weights and trained with a small learning rate, such a model is able to successfully recover the low-rank matrix which labeled the data, even if the problem is highly over-determined and no additional regularization is applied. In their proof of low-rank recovery for such models, show that the model remains lowrank throughout the optimization process, leading to the successful generalization. explore the dynamics of such models, showing the singular values are learned at different rates and that deeper models exhibit stronger incremental learning dynamics. Our work deals with a more simplified setting, allowing us to determine explicitly under which conditions depth leads to this dynamical phenomenon. Finally, the learning dynamics of nonlinear models have been studied as well. and study the gradient flow dynamics of shallow ReLU networks under restrictive distributional assumptions, show that shallow networks learn functions of gradually increasing frequencies and show how deep ReLU networks correlate with linear classifiers in the early stages of training. These findings, along with others, suggest that the generalization ability of deep networks is at least in part due to the incremental learning dynamics of gradient descent. Following this line of work, we begin by explicitly defining the notion of incremental learning for a toy model which exhibits this sort of behavior. Analyzing the dynamics of the model for gradient flow and gradient descent, we characterize the effect of the model's depth and initialization scale on incremental learning, showing how deeper models allow for incremental learning in larger (realistic) initialization scales. Specifically, we show that a depth-2 model requires exponentially small initialization for incremental learning to occur, while deeper models only require the initialization to be polynomially small. Once incremental learning has been defined and characterized for the toy model, we generalize our theoretically and empirically for larger linear and quadratic models. Examples of incremental learning in these models can be seen in figure 1, which we discuss further in section 4. We begin by analyzing incremental learning for a simple model. This will allow us to gain a clear understanding of the phenomenon and the conditions for it, which we will later be able to apply to a variety of other models in which incremental learning is present. Our simple linear model will be similar to the toy model analyzed by. Our input space will be X = R d and the hypothesis space will be linear models with non-negative weights, such that: We will introduce depth into our model, by parameterizing σ using w ∈ R d ≥0 in the following way: Where N represents the depth of the model. Since we restrict the model to having non-negative weights, this parameterization doesn't change the expressiveness, but it does radically change it's optimization dynamics. Assuming the data is labeled by some σ * ∈ R d ≥0, we will study the dynamics of this model for general N under a depth-normalized 1 squared loss over Gaussian inputs, which will allow us to derive our analytical solution: We will assume that our model is initialized uniformly with a tunable scaling factor, such that: 2.2 GRADIENT FLOW ANALYTICAL SOLUTIONS Analyzing our toy model using gradient flow allows us to obtain an analytical solution for the dynamics of σ(t) along with the dynamics of the loss function for a general N. For brevity, the following theorem refers only to N = 1, 2 and N → ∞, however the solutions for 3 ≤ N < ∞ are similar in structure to N → ∞, but more complicated. We also assume σ * i > 0 for brevity, however we can derive the solutions for σ * i = 0 as well. Note that this is a special case adaptation of the one presented in for deep linear networks: Theorem 1. Minimizing the toy linear model described in with gradient flow over the depth normalized squared loss, with Gaussian inputs and weights initialized as in and assuming σ * i > 0 leads to the following analytical solutions for different values of N: Proof. The gradient flow equations for our model are the following: Given the dynamics of the w parameters, we may use the chain rule to derive the dynamics of the induced model, σ:σ This differential equation is solvable for all N, leading to the solutions in the theorem. Taking, which is also solvable. for σ * i ∈ {12, 6, 4, 3} according to the analytical solutions in theorem 1, under different depths and initializations. The first column has all values converging at the same rate. Notice how the deep parameterization with small initialization leads to distinct phases of learning, where values are learned incrementally (bottom-right). The shallow model's much weaker incremental learning, even at small initialization scales (second column), is explained in theorem 2. Analyzing these solutions, we see how even in such a simple model depth causes different factors of the model to be learned at different rates. Specifically, values corresponding to larger optimal values converge faster, suggesting a form of incremental learning. This is most clear for N = 2 where the solution isn't implicit, but is also the case for N ≥ 3, as we will see in the next subsection. These dynamics are depicted in figure 2, where we see the dynamics of the different values of σ(t) as learning progresses. When N = 1, all values are learned at the same rate regardless of the initialization, while the deeper models are clearly biased towards learning the larger singular values first, especially at small initialization scales. Our model has only one optimal solution due to the population loss, but it is clear how this sort of dynamic can induce sparse solutions -if the model is able to fit the data after a small amount of learning phases, then it's obtained will be sparse. Alternatively, if N = 1, we know that the dynamics will lead to the minimal 2 norm solution which is dense. We explore the sparsity inducing bias of our toy model by comparing it empirically 2 to a greedy sparse approximation algorithm in appendix D, and give our theoretical in the next section. Equipped with analytical solutions for the dynamics of our model for every depth, we turn to study how the depth and initialization effect incremental learning. focuses on incremental learning in depth-2 models at the limit of σ 0 → 0, we will study the phenomenon for a general depth and for σ 0 > 0. First, we will define the notion of incremental learning. Since all values of σ are learned in parallel, we can't expect one value to converge before the other moves at all (which happens for infinitesimal initialization as shown by). We will need a more relaxed definition for incremental learning in finite initialization scales. Definition 1. Given two values σ i, σ j such that σ * i > σ * j > 0 and both are initialized as σ i = σ j = σ 0 < σ * j, and given two scalars s ∈ (0, 1 4) and f ∈ (3 4, 1), we call the learning of the values (s, f)-incremental if there exists a t for which: In words, two values have distinct learning phases if the first almost converges (f ≈ 1) before the second changes by much (s 1). Note that for any N, σ(t) is monotonically increasing and so once σ j (t) = sσ * j, it will not decrease to allow further incremental learning. Given this definition of incremental learning, we turn to study the conditions that facilitate incremental learning in our toy model. Our main is a dynamical depth separation , showing that incremental learning is dependent on Proof sketch (the full proof is given in appendix A). Rewriting the separable differential equation in to calculate the time until σ(t) = ασ *, we get the following: The condition for incremental learning is then the requirement that t f (σ i) ≤ t s (σ j), ing in: We then relax/restrict the above condition to get a necessary/sufficient condition on σ 0, leading to a lower and upper bound on σ th 0. Note that the value determining the condition for incremental learning is -if two values are in the same order of magnitude, then their ratio will be close to 1 and we will need a small initialization to obtain incremental learning. The dependence on the ratio changes with depth, and is exponential for N = 2. This means that incremental learning, while possible for shallow models, is difficult to see in practice. This explains why changing the initialization scale in figure 2 changes the dynamics of the N ≥ 3 models, while not changing the dynamics for N = 2 noticeably. The next theorem extends part of our analysis to gradient descent, a more realistic setting than the infinitesimal learning rate of gradient flow: Theorem 3. Given two values σ i, σ j of a depth-2 toy linear model as in, such that and the model is initialized as in, and given two scalars s ∈ (0, 1 4) and f ∈ (3 4, 1), and assuming σ * j ≥ 2σ 0, and assuming we optimize with gradient descent with a learning rate η ≤ c σ * c < 2(√ 2 − 1) and σ * 1 the largest value of σ *, then the largest initialization value for which the learning phases of the values are (s, f)-incremental, denoted σ th 0, is lower and upper bounded in the following way: Where A and B are defined as: We defer the proof to appendix B. Note that this , while less elegant than the bounds of the gradient flow analysis, is similar in nature. Both A and B simplify to r when we take their first order approximation around c = 0, giving us similar bounds and showing that the condition on σ 0 for N = 2 is exponential in gradient descent as well. While similar gradient descent are harder to obtain for deeper models, we discuss the general effect of depth on the gradient decent dynamics in appendix C. So far, we have only shown interesting properties of incremental learning caused by depth for a toy model. In this section, we will relate several deep models to our toy model and show how incremental learning presents itself in larger models as well. The task of matrix sensing is a generalization of matrix completion, where our input space is X = R d×d and our model is a matrix W ∈ R d×d, such that: , we introduce depth by parameterizing the model using a product of matrices and the following initialization scheme (W i ∈ R d×d): Note that when d = 1, the deep matrix sensing model reduces to our toy model without weight sharing. We study the dynamics of the model under gradient flow over a depth-normalized squared loss, assuming the data is labeled by a matrix sensing model parameterized by a PSD W * ∈ R d×d: The following theorem relates the deep matrix sensing model to our toy model, showing the two have the same dynamical equations: Theorem 4. Optimizing the deep matrix sensing model described in with gradient flow over the depth normalized squared loss , with weights initialized as in leads to the following dynamical equations for different values of N: Where σ i and σ * i are the ith singular values of W and W *, respectively, corresponding to the same singular vector. The proof follows that of and and is deferred to appendix E. Theorem 4 shows us that the bias towards sparse solutions introduced by depth in the toy model is equivalent to the bias for low-rank solutions in the matrix sensing task. This bias was studied in a more general setting in , with empirical supporting the effect of depth on the obtainment of low-rank solutions under a more natural loss and initialization scheme. We recreate and discuss these experiments and their connection to our analysis in appendix E, and an example of these dynamics in deep matrix sensing can also be seen in panel (a) of figure 1. By drawing connections between quadratic networks and matrix sensing (as in), we can extend our to these nonlinear models. We will study a simplified quadratic network, where our input space is X = R d and the first layer is parameterized by a weight matrix W ∈ R d×d and followed by a quadratic activation function. The final layer will be a summation layer. We assume, like before, that the labeling function is a quadratic network parameterized by W * ∈ R d×d. Our model can be written in the following way, using the following orthogonal initialization scheme: Immediately, we see the similarity of the quadratic network to the deep matrix sensing model with N = 2, where the input space is made up of rank-1 matrices. However, the change in input space forces us to optimize over a different loss function to reproduce the same dynamics: Definition 2. Given an input distribution over an input space X with a labeling function y: X → R and a hypothesis h, the variance loss is defined in the following way: Note that minimizing this loss function amounts to minimizing the variance of the error, while the squared loss minimizes the second moment of the error. We note that both loss functions have the same minimum for our problem, and the dynamics of the squared loss can be approximated in certain cases by the dynamics of the variance loss. For a complete discussion of the two losses, including the cases where the two losses have similar dynamics, we refer the reader to appendix F. Theorem 5. Minimizing the quadratic network described and initialized as in with gradient flow over the variance loss defined in leads to the following dynamical equations: Where σ i and σ * i are the ith singular values of W and W *, respectively, corresponding to the same singular vector. We defer the proof to appendix F and note that these dynamics are the same as our depth-2 toy model, showing that shallow quadratic networks can exhibit incremental learning (albeit requiring a small initialization). While incremental learning has been described for deep linear networks in the past, it has been restricted to regression tasks. Here, we illustrate how incremental learning presents itself in binary classification, where implicit bias have so far focused on convergence at t → ∞ (, ,). Deep linear networks with diagonal weight matrices have been shown to be biased towards sparse solutions when N > 1 in , and biased towards the max-margin solution for N = 1. Instead of analyzing convergence at t → ∞, we intend to show that the model favors sparse solutions for the entire duration of optimization, and that this is due to the dynamics of incremental learning. Our theoretical illustration will use our toy model as in (initialized as in) as a special weightshared case of deep networks with diagonal weight matrices, and we will then show empirical for the more general setting. We analyze the optimization dynamics of this model over a separable where y i ∈ {±1}. We use the exponential loss ((f (x), y) = e −yf (x) ) for the theoretical illustration and experiment on the exponential and logistic losses. Computing the gradient for the model over w, the gradient flow dynamics for σ become: We see the same dynamical attenuation of small values of σ that is seen in the regression model, caused by the multiplication by σ. From this, we can expect the same type of incremental learning to occur -weights of σ will be learned incrementally until the dataset can be separated by the current support of σ. Then, the dynamics strengthen the growth of the current support while relatively attenuating that of the other values. Since the data is separated, increasing the values of the current support reduces the loss and the magnitude of subsequent gradients, and so we should expect the support to remain the same and the model to converge to a sparse solution. Granted, the above description is just intuition, but panel (c) of figure 1 shows how it is born out in practice (similar are obtained for the logistic loss). In appendix G we further explore this model, showing deeper networks have a stronger bias for sparsity. We also observe that the initialization scale plays a similar role as before -deep models are less biased towards sparsity when σ 0 is large. In their work, show an equivalence between the diagonal network and the circular-convolutional network in the frequency domain. According to their , we should expect to see the same sparsity-bias of diagonal networks in convolutional networks, when looking at the Fourier coefficients of σ. An example of this can be seen in panel (d) of figure 1, and we refer the reader to appendix G for a full discussion of their convolutional model and it's incremental learning dynamics. Gradient-based optimization for deep linear models has an implicit bias towards simple (sparse) solutions, caused by an incremental search strategy over the hypothesis space. Deeper models have a stronger tendency for incremental learning, exhibiting it in more realistic initialization scales. This dynamical phenomenon exists for the entire optimization process for regression as well as classification tasks, and for many types of models -diagonal networks, convolutional networks, matrix completion and even the nonlinear quadratic network. We believe this kind of dynamical analysis may be able to shed light on the generalization of deeper nonlinear neural networks as well, with shallow quadratic networks being only a first step towards that goal. Proof. Our strategy will be to define the time t α for which a value reaches a fraction α of it's optimal value, and then require that t f (σ i) ≤ t s (σ j). We begin with recalling the differential equation which determines the dynamics of the model:σ Since the solution for N ≥ 3 is implicit and difficult to manage in a general form, we will define t α using the integral of the differential equation. The equation is separable, and under initialization of σ 0 we can describe t α (σ) in the following way: Incremental learning takes place when σ i (t f) = f σ * i happens before σ j (t s) = sσ * j. We can write this condition in the following way: Plugging in σ i = rσ j and rearranging, we get the following necessary and sufficient condition for incremental learning: Our last step before relaxing and restricting our condition will be to split the integral on the left-hand side into two integrals: At this point, we cannot solve this equation and isolate σ 0 to obtain a clear threshold condition on it for incremental learning. Instead, we will relax/restrict the above condition to get a necessary/sufficient condition on σ 0, leading to a lower and upper bound on the threshold value of σ 0. To obtain a sufficient (but not necessary) condition on σ 0, we may make the condition stricter either by increasing the left-hand side or decreasing the right-hand side. We can increase the left-hand side by removing r from the left-most integral's denominator (r > 1) and then combine the left-most and right-most integrals: Next, we note that the integration bounds give us a bound on σ for either integral. This means we can replace 1 − σ σ * j with 1 on the right-hand side, and replace 1 − σ rσ * j with 1 − f on the left-hand side: We may now solve these integrals for every N and isolate σ 0, obtaining the lower bound on σ th 0. We start with the case where N = 2: Rearranging to isolate σ 0, we obtain our : For the N ≥ 3 case, we have the following after solving the integrals: For simplicity we may further restrict the condition by removing the term 1 rf σ * j 1− 2 N. Solving for σ 0 gives us the following: To obtain a necessary (but not sufficient) condition on σ 0, we may relax the condition in either by decreasing the left-hand side or increasing the right-hand side. We begin by rearranging the equation: Like before, we may use the integration bounds to bound σ. Plugging in σ = sσ * j for all integrals decreases the left-hand side and increases the right-hand side, leading us to the following: Rearranging, we get the following inequality: We now solve the integrals for the different cases. For N = 2, we have: Rearranging to isolate σ 0, we get our condition: Finally, for N ≥ 3, we solve the integrals to give us: Rearranging to isolate σ 0, we get our condition: For a given N, we derived a sufficient condition and a necessary condition on σ 0 for (s, f)-incremental learning. The necessary and sufficient condition on σ 0, which is the largest initialization value for which we see incremental learning (denoted σ th 0), is between the two derived bounds. The precise bounds can possibly be improved a bit, but the asymptotic dependence on r is the crux of the matter, showing the dependence on r changes with depth with a substantial difference when we move from shallow models (N = 2) to deeper ones (N ≥ 3) Theorem. Given two values σ i, σ j of a depth-2 toy linear model as in, such that and the model is initialized as in, and given two scalars s ∈ (0, 1 4) and f ∈ (3 4, 1), and assuming σ * j ≥ 2σ 0, and assuming we optimize with gradient descent with a learning rate η ≤ c σ * 1 for c < 2(√ 2 − 1) and σ * 1 the largest value of σ *, then the largest initialization value for which the learning phases of the values are (s, f)-incremental, denoted σ th 0, is lower and upper bounded in the following way: Where A and B are defined as: Proof. To show our for gradient descent and N = 2, we build on the proof techniques of theorem 3 of. We start by deriving the recurrence relation for the values σ(t) for general depth, when t now stands for the iteration. Remembering that w n i = σ i, we write down the gradient update for w i (t): Raising w i (t) to the N th power, we get the gradient update for the σ values: Next, we will prove a simple lemma which gives us the maximal learning rate we will consider for the analysis, for which there is no overshooting (the values don't grow larger than the optimal values). Lemma 1. For the gradient update in, assuming for c ≤ 1, we have: Defining r i = σi σ * i and dividing both sides by σ * i, we have: It is enough to show that for any 0 ≤ r ≤ 1, we have that re r 1− 2 N (1−r) ≤ 1, as over-shooting occurs when r i (t) > 1. Indeed, this function is monotonic increasing in 0 ≤ r ≤ 1 (since the exponent is non negative), and equals 1 when r = 1. Since r = 1 is a fixed point and no iteration that starts at r < 1 can cross 1, then r i (t) ≤ 1 for any t. This concludes our proof. Under this choice of learning rate, we can now obtain our incremental learning for gradient descent when N = 2. Our strategy will be bounding σ i (t) from below and above, which will give us a lower and upper bound for t α (σ i). Once we have these bounds, we will be able to describe either a necessary or a sufficient condition on σ 0 for incremental learning, similar to theorem 2. The update rule for N = 2 is: Next, we plug in η = Where in the fourth line we use the inequality 1 1+x ≥ 1 − x, ∀x ≥ 0. We may now subtract 1 σ * i from both sides to obtain: We may now obtain a bound on t α (σ i) by plugging in σ i (t) = ασ * i and taking the log: Rearranging (note that log 1 − cR i − c 2 4 R 2 i < 0 and that our choice of c keeps the argument of the log positive), we get: Next, we follow the same procedure for an upper bound. Starting with our update step: Where in the last line we use the inequality from both sides, we get: Rearranging like before, we get the bound on the α-time: Given these bounds, we would like to find the conditions on σ 0 that allows for (s, f)-incremental learning. We will find a sufficient condition and a necessary condition, like in the proof of theorem 2. A sufficient condition for incremental learning will be one which is possibly stricter than the exact condition. We can find such a condition by requiring the upper bound of t f (σ i) to be smaller than the lower bound on t s (σ j). This becomes the following condition: and rearranging, we get the following: We may now take the exponent of both sides and rearrange again, remembering = r > 1, to get the following condition: Now, we will add the very reasonable assumption that σ * j ≥ 2σ 0, which allows us to replace Now we can rearrange and isolate σ 0 to get a sufficient condition for incremental learning: A necessary condition for incremental learning will be one which is possibly more relaxed than the exact condition. We can find such a condition by requiring the lower bound of t f (σ i) to be smaller than the upper bound on t s (σ j). This becomes the following condition: log 1−cRj +c 2 R 2 j and rearranging, we get the following: We may now take the exponent of both sides and rearrange again, remembering σ * i σ * j = r > 1, to get the following condition: We may now relax the condition further, by removing the r from the denominator of the left-hand side and the σ 0 from the numerator. This gives us the following: Finally, rearranging gives us the necessary condition: While we were able to generalize our to gradient descent for N = 2, our proof technique relies on the ability to get a non-implicit solution for σ(t) which we discretized and bounded. This is harder to generalize to larger values of N, where the solution is implicit. Still, we can informally illustrate the effect of depth on the dynamics of gradient descent by approximating the update rule of the values. We start by reminding ourselves of the gradient descent update rule for σ, for a learning rate η = c To compare two values in the same scales, we will divide both sides by the optimal value σ * i and look at the update step of the ratio r i = We will focus on the early stages of the optimization process, where r 1. This means we can neglect the 1 − r i (t) term in the update step, giving us the approximate update step we will use to compare the general i, j values: We would like to compare the dynamics of r i and r j, which is difficult to do when the recurrence relation isn't solvable. However, we can observe the first iteration of gradient descent and see how depth affects this iteration. Since we are dealing with variables which are ratios of different optimal values, the initial values of r are different. Denoting r = σ * i σ * j, we can describe the initialization of r j using that of r i: Plugging in the initial conditions and noting that R i = rR j, we get: We see that the two ratios have a similar update, with the ratio of optimal values playing a role in how large the initial value is versus how large the added value is. When we use a small learning rate, we have a very small c which means we can make a final approximation and neglect the higher order terms of c: We can see that while the initial conditions favor r j, the size of the update for r i is larger by a factor of r N −1 N when the initialization and learning rates are small. This accumulates throughout the optimization, making r i eventually converge faster than r j. The effect of depth here is clear -the deeper the model, the larger the relative step size of r i and the faster it converges relative to r j. Learning our toy model, when it's incremental learning is taken to the limit, can be described as an iterative procedure where at every step an additional feature is introduced such that it's weight is non-zero and then the model is optimized over the current set of features. This description is also relevant for the sparse approximation algorithm orthogonal matching pursuit , where the next feature is greedily chosen to be the one which most improves the current model. While the toy model and OMP are very different algorithms for learning sparse linear models, we will show empirically that they behave similarly. This allows us to view incremental learning as a continuous-time extension of a greedy iterative algorithm. To allow for negative weights in our experiments, we augment our toy model as in the toy model of. Our model will have the same induced form as before: However, we parameterize σ using w +, w − ∈ R d in the following way: We can now treat this algorithm as a sparse approximation pursuit algorithm -given a dictionary D ∈ R d×n and an example x ∈ R d, we wish to find the sparsest α for which Dα ≈ x by minimizing the 0 norm of α subject to ||Dα − x|| 2 2 = 0 3. Under this setting, we can compare OMP to our toy model by comparing the sets of features that the two algorithms choose for a given example and dictionary. In figure 3 we run such a comparison. Using a dictionary of 1000 atoms and an example of dimensionality 80 sampled from a random hidden vector of a given sparsity s, we run both algorithms and record the first s features chosen 4. Figure 3: Empirical comparison of the dynamics of the toy model to OMP. The toy model has a depth of 5 and was initialized with a scale of 1e-4 and a learning rate of 3e-3. We compare the fraction of agreement between the sets of first s features selected of the two algorithms for every given sparsity level s, averaged over 100 experiments (the shaded regions are empirical standard deviations). For example, for sparsity level 3, we look at the sets of first 3 features selected by each algorithm and calculate the fraction of them that appear in both sets. For every sparsity s, we plot the mean fraction of agreement between the sets of features chosen by OMP and the toy model over 100 experiments. We see that the two algorithms choose very similar features at the beginning, suggesting that the deep model approximates the discrete behavior of OMP. Only when the number of features increases do we see that the behavior of the two models begins to differ, caused by the fact that the toy model has a finite initialization scale and learning rate. These experiments demonstrate the similarity between the incremental learning of deep models and the discrete behavior of greedy approximation algorithms such as OMP. Adopting this view also allows us to put our finger on another strength of the dynamics of deep models -while greedy algorithms such as OMP require the analytical solution or approximation of every iterate, the dynamics of deep models are able to incrementally learn any differentiable function. For example, looking back at the matrix sensing task and the classification models in section 4, we see that while there isn't an immediate and efficient extension of OMP for these settings, the dynamics of learning deep models extends naturally and exhibits the same incremental learning as OMP. Theorem. Minimizing the deep matrix sensing model described in with gradient flow over the depth normalized squared loss, with Gaussian inputs and weights initialized as in leads to the following dynamical equations for different values of N: Where σ i and σ * i are the ith singular values of W and W *, respectively, corresponding to the same singular vectors. Proof. We will adapt the proof from for multilayer linear networks. The gradient flow equations for W n, n ∈ [N] are:, U diagonalizes all W n matrices at initialization such that D n = U W n U T = N √ σ 0 I. Making this change of variables for all W n, we get: Rearranging, we get a set of decoupled differential equations for the singular values of W n: Note that since these matrices are all diagonal at initialization, the above dynamics ensure that they remain diagonal throughout the optimization. Denoting σ n,i as the i'th singular value of W n and σ i as the i'th singular value of W, we get the following differential equation: Since we assume at initialization that ∀n, m, i: σ n,i = σ m,i = N √ σ 0, the above dynamics are the same for all singular values and we get ∀n, m, i: σ n,i (t) = σ m,i (t) = N σ i (t). We may now use this to calculate the dynamics of the singular value of W, since they are the product the the singular values of all W n matrices. Denoting σ −n,i = k =n σ k,i and using the chain rule: Our analytical are only applicable for the population loss over Gaussian inputs. These conditions are far from the ones used in practice and studied in , where the problem is over-determined and the weights are drawn from a Gaussian distribution with a small variance. To show our regarding incremental learning extend qualitatively to more natural settings, we empirically examine the deep matrix sensing model in this natural setting for different depths and initialization scales as seen in figure 4. Notice how incremental learning is exhibited even when the number of examples is much smaller than the number of parameters in the model. While we can't rely on our theory for describing the exact dynamics of the optimization for these kinds of over-determined problems, the qualitative we get from it are still applicable. Another interesting phenomena we should note is that once the dataset becomes very small (the second row of the figure), we see all "currently active" singular values change at the beginning of every new phase (this is best seen in the bottom-right panel). This suggests that since there is more than one optimal solution, once we increase the current rank of our model it may find a solution that has a different set of singular values and vectors and thus all singular values change at the beginning of a new learning phase. This demonstrates the importance of incremental learning for obtaining The columns correspond to different parameterization depths, while the rows correspond to different dataset sizes. In both cases the problem is over-determined, since the number of examples is smaller than the number of parameters. Since the original matrix is rank-4, we can recognize an unsuccessful recovery when all five singular values are nonzero, as seen clearly for both depth-1 plots. sparse solutions -once the initialization conditions and depth are such that the learning phases are distinct, gradient descent finds the optimal rank-i solution in every phase i. For these dynamics to successfully recover the optimal solution at every phase, the phases need to be far enough apart from each other to allow for the singular values and vectors to change before the next phase begins. Theorem. Minimizing the quadratic network described and initialized as in with gradient flow over the variance loss defined in with Gaussian inputs leads to the following dynamical equations:σ Where σ i and σ * i are the ith singular values of W and W *, respectively, corresponding to the same singular vectors. Proof. Our proof will follow similar lines as the analysis of the deep matrix sensing model. Taking the expectation of the variance loss over Gaussian inputs for our model gives us: Following the gradient flow dynamics over W leads to the following differential equation: We can now calculate the gradient flow dynamics of W T W using the chain rule: Now, under our initialization W T 0 W 0 = σ 0 I, we get that W T W and W T * W * are simultaneously diagonalizable at initialization by some matrix U, such that the following is true for diagonal D and D *: Multiplying equation by U and U T gives us the following dynamics for the singular values of These matrices are diagonal at initialization, and remain diagonal throughout the dynamics (the offdiagonal elements are static according to these equations). We may now look at the dynamics of a single diagonal element, noticing it is equivalent to the depth-2 toy model: It may seem that the variance loss is an unnatural loss function to analyze, since it isn't used in practice. While this is true, we will show how the dynamics of this loss function are an approximation of the square loss dynamics. We begin by describing the dynamics of both losses, showing how incremental learning can't take place for quadratic networks as defined over the squared loss. Then, we show how adding a global bias to the quadratic network leads to similar dynamics for small initialization scales. in the previous section, we derive the differential equations for the singular values of W T W under the variance loss:σ We will now derive similar equations for the squared loss. The scaled squared loss in expectation over the Gaussian inputs is: Figure 5: Quadratic model's evolution of top-5 singular values for a rank-4 labeling function. The rows correspond to whether or not a global bias is introduced to the model. The first two columns are for a large dataset (one optimal solution) and the last two columns are for a small dataset (overdetermined problem). When a bias is introduced, it is initialized to it's optimal value at initialization. Note how without the bias, the singular values are learned together and there is over-shooting of the optimal singular value caused by the coupling of the dynamics of the singular values. For the small datasets, we see that the model with no bias reaches a solution with a larger rank. Once a global bias is introduced, the dynamics become more incremental as in the analysis of the variance loss. Note that in this case the solution obtained for the small dataset is the optimal low-rank solution. the bias isn't optimal, and so incremental learning can still take place (assuming a small enough initialization). Under these considerations, we say that the dynamics of the squared loss for a quadratic network with an added global bias resemble the idealized dynamics of the variance loss for a depth-2 linear model which we analyze formally in the paper. In figure 5 we experimentally show how adding a bias to a quadratic network does lead to incremental learning similar to the depth-2 toy model. In section 4.3 we viewed our toy model as a special case of the deep diagonal networks described in , expected to be biased towards sparse solutions. Figure 6 shows the dynamics of the largest values of σ for different depths of the model. We see that the same type of incremental learning we saw in earlier models exists here as well -the features are learned one by one in deeper models, ing in a sparse solution. The leftmost panel shows how the initialization scale plays a role here as well, with the solution being more sparse when the initialization is small. We should note that these do not defy the of (from which we would expect the initialization not to matter), since their deal with the solution at t → ∞. The linear circular-convolutional network of deals with one-dimensional convolutions with the same number of outputs as inputs, such that the mapping from one hidden layer to the next is parameterized by w n and defined to be: shaded regions denoting empirical standard deviations. We see that depth-1 models reach similar to the max-margin SVM solution as predicted by , while deeper models are highly correlated with the sparse solution, with this correlation increasing when the initialization scale is small. The other panels show the evolution of the absolute values of the top-5 weights of σ for the smallest initialization scale. Note that as we increase the depth, incremental learning is clearly presented. The final layer is a fully connected layer parameterized by w N ∈ R d, such that the final model can be written in the following way: This lemma connects the convolutional network to the diagonal network, and thus we should expect to see the same incremental learning of the values of the diagonal network exhibited by the Fourier coefficients of the convolutional network. In figure 7 we see the same plots as in figure 6 but for the Fourier coefficients of the convolutional model. We see that even when the model is far from the toy parameterization (there is no weight sharing and the initialization is with random Gaussian weights), incremental learning is still clearly seen in the dynamics of the model. We see how the inherent reason for the sparsity towards sparse solution found in is the of the dynamics of the model -small amplitudes are attenuated while large ones are amplified. Published as a conference paper at ICLR 2020 We see that depth-1 models reach similar to the max-margin SVM solution, while deeper models are highly correlated with the optimal sparse solution. The other panels show the evolution of the amplitudes of the top-5 frequencies of σ for the smallest initialization scale. Note that as we increase the depth, incremental learning is clearly presented."

Coupled rule-exemplar supervision and a implication loss helps to jointly learn to denoise rules and imply labels.

Represent the TLDR for finding the corresponding scientific article.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.