Question: represent the text to find a very short summary (1-2 sentences max)!

Question: Extreme Classification Methods have become of paramount importance, particularly for Information Retrieval (IR) problems, owing to the development of smart algorithms that are scalable to industry challenges. One of the prime class of models that aim to solve the memory and speed challenge of extreme multi-label learning is Group Testing. Multi-label Group Testing (MLGT) methods construct label groups by grouping original labels either randomly or based on some similarity and then train smaller classifiers to first predict the groups and then recover the original label vectors. Recently, a novel approach called MACH (Merged Average Classifiers via Hashing) was proposed which projects the huge label vectors to a small and manageable count-min sketch (CMS) matrix and then learns to predict this matrix to recover the original prediction probabilities. Thereby, the model memory scales O(logK) for K classes. MACH is a simple algorithm which works exceptionally well in practice. Despite this simplicity of MACH, there is a big gap between the theoretical understanding of the trade-offs with MACH. In this paper we fill this gap. Leveraging the theory of count-min sketch we provide precise quantification of the memory-identifiablity tradeoffs. We extend the theory to the case of multi-label classification, where the dependencies make the estimators hard to calculate in closed forms. To mitigate this issue, we propose novel quadratic approximation using the Inclusion-Exclusion Principle. Our estimator has significantly lower reconstruction error than the typical CMS estimator across various values of number of classes K, label sparsity and compression ratio. Extreme Classification has taken center-stage of Data Mining and Information Retrieval research in the past few years (; b; ;). It refers to the vanilla multiclass and multilabel classification problems where the number of classes K is significantly large. A large number of classes K brings a new set of computational and memory challenges in training and deploying classifiers. There have been several paradigms of models that tackle the scale challenge of Extreme Classification like 1-vs-all methods (b; ; Babbar & Schölkopf, 2017), tree based methods (a;), embedding models , etc. (as noted on the popular Extreme Classification Repository). One of the recent approaches proposed to alleviate the scale challenge of Multilabel Classification is Group Testing (; ;). In this method, all labels are grouped randomly into m groups/clusters. Each label may go into more than one group. We first train a classifier that predicts which of these clusters the input belongs to (treating each cluster as a separate label in a multilabel setting). For any given input, we first predict the clusters into which the true labels of the input may have been pooled. We can then identify all the true labels by taking an intersection over the inverted clusters. This approach suffers from a critical problem that even tree based approaches have, i.e., hard assignment of clusters. Since the recovery of true labels depends solely on hard-prediction of clusters, a mistake in the cluster prediction can cost us dearly in the final label prediction. Also, since the labels are pooled randomly, each individual meta-classifier is a weak and noisy one. In a recent development, Merged Average Classifiers via Hashing (MACH) was proposed that alleviates the hard-prediction problem in Group Testing methods by identifying the best labels based on the sum of prediction probabilities of the respective groups for a given input. In the hindsight, MACH subtly learns to predict a count-min sketch (CMS) matrix of the original probability vector. For the case of multiclass classification (every input having just a single label unlike multilabel), MACH proposes an unbiased estimator to recover the original K dimensional probability vector from the predicted CMS matrix. Multiclass classification naturally fits into the count-min sketch setting as no two labels can appear simultaneously for a given input. But the proposed theory does not naturally extend to multilabel learning. Further, the variance and error bounds for multiclass classification rely heavily on the choice of number of hash tables and the size of each hash table. That aspect has not been explored in prior work. Our Contributions: In this work we broadly make the following contributions: 1) We revisit MACH with a thorough analysis of proposed reconstruction estimator for multiclass learning. In particular, we prove that the variance of estimation is inversely proportional to the product of product of number of hash tables and size of each hash table (in theorem 2). 2) We also obtain a lower bound on hash table hyperparametrs given a tolerance to prediction error (in Theorems 4 and 5). 3) We propose a novel reconstruction estimator for the case of multilabel learning using InclusionExclusion principle (in theorem 6). This estimator comes out as a solution to a quadratic equation (hence we code-name it as 'quadratic estimator'). 4) We simulate multilabel learning setting by generating K dimensional probability vectors and their proxy CMS measurements. We then reconstruct the probability vector using both the mean estimator and the quadratic estimator and show that the reconstruction Mean-Squared Error (MSE) is significantly lower for the new estimator. Count-Min Sketch: Count-Min Sketch (CMS) was proposed to solve the frequency counting problem in large streaming setting. Assume that we have an infinite stream of elements e 1, e 2, e 3,... coming in. Each of these elements can take any value between K distinct ones. Here, K is very large and we cannot afford to store an array of counts to store every element's frequency (limited memory setting). We need a sub-linear efficient data structure from which we can retrieve the frequency of every element. In Count-Min Sketch , we basically assign O(log K)'signatures' to each class using 2-universal hash functions. We use O(log K) different hash functions H 1, H 2, H 3,..., H O(log K), each mapping any class i to a small range of buckets B << K, i.e., H j (i) ∈ {0, 1, 2, ..., B}. We maintain a counting-matrix C of order O(log K) * B. If we encounter class i in the stream of classes, we increment the counts in cells H 1 (i), H 2 (i)....., H O(log K) (i). It is easy to notice that there will be collisions of classes into these counting cells. Hence, the counts for a class in respective cells could be over-estimates of the true count. During inference, we want to know the frequency of a particular element say a 1. We simply go to all the cells where a 1 is mapped to. Each cell gives and over-estimated value of the original frequency of a 1. To reduce the offset of estimation, the algorithm proposes to take the minimum of all the estimates as the approximate frequency, i.e., n approx (An example illustration of CMS is shown in figure 1. Connecting CMS and Extreme Classification: Given a data instance x, a vanilla classifier outputs the probabilities p i, i ∈ {1, 2, ..., K}. We want to essentially compress the information of these K numbers to log K,i.e., we can only keep track of log K = BR measurements. Ideally, without any assumption, we cannot compress the information in K numbers to anything less than O(K), if we want to retain all information. However, in classification, the most informative quantity is the identity of arg max p i. If we can identify a scheme that can recover the high probability classes from smaller measurement vector, we can train a small-classifier to map an input to these measurements instead of the big classifier. The foremost class of models to accomplish this task are Encoder and Decoder based models like Compressive Sensing . The connection between compressed sensing and extreme classification was identified in prior works . We provide an intuitive explanation of why compressed sensing or any other sketching algorithm does work like count-min sketch in the appendix A. Figure 2: Schematic diagram of MACH. Both the input and the label vector are independently hashed R times (label vector is hashed from K to B, K being number of classes and B being number of buckets in each of the R hash tables). Small models are then trained in parallel. MACH is a new paradigm for extreme classification that uses universal hashing to reduce memory and computations. MACH randomly merges K classes into B meta-classes or buckets (B K). We then runs any offthe shelf classifier (typically simple feed forward neural networks) to predict the meta classes. This process is repeated R number of times, changing the hash function each time (or by simply changing the random seed of the same hash function, to induce a different random pooling each time). During prediction, MACH aggregates the output from each of the R small meta classifiers to retrieve the best class. In the schema shown in figure 2, the input is assumed to be a large dimensional sparse vector. In order to reduce model size from both ends (input and output), the sparse input can also be feature hashed to a manageable dimension. Please note that the theoretical analysis of MACH is agnostic to the input feature hashing. We are only concerned with retrieving the most relevant labels from the meta-class predictions. The subsequent sections formalize the algorithm and quantify the mean, variance, error bounds and hyper-parameter bounds. We begin with emphasizing that MACH does not assume any dependence among the classes. This is a fairly strong assumption because often in extreme classification, the labels have strong correlations. More so, this assumption is intrinsically violated in the case of multilabel learning. Nevertheless, MACH works extremely well in practice, particularly at industry scale challenges. Let there be K classes originally. We'll hash them to B meta-classes using a universal hash function. We repeat this process R times each with a different hash function (can be obtained by simply changing the random seed each time). We only have an R * B matrix that holds all information about the original probability vector of K dimensions (R * B K). Typical classification algorithms model the probability P r(y = i|x) = p i where i ∈ {0, 1, 2...K − 1}. With MACH, we bypass the hassle of training a huge last layer by instead modelling P r(y = b|x) = P j b for every hash function h j, where b ∈ {0, 1, 2, ..., B − 1} and j ∈ {0, 1, 2, ..., R − 1}. During prediction, we sought to recover the K vector from P j b matrix using an unbiased estimator as shown in subsequent sections. P j hj (i) stands for the probability of the bin (meta-class) that i th class is hashed into in j th repetition. Our goal is to obtain an unbiased estimator of p i in terms of {P From here on, the analysis diverges between Multiclass and Multilabel classification problems. We have With the above equations, given the R classifier models, an unbiased estimator of p i is: Theorem 1. Proof: Proof for this theorem has been given in . For clarity and coherence, we show the proof again here. For any j, we can always write where 1 hj (k)=hj (i) is an indicator random variable (generically denoted by I k from here on) suggesting whether class k has been hashed into the same bin as class i using hash function j. Since the hash function is universal, the expected value of the indicator is 1 B (each class will uniformly be binned into one of the B buckets). Thus This is because the expression k =i p k = 1−p i as the total probability sum up to one. Simplifying, we get. Using linearity of expectation and the fact that E(P j hj (i) ) = E(P k hj (i) ) for any j = k, it is not difficult to see that this value is also equal to Proof: Using the known V ar(aX + b) = a 2 V ar(X) and the fact that variance accumulates over sum of i.i.d random variables, we can write We first need to get V ar(P j hj (i) ). From eqn. 3, Hence, It's easy to see Hence, by merging eqns. 5 and 6, we get We can observe that larger the original probability p i, lower the variance of estimation which suggests that the higher probabilities are retained with high certainty and the lower probabilities are prone to noise. Since we only care for the correct prediction of the best class, we can offset the noise by increasing R. is also the computational complexity of prediction. With MACH, the memory complexity is O(BRd) and the computational complexity is O(BRd + KR) (including inference). To obtain significant savings, we want BR to be significantly smaller than K. We next show that BR ≈ O(log K) is sufficient for uniquely identifying the final class with high probability. Also, we need to tune the two knobs R and B for optimal performance on recovering the original probabilities. The subsequent theorems facilitate the prior knowledge of reasonable values of R and B based on our reconstruction error tolerance. In , the following theorem has been proven Theorem 3. For any B, R = log, guarantees that all pairs of classes c i and c j are distinguishable from each other with probability greater than 1 − δ 1. The above theorem specifies a bound such that no two pair of classes end up in the same bucket on all R hash functions. While this is simple and intuitive, it does not take into account the ease of classification. To be precise, when the difference between the probability of best class and the 2 nd best class is low (predictions are spurious), it is much harder to identify the best class as oppposed to when the difference is higher. Theorem 3 is completely agnostic to such considerations. Hence, the next theorems quantifies the requirements on R, B based on our tolerance to recovery error between p i andp i and also the ease of prediction (given by the difference between the p i and p j where i and j are the two best classes respectively). for any random variable X. For our proposed unbiased estimator in theorem 1, we have For a large enough B, B−1 B ≈ 1. Hence, we get the desired If the best class i * has p i * > α and we primarily care for recovering p i * with high probability, then we have RB > and classes i and j are the first and second best respectively (p i > p j > p k f or every k = i, j), Hence, based on the previous two theorems, we can get a reasonable estimate of what bucket size B should we choose and how many models that we need to train in parallel. The major difference between multi-class and multi-label classification from an analysis perspective is that eqn. 1 does not apply anymore. Hence, all the subsequent derivations do not apply in the case of multi-label classification. In the following theorems, we'll derive an approximate estimator using inclusion-exclusion principle to recover original probability vectors from MACH measurements for the case of multi-label classification. Each p i independently takes a value in. If we do not assume any relation between p i, it would be very difficult to derive an estimator. The most realistic assumption on the probability vectors is sparsity. Most real datasets have only few labels per sample even when the number of classes K is huge. For the purpose of analysis, we will assume that where V is the average of number of active labels per input. Theorem 6. Proof: P j hj (i) is the probability of union of all classes that have been hashed to bin h j (i) in j th hash function. Hence, using inclusion-exclusion principle, it can be written as Since all classes are independent of each other, we have Aggregating similar terms, we get In typical multilabel dataset, K runs into the order of millions where B is a few thousands. If we ignore all terms with B in denominator, we essentially end up with a plain mean estimator hj (i) ). We ideally want to use all terms but it is very cumbersome to analyze the summation (please note that the summation doesn't simplify to exponential as we have the clause k j = k l in each summation). In our case, we empirically show later on that even by limiting the expression to first order summation (ignore all terms B 2 or higher powers of B in denominator), we get a much better estimator for true probability. We can simplify the above expression into Solving for p i, we get our desired Unfortunately, proposing an unbiased estimator using the above is hard. One intuitive estimator that can potentially work isp i = Using Jensen's inequality (specifically, E[ Hence, E[p i] ≤ p i and we do not have an unbiased estimator. Nevertheless, the next section details simulation experiments that corroborate that our proposed estimator for multilabel classification has much lower mean-squared-error (MSE) than a plain mean estimator. To simulate the setup for multi-label MACH, we perform the following steps: • Choose a base prob ∈ which says how confident the prediction in the original probability vector is. • Initialize a K dimensional vector p orig = (p 1, p 2, ...., p K) with all zeros. We then implant the value base prob in int(V base prob) number of random locations. We now have a vector p orig which obeys p i = V. • Generate 1000 samples of K dimensional label vectors where each dimension i is a Bernoulli random variable with probability p i. These sample labels are realizations of p orig. • Merge each sample label vector into B dimensional binary labels where a bucket b is an OR over the constituent classes {i : h j (i) = b}. We repeat this step for R different hash functions,i.e., for all j ∈ 1, 2,..., R. • For each of R repetitions, calculate the mean of the respective B dimensional labels to get • Reconstruct p approx using theorem 6 and {P j : j = 1, 2, .., R}. • Calculate L2-norm of p orig − p approx • Repeat all above steps for 10000 times (generating a different p orig each time) and report the average L2-norm from the last step (it serves as the reconstruction MSE, lower the better). Following the above steps, we show the comparison of our proposed quadratic estimator in theorem 6 against the plain mean estimator by varying the values of K, B, V and base prob in figure 3. We can infer the following insights from the plots: • As K increases, the MSE grows. This is expected because the reconstructed vector has a small non-zero probability for many of the K classes and this induces noise and hence MSE grows. But the top classes are still retrieved with high certainty. • For any K, V, base prob, the MSE decreases when B increases which is expected (fewer collisions of classes and hence less noisier predictions). As the MSE gets lower, the gains from the square-root estimator are also low. This is good because in scenarios where B and R are small, we can do much better recovery using the proposed estimator. • For any K, B, base prob the MSE increases with V. This is again natural because larger V induces more'true' class collisions and hence the retrieval becomes fuzzy. • For any K, B, V the MSE decreases with base prob, albeit with much little difference than previous cases. This is interesting because a high base prob means that we have few but highly confident'true' classes among K. On the other hand, lower base prob indicates that'true' classes are scattered among a larger subset among K classes. Yet, MACH recovers the original probabilities with commendably low MSE. Varying B for K = 10000 Varying base prob for K = 10000 Varying B for K = 100000 Varying V for K = 100000 Varying base prob for K = 100000 Varying B for K = 1000000 Varying B for K = 1000000 Varying base prob for K = 1000000 Figure 3: Reconstruction Error (MSE) comparison between 1) vanilla mean estimator (plotted in magenta) and 2) proposed square-root estimator (plotted in green); for various configurations of K,B and V. The value of K varies as 10000, 100000, 1000000 for the 1 st, 2 nd and 3 rd rows respectively. In each row, the first plot fixes V, base prob and compares various values of B. The 2 nd plot fixes B, base prob and compares different values of B. The 3 rd one fixes B, V and compares different values of base prob. In all cases, we notice that the square-root estimator is consistently and significantly lower in MSE than the corresponding mean estimator. We perform a rigorous theoretical analysis of using Count-Min-Sketch for Extreme Classification and come up with error bounds and hyper-parameter constraints. We identify a critical shortcoming of reconstruction estimators proposed in prior research. We overcome the shortcoming by treating each bucket in a hash table as a union of merged original classes. Using inclusion-exclusion principle and a controlled label sparsity assumption, we come up with an approximate estimator to reconstruct original probability vector from the predicted Count-Min Sketch measurements. Our new estimator has significantly lower reconstruction MSE than the prior estimator. Why not Compressive Sensing or Count-Sketch? The measurements in Compressive Sensing are not a probability distribution but rather a few linear combinations of original probabilities. Imagine a set of classes {cats, dogs, cars, trucks}. Suppose we want to train a classifier that predicts a compressed distribution of classes like {0.6 * cars + 0.4 * cats, 0.5 * dogs + 0.5 * trucks}. There is no intuitive sense to these classes and we cannot train a model using softmax-loss which has been proven to work the best for classification. We can only attempt to train a regression model to minimize the norm(like L 1 -norm or L 2 -norm) between the projections of true K-vector and the predicted K-vectors(like in the case of

How to estimate original probability vector for millions of classes from count-min sketch measurements -  a theoretical and practical setup.

Represent the short summary for finding the corresponding scientific article (long)

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models. Deep neural network have become a widely used model for machine learning, achieving state-ofthe-art on many tasks. The most common task these models are used for is to perform classification, as in the case of convolutional neural networks (CNNs) used to classify images to a semantic category. CNN models are currently considered the standard for visual tasks, allowing far better accuracy than preceding approaches BID21 BID42.Training NN models and using them for inference requires large amounts of memory and computational resources, thus, extensive amount of research has been done lately to reduce the size of networks. BID6 used weight sharing and specification, BID28 used mixed precision to reduce the size of the neural networks by half. BID44 and BID19 used low rank approximations to speed up NNs. BID17, BID24 and BID51, used a more aggressive approach, in which weights, activations and gradients were quantized to further reduce computation during training. Although aggressive quantization benefits from smaller model size, the extreme compression rate comes with a loss of accuracy. Past work noted the fact that predefined BID31 and random BID15 projections can be used together with a learned affine transformation to achieve competitive on several tasks. In this study suggest the reversed proposal -that common NN models used can learn useful representation even without modifying the final output layer, which often holds a large number of parameters that grows linearly with number of classes. Convolutional neural networks (CNNs) are commonly used to solve a variety of spatial and temporal tasks. CNNs are usually composed of a stack of convolutional parameterized layers, spatial pooling layers and fully connected layers, separated by non-linear activation functions. Earlier architectures of CNNs BID22 BID21 used a set of fully-connected layers at later stage of the network, presumably to allow classification based on global features of an image. The final classifier can also be replaced with a convolutional layer with output feature maps matching the number of classes, as demonstrated by BID40.Despite the enormous number of trainable parameters these layers added to the model, they are known to have a rather marginal impact on the final performance of the network BID48 and are easily compressed and reduced after a model was trained by simple means such as matrix decomposition and sparsification BID6. Further more, modern architecture choices are characterized with the removal of most of the fully connected layers BID25 BID42, which was found to lead to better generalization and overall accuracy, together with a huge decrease in the number of trainable parameters. Additionally, numerous works showed that CNNs can be trained in a metric learning regime BID2 BID35 BID11, where no explicit classification layer was introduced and the objective regarded only distance measures between intermediate representations. BID7 suggested an all-convolutional network variant, where they kept the original initialization of the classification layer fixed with no negative impact on performance on the Cifar10 dataset. All of these properties provide evidence that fully-connected layers are in fact redundant and play a small role in learning and generalization. Despite the apparent minor role they play, fully-connected layers are still commonly used as classification layers, transforming from the dimension of network features N to the number of required class categories C. Therefore, each classification model must hold N · C number of trainable parameters that grows in a linear manner with the number of classes. This property still holds when the fully-connected layer is replaced with a convolutional classifier as shown by BID40.In this work we claim that for common use-cases of convolutional network, the parameters used for the final classification transform are completely redundant, and can be replaced with a predetermined linear transform. As we will show for the first time, this property holds even in largescale models and classification tasks, such as recent architectures trained on the ImageNet benchmark BID4 ).The use of a fixed transform can, in many cases, allow a huge decrease in model parameters, and a possible computational benefit. We suggest that existing models can, with no other modification, devoid their classifier weights, which can help the deployment of those models in devices with low computation ability and smaller memory capacity. Moreover, as we keep the classifier fixed, less parameters need to be updated, reducing the communication cost for models deployed in distributed systems. The use of a fixed transform which does not depend on the number classes can allow models to scale to a large number of possible outputs, without a linear cost in the number of parameters. We also suggest that these finding might shed light on the importance of the preceding non-linear layers to learning and generalization. We focus our attention on the final representation obtained by the network (the last hidden layer), before the classifier. We denote these representation as x = F (z; θ) where F is assumed to be a deep neural network with input z and parameters θ, e.g., a convolutional network, trained by backpropagation. In common NN models, this representation is followed by an additional affine transformation DISPLAYFORM0 where W and b are also trained by back-propagation. For input x of N length, and C different possible outputs, W is required to be a matrix of N × C. Training is done using cross-entropy loss, by feeding the network outputs through a softmax activation DISPLAYFORM1 and reducing the expected negative log likelihood with respect to ground-truth target t ∈ {1, . . ., C}, by minimizing DISPLAYFORM2 where w i is the i-th column of W. To evaluate our conjecture regarding the importance of the final classification transformation, we replaced the trainable parameter matrix W with a fixed orthonormal projection Q ∈ R N ×C, such that ∀i = j: q i · q j = 0 and q i 2 = 1, where q i is the ith column of Q. This can be ensured by a simple random sampling and singular-value decomposition As the rows of classifier weight matrix are fixed with an equally valued L 2 norm, we find it beneficial to also restrict the representation of x by normalizing it to reside on the n-dimensional spherê DISPLAYFORM0 This allows faster training and convergence, as the network does not need to account for changes in the scale of its weights. We now face the problem that q i ·x is bounded between −1 and 1. This causes convergence issues, as the softmax function is scale sensitive, and the network is affected by the inability to re-scale its input. This is similar to the phenomenon described by BID45 with respect to softmax function used for attention mechanisms. In the same spirit, we can amend this issue with a fixed scale T applied to softmax inputs f (y) = softmax(1 T y), also known as a softmax temperature. However, this introduces an additional hyper-parameter which may differ between networks and datasets. Instead, we suggest to introduce a single scalar parameter α to learn the softmax scale, effectively functioning as an inverse of the softmax temperature 1 T. Using normalized weights and an additional scale coefficient is similar in spirit to weightnormalization BID34, with the difference that we use a single scale for all entries in the weight matrix, in contrast to a scale for each row that BID34 uses. We keep the additional vector of bias parameters b ∈ R C, and train using the same negative-loglikelihood criterion. More explicitly, our classifier output is now DISPLAYFORM1 and we minimize the loss: DISPLAYFORM2 where we recall x is the final representation obtained by the network for a specific sample, and t ∈ {1, . . ., C} is the ground-truth label for that sample. Observing the behavior of the α parameter over time revealed a logarithmic growth depicted in graph 1. Interestingly, this is the same behavior exhibited by the norm of a learned classifier, first described by and linked to the generalization of the network. This was recently explained by the under-review work of BID39 as convergence to a max margin classifier. We suggest that using a single parameter will enable a simpler examination and possible further exploration of this phenomenon and its implications. We note that as −1 ≤ q i ·x ≤ 1, we also found it possible to train the network with a simple cosine angle loss:L(x, t) = q i ·x − 1, if i = t, q i ·x + 1, otherwise. allowing to discard the softmax function and its scale altogether, but ing in a slight decrease in final validation accuracy compared to original models. We further suggest the use of a Hadamard matrix BID9 as the final classification transform. Hadamard matrix H is an n × n matrix, where all of its entries are either +1 or −1. Further more, H is orthogonal, such that HH T = nI n where I n is the identity matrix. We can use a truncated Hadamard matrixĤ ∈ {−1, 1}C×N where all C rows are orthogonal as our final classification layer such that y =Ĥx + bThis usage allows two main benefits:• A deterministic, low-memory and easily generated matrix that can be used to classify.• Removal of the need to perform a full matrix-matrix multiplication -as multiplying by a Hadamard matrix can be done by simple sign manipulation and addition. We note that n must be a multiple of 4, but it can be easily truncated to fit normally defined networks. We also note the similarity of using a Hadamard matrix as a final classifier to methods of weight binarization such as the one suggested by BID3. As the classifier weights are fixed to need only 1-bit precision, it is now possible to focus our attention on the features preceding it.3 EXPERIMENTAL ImageNet 75.3% 75.3% 25,557,032 8.01% DenseNet169 BID14 ImageNet 76.2% 76% 14,149,480 11.76% ShuffleNet BID50 ImageNet 65.9% 65.4% 1,826,555 52.56% We used the well known Cifar10 and Cifar100 datasets by BID20 as an initial test-bed to explore the idea of a fixed classifier. Cifar10 is an image classification benchmark dataset containing 50, 000 training images and 10, 000 test images. The images are in color and contain 32 × 32 pixels. There are 10 possible classes of various animals and vehicles. Cifar100 holds the same number of images of same size, but contains 100 different classes. We trained a residual network of on the Cifar10 dataset. We used a network of depth 56 and the same hyper-parameters used in the original work. We compared two variants: the original model with a learned classifier, and our version, where a fixed transformation is used. The shown in figure 2 demonstrate that although the training error is considerably lower for the network with learned classifier, both models achieve the same classification accuracy on the validation set. Our conjecture is that with our new fixed parameterization, the network can no longer increase the norm of a given sample's representation -thus learning its label requires more effort. As this may happen for specific seen samples -it affects only training error. We also compared using a fixed scale variable α at different values vs. a learned parameter. Results for α = {0.1, 1, 10} are depicted in figure 3 for both training and validation error. As can be seen, similar validation accuracy can be obtained using a fixed scale value (in this case α = 1 or 10 will suffice) at the expense of another hyper-parameter to seek. In all our experiments we opted to train this parameter instead. In all experiments the α scale parameter was regularized with the same weight decay coefficient used on original classifier. We then followed to train a model on the Cifar100 dataset. We used the DenseNet-BC model of BID14 with depth of 100 layers and k = 12. We continued to train according to the original regime and setting described for this network and dataset. Naturally, the higher number of classes caused the number of parameters to grow and encompass about 4% of the whole model. Validation accuracy for the fixed-classifier model remained equally good as the original model, and we continued to observe the same training curve. In order to validate our on a more challenging dataset, we used the Imagenet dataset introduced by BID4. The Imagenet dataset spans over 1000 visual classes, and over 1.2 million samples. CNNs used to classify Imagenet such as BID21,, BID43 usually have a hidden representation leading to the final classifier of at least 1024 dimensions. This architectural choice, together with the large number of classes, causes the size of classifier to exceed millions of parameters and taking a sizable share from the entire model size. We evaluated our fixed classifier method on Imagenet using Resnet50 by with the same training regime and hyper-parameters. By using a fixed classifier, approximately 2-million parameters were removed from the model, accounting for about 8% of the model parameters. Following the same procedure, we trained a Densenet169 model BID14 for which a fixed classifier reduced about 12% of the parameters. Similarly to on Cifar10 dataset, we observed the same convergence speed and approximately the same final accuracy on both the validation and training sets. Furthermore, we were interested in evaluating more challenging models where the classifier parameters constitutes the majority amount. For this reason we chose the Shufflenet architecture BID50, which was designed to be used in low memory and limited computing platforms. The Shufflenet network contains about 1.8 million parameters, out of which 0.96 million are part of the final classifier. Fixing the classifier ed with a model with only 0.86 million parameters. This model was trained and found, again, to converge to similar validation accuracy as the original. Interestingly, this method allowed Imagenet training in an under-specified regime, where there are more training samples than number of parameters. This is an unconventional regime for modern deep networks, which are usually over-specified to have many more parameters than training samples BID49. Moreover, many recent theoretical related to neural network training BID47 BID33 BID36 BID37 and even generalization BID5 BID0 BID46 usually assume over-specification. TAB0 summarizes our fixed-classifier on convolutional networks, comparing to originally reported . We offer our drop-in replacement for learned classifier that can be used to train models with fixed classifiers and replicate our 1. As language modeling requires classification of all possible tokens available in the task vocabulary, we were interested to see if a fixed classifier can be used, possible saving a very large number of trainable parameters (vocabulary size can have tens or even hundreds of thousands of different words). Recent works have already found empirically that using the same weights for both word embedding and classifier can yield equal or better than using a separate pair of weights BID18 BID32 BID45. This is compliant with our findings that the linear classifier is largely redundant. To examine further reduction in the number of parameters, we removed both classifier and embedding weights and replaced them with a fixed transform. We trained a language model on the WikiText2 dataset described in BID26, using the same setting in BID27. We used a recurrent model with 2-layers of LSTM BID10 and embedding + hidden size of 512. As the vocabulary of WikiText2 holds about 33K different words, the expected number of parameters in embedding and classifier is about 34-million. This number makes for about 89% from the 38M parameters used for the whole model. We found that using a random orthogonal transform yielded poor compared to learned embedding. We suspect that, in oppose to image classification benchmarks, the embedding layer in language models holds information of the words similarities and relations, thus requiring a fine initialization. To test our intuition, we opted to use pre-trained embeddings using word2vec algorithm by BID29 or PMI factorization as suggested by BID23. We find that using fixed word2vec embeddings, we achieve much better . Specifically, we use 89% less parameters than the fully learned model, and obtain only somewhat worse perplexity. We argue that this implies a required structure in word embedding that stems from semantic relatedness between words and the natural imbalance between classes. However, we suggest that with a much more cost effective ways to train word embeddings (e.g., BID29), we can narrow the gap and avoid their cost when training bigger models. In the last couple of years a we observe a rapid growth in the number of classes benchmark datasets contain, for example: Cifar100 BID20 ), ImageNet1K, ImageNet22k (and language modeling BID26 . Therefore the computational demands of the final classifier will increase as well and should be considered no less than the architecture chosen. We use the work by BID41 as our use case, which introduced JFT-300M -an internal Google dataset with over 18K different classes. Using a Resnet50, with a 2048 sized representation, this led to a model with over 36M parameters. This means that over 60% of the model parameters reside in the final classification layer. BID41 further describes the difficulty in distributing this amount of parameters between the training servers, and the need to split them between 50 sub-layers. We also note the fact that the training procedure needs to account for synchronization after each parameter update -which must incur a non-trivial overhead. Our work can help considerably in this kind of scenario -where using a fixed classifier removes the need to do any gradient synchronization for the final layer. Furthermore, using a Hadamard matrix, we can remove the need to save the transformation altogether, and make it more efficient, allowing considerable memory and computational savings. We argue that our method works due to the ability of preceding layers in the network to learn separable representations that are easily classified even when the classifier itself is fixed. This property can be affected when the ratio between learned features and number of classes is small -that is, when C > N . We've been experimenting with such cases, for example Imagenet classification (C = 1000) using mobilenet-0.5 BID13 where N = 512, or reduced version of ResNet where N = 256. In both scenarios, our method converged similarly to a fully learned classifier reaching the same final validation accuracy. This is strengthening our finding, showing that even in cases in which C > N, fixed classifier can provide equally good . Another possible issue may appear when the possible classes are highly correlated. As a fixed orthogonal classifier does not account for this kind of correlation, it may prove hard for the network to learn in this case. This may suggest another reason for the difficulties we experienced in training a language model using an orthogonal fixed classifier, as word classes tend to have highly correlated instances. Understanding that linear classifiers used in NN models are largely redundant allows us to consider new approaches in training and understanding these models. Recent works BID1 suggested a connection between generalization capabilities of models and various norm-related quantities of their weights. Such might be potentially simplified in our model, since we have a single scalar variable (i.e., scale), which seems to be the only relevant parameter in the model (since we normalize the last hidden layer, and fix the last weight layer).The use of fixed classifiers might be further simplified in Binarized Neural Networks BID16, where the activations and weights are restricted to ±1 during propagations. In this case the norm of the last hidden layer is constant for all samples (equal to the square root of the hidden layer width). This constant can be absorbed into the scale constant α, and there is no need in a per-sample normalization as in eq. 1.We also plan to further explore more efficient ways to learn word embedding, where similar redundancy in classifier weights may suggest simpler forms of token representations -such as low-rank or sparse versions, allowing similar benefits to the fixed transformations we suggested. In this work we suggested removing the parameters from the classification layer used in deep neural networks. We showed empirical suggesting that keeping the classifier fixed cause little or no decline in classification performance for common balanced datasets such as Cifar and Imagenet, while allowing a noticeable reduction in trainable parameters. We argue that fixing the last layer can reduce the computational complexity for training as well as the communication cost in distributed learning. Furthermore, using a Hadamard matrix as classifier might lead to some computational benefits when properly implemented, and save memory otherwise spent on large amount of transformation coefficients. As datasets tend to become more complex by time (e.g., Cifar100, ImageNet1K, ImageNet22k, JFT-300M, and language modeling) we believe that resource hungry affine transformation should remain fixed during training, at least partially. We also found that new efficient methods to create pre-defined word embeddings should be explored, as they require huge amount of parameters that can possibly be avoided when learning a new task. Based on these findings, we recommend future research to focus on representations learned by the non-linear part of neural networks -up to the final classifier, as it seems to be highly redundant.

Represent the TLDR for finding the corresponding scientific article

You can fix the classifier in neural networks without losing accuracy

represent the tldr for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the input

Question: This paper introduces NEMO, an approach to unsupervised object detection that uses motion---instead of image labels---as a cue to learn object detection. To discriminate between motion of the target object and other changes in the image, it relies on negative examples that show the scene without the object. The required data can be collected very easily by recording two short videos, a positive one showing the object in motion and a negative one showing the scene without the object. Without any additional form of pretraining or supervision and despite of occlusions, distractions, camera motion, and adverse lighting, those videos are sufficient to learn object detectors that can be applied to new videos and even generalize to unseen scenes and camera angles. In a baseline comparison, unsupervised object detection outperforms off-the shelf template matching and tracking approaches that are given an initial bounding box of the object. The learned object representations are also shown to be accurate enough to capture the relevant information from manipulation task demonstrations, which makes them applicable to learning from demonstration in robotics. An example of object detection that was learned from 3 minutes of video can be found here: http://y2u.be/u_jyz9_ETz4 Object-based representations are a powerful abstraction of our world. Since these representations remove large amounts of information-an image of size 120 × 160 for example has 120 × 160 × 3 = 57.600 dimensions, while the coordinates of an object in that image only have 2 dimensions-object-based representations enable efficient generalization, simulation, planning, communication, etc. But grounding objects in sensory input currently relies on supervised learning, which requires a high number of labeled images, e.g. 500.000 manually annotated segments to learn 80 objects BID22. This paper takes a step towards replacing this labor-intensive supervision by learning to detect objects from videos that can be gathered quickly with minimal supervision and by exploiting the physical properties of objects. A physical object is a collection of matter that moves as a unit. Motion, in turn, can be a strong cue to learn object detection and replace the need for supervision in the form of labeled images. Given a video of a moving object, we can learn object-based representations by optimizing them to describe physically plausible motion BID16. But this approach only works in the absence of visual distractions. With camera motion, other moving objects, or changes in the , motion alone is not sufficient to learn such representations because many features in the image move in a physically plausible way. This paper improves on previous approaches by learning to ignore visual distractions through negative examples, i.e., videos of the scene without the target object but with the distractions. These negative videos are easy to collect because they do not need to be in sync with the positive ones, i.e., they do not need to have the same sequence of camera movements or the same object motions. This paper also addresses the challenge Figure 1: Learning to detect an object from 3 min of video. Left to right: training video of a pen in hand, negative video without pen, two test videos with per frame detections shown as black dots. [video link] of changes between training and test videos, e.g. due to different lighting or changes in the . Those changes can be harmful if an object representation is extracted using a standard pyramid-shaped convolutional network because every pixel directly affects the output, even if it is far from the object's location. Therefore, this paper uses a spatial encoder architecture that uses a spatial softmax output BID5, which is only affected by the strongest local activations, making it invariant to many visual distractions. The contribution of this paper is to demonstrate unsupervised object detection based on data that is easy and fast to collect. This is achieved by formulating the use of negative examples for object detection as a loss function, combining it with motion-based learning objectives, and using these objectives to train a spatial encoder network using a combination of random search and gradient descent. The ing method is called learning from negative examples and motion (NEMO). A glimpse of the are shown in Figure 1.Experimental in Section 4 show that NEMO can learn new objects from only two short videos of a few minutes, without using pretrained models and without using supervision beyond marking these videos as positive and negative. The also show that NEMO can learn object detection in the presence of frequent occlusions, distractions, camera motion, and changes in lighting and . Even though it uses data that can be collected in seconds to minutes, the learned object detection generalizes to new scenes and camera angles and outperforms template matching and tracking baselines. The experiments also show how the learned object representations can be useful to demonstrate tasks such as writing or pick-and-place tasks, e.g. to make robot learning more data-efficient. This work is strongly related to physics-based representation learning, where a latent representation is learned by optimizing consistency with physics. BID30 learn to map images to latent representations by optimizing consistency with a known dynamics model. BID15 and BID16 make more general assumptions about physical interactions and define them as learning objectives. A number of approaches combine physical assumptions with image reconstruction to learn latent representations BID8 BID31 BID5. BID7 learn to embed object regions in an image using spatio-temporal consistency. BID13 learn object embeddings from self-supervised interactions based on object persistence. BID14 learn representations that are equivariant to known ego motion. BID29 learn latent representations from multiple synchronous videos of motion. While these approaches are similar to this paper in spirit, they learn image embeddings, while this paper learns to detect objects in the image coordinates. This more constrained object-based representation makes the presented approach particularly robust and efficient. This paper is also connected to the idea of active perception BID2, where action is used to facilitate perception. Motion has been used for a long time to identify and track objects BID23, to segment them BID6, to understand their articulation BID18, and so on. Recently, this idea has been combined with learning in order to generalize beyond the observed motion, e.g. to learn object segmentation from videos of moving objects BID27 and from videos generated by robot The variation loss based on two frames at times t and t + d enforces variation between detected object locations, ing in a gradient that pushes z (t) and z DISPLAYFORM0 apart. (c) The slowness loss enforces object detections in frames t and t + 1 to be close, which produces a gradient that pulls z (t) and z (t+1) together. (d) The presence loss enforces that the object is detected in the positive frame t rather than in the negative frame t −, which creates a gradient that increases activations in the positive frame and decreases them in the negative frame.interactions BID28. This paper goes into the same direction for learning object detection by introducing ideas from representation learning and by leveraging negative examples. Labeling the training videos as positive and negative examples can also be viewed as weakly supervised learning, which deals with learning from labels that are only partially informative. Weakly supervised object detection relies on image-wide labels to learn to localize the corresponding objects in the image BID26 BID25. While these approaches use image-wide labels to replace object location labels, which are more difficult to obtain, this paper go a step further and only uses per-video labels and compensates this reduction of supervision by adding motion as a cue for learning object detection. The key idea of NEMO is to learn to detect an object from two videos, a positive video that shows the target object in motion and a negative video of the same scene without that object. These videos are used to optimize two objectives: 1) Learn to detect something that moves in a physically plausible way in the positive video, such that its location varies over time without having instantaneous jumps, which is defined below as a combination of a variation loss and a slowness loss. 2) Learn to detect something that is present in the positive video but not in the negative video, which is defined as a presence loss. These objectives are used to train a spatial encoder network, which produces an object detection based on the strongest activation after a stack of convolutions. Optimization is done by a combination of random search and gradient descent. We will now look in detail into each of these components. NEMO's network architecture is based on the encoder part of deep spatial autoencoders BID5 and therefore called a spatial encoder. The spatial encoder is a stack of convolutional layers without pooling or subsampling, which uses residual connections BID10, batch normal-ization BID12, and ReLU nonlinearities BID24. The spatial encoder has n residual blocks, each with c channels and kernel size k (see FIG0 . The experiments in this paper used c = 32 channels, up to n = 10 layers, and kernel sizes k ∈ {3, 5, 7}. Since the parameters n and k control the receptive field of the object detector, they must roughly match the size of the target object. The experiments used k = 7 for learning to detect the Roomba and k = 3 for all other objects. The output layer of the spatial encoder has a single channel, followed by a spatial softmax, which produces a probability distribution over the object's location in the image. The mean and mode of this distribution estimate the object's location. The mode can be used during inference because it is more robust to distractions but not during gradient-based training since it is not differentiable. The spatial encoder is trained by minimizing a combination of three losses-variation, slowness, and presence (see FIG0, which are defined here. Let us denote the input image at time t as I (t) ∈ R h×w where h and w are the height and width of the image. We will refer to the spatial encoder as f with parameters θ, and the output of f before the spatial softmax as DISPLAYFORM0. By applying the spatial softmax across image coordinates i and j, we get a probability image P (t) ∈ R h×w and its mean z (t) ∈ R 2 normalized to [−1, 1] as DISPLAYFORM1 The first two losses, variation and slowness, operate on the mean z in positive frames. Together, they measure whether the detected object location z (t) moves in a physically plausible way by comparing pairs of z (t) for different t. The variation loss encodes the assumption that the target object does not stay still by enforcing that z t+d is different from z t for d in some range [d min, d max]. The variation loss measures the proximity using e −distance, which is 1 if z t = z t+d and goes to 0 with increasing distance BID16. DISPLAYFORM2 The hyperparameter β scales this distance and controls how far z t and z t+d need to be apart. The hyperparameters d min and d max define for which time differences variation is enforced; d min should be large enough that the object has typically changed its location in the image after d min frames; d max should be small enough that slower changes in the typically take longer than d max frames. All experiments in this paper use the parameters β = 10, d min = 50, and d max = 100.The slowness loss encodes the assumption that objects move with relatively low velocities, i.e., that their locations at time t and t + 1 are typically close to each other. Consequently, this loss measures the squared distance between z in consecutive time steps t and t + 1, which favors smooth over erratic object trajectories BID32 BID15. DISPLAYFORM3 The presence loss encodes the assumption that the object is present in the positive video but not in the negative one. Taking a positive frame t and a negative frame t −, we can compute the probability of the object being somewhere in the positive frame q (t,t −) by computing the spatial softmax jointly over both frames and summing over all pixels. The loss is then defined as negative log probability. DISPLAYFORM4 Under review as a conference paper at ICLR 2019These three losses are combined in a weighted sum, DISPLAYFORM5, where the weights were chosen such that all gradients have the same order of magnitude. All experiments in this paper use w variation = 2, w slowness = 10, and w presence = 1. The losses are optimized from minibatches of size b, such that every minibatch includes b samples of consecutive frames {(I (t), I (t+1) )} t and b samples of frames d ∈ [d min, d max] steps apart {(I (t), I (t+d) )} t,d, which are used to compute the variation and slowness losses. The presence loss uses all combinations of the positive frames in {(I (t), I (t+d) )} t,d with b negative frames {I (t −) } t − ing in 2b 2 pairs to average over. All experiments use b = 10. For a good initialization of the spatial softmax output, O (t) is divided by a temperature α = 10 before the softmax is applied. For numerical stability of the gradient computation, Gaussian noise ∼ N (µ = 0, σ = 10 −5) is added to z t and q DISPLAYFORM0 When L(θ) is optimized with an the adaptive gradient descent method Adam BID19, it either converges very quickly within a few hundred gradient descent steps or-for some objects-gets stuck in a local optimum. NEMO addresses this problem by optimizing the loss with a combination of random search and gradient descent. It initializes the spatial encoder m = 10 times, optimizes each by a small number, e.g. 200, gradient descent steps and then finetunes the best model in additional 1000 gradient steps. Stopping the training this early in very fast training time of about 10 minutes on a single GPU and seemed to improve generalization compared to training to convergence, although this needs to be further investigated. The following experiments evaluate object detection for a number of different objects without using pretrained networks and without using any labeled images, instead relying on a few minutes of positive videos showing a moving object and negative videos of the same scene that are quick and easy to obtain (in total about 20 minutes of video for all experiments). All show object detection on test videos not used for training. The show per-frame detections in subsampled 120 × 160 or 90 × 160 videos without applying any tracking or filtering. The experiments evaluate NEMO in three different settings, testing detection accuracy with static and moving cameras for single and multiple objects and generalization to new scenes and camera angles. The experiments also illustrate how learned object detection can enable learning from demonstration, and provide a comparison to template matching and tracking that shows a substantial advantage over these baselines. The data and code based on TensorFlow BID0 and Keras BID4 to reproduce all experiments will be published with this paper. In the first experiment (see Figure 1), the algorithm learned to detect a pen in a hand from a positive video in which the pen was moved in random spirals across the table and a negative video of the hand moving without the pen. Testing the learned object detector on unseen videos of writing "hello" and "world" shows precise and stable per frame object detection from only 2.5 minutes of video. Note how the method learned to be invariant to distractions such as position of the arm. Moving the camera causes motion in the entire video, which makes it more difficult to learn object detection based on motion. This experiment evaluates NEMO on such videos that were recorded by moving the camera in a way that keeps the object in video without constantly centering on it, because that would cancel any object motion in the image. For this set of experiments, the object detector uses the current frame and additionally a difference image from the previous frame as input, which produced more consistent . The in Figure 3 show that NEMO works despite camera motion, that it can handle difficult lighting conditions as for the toy car, and that the learned object detection can even transfer to different scenes, as for the car detector that was trained in the balcony scene and tested in the hallway. The total training data used for learning all three objects is just under 12 minutes of video. This experiment tests detection of multiple objects in a table top setting with five objects. The training data for this experiment are five videos of about 2 minutes, with one video per object in which the object is randomly moved on the table without any other object being present. Each video was used as positive and the remaining four as negative examples to train a different object detector. The in Figure 4 show the output of these five object detectors in a test video in which objects are randomly rearranged on the table. These show mostly correct detections, even under occlusions by the hand and other objects, and generalization to a different camera angle never seen during training. The video linked in the caption reveals that the object detector is not perfect. It works most of time, but generates two kinds of errors: 1) It occasionally errorneously detects the hand as being the target object even when the object is visible-this is a clear error. 2) More often, it detects the object in a wrong location when it is actually occluded by the hand. This is because, in the current version, the spatial encoder cannot return nothing, it always returns the most likely object location. This could be resolved by estimating the detection uncertainty and only returning the object location when the certainty exceeds a threshold. Since the straight-forward approach of using the value of the pre-softmax activations as a certainty estimate did not produce satisfying , this should be addressed in future work. This experiment illustrates how object-based representations learned by NEMO can enable subsequent learning such as learning from demonstrations (see Figure 5). The experiment uses the object detectors learned in the last experiment and applies them to demonstrations for three pick-and-place tasks. Comparing the object locations in the first and last frame of the demonstration reveals which objects were moved to which locations. Simple statistics over the final object locations across all demonstrations of a task describe the task in a way that could be used as the goal for a planning algorithm or as reward for reinforcement learning. Since there is no other approach for learning object detection from positive videos of moving objects and negative videos without those objects, there is no baseline to fairly compare against. To still put NEMO's performance in the context of existing work on object detection and tracking, this experiment compares it to established template matching and tracking approaches using their OpenCV implementations BID3: template matching with normalized cross correlation BID21, tracking with online boosting (OLB, BID9, tracking with online multiple instance learning , tracking with kernelized correlation filters , and tracking-learning-detection (TLD, BID17. Since all of these baselines need some form of supervision, they were given bounding boxes around the initial object locations to initialize tracking or extract an object template. The baselines were applied to the non-subsampled video with resolution 480 × 640. Figure 6 compares NEMO to these baselines for three test videos from the experiments above, for which 20 equally spaced frames across the video were manually labeled. The show that detecting or tracking the pen is relatively easy. NEMO, MIL, and KCF retain a very low error rate in this video. But for the other two videos, all tracking methods quickly diverge and template matching performs poorly as well, while NEMO solves these videos almost perfectly. The tracking methods fail in these videos due to substantial occlusions during object motion in the table top setting and difficult lighting conditions and fast motion in the toy car example. Template matching cannot handle the change in appearance of the objects based on lighting, scale, and orientation. The additional videos used by NEMO for learning the object detector, however, lead to robustness to these variations, which shows the advantage of unsupervised learning. This paper presented NEMO, a novel approach to unsupervised object detection from short videos of moving objects and negative videos of scenes without those objects. By demonstrating data-efficient and robust object detection without the use of image labels, this paper opens up new research directions. There are a number of extensions that would improve the presented approach. Combining it with ensemble methods, for example, could provide an uncertainty estimate required to infer whether the object is visible in the current frame. Integrating the approach with tracking or filtering could exploit temporal consistency not only during training but also during inference. For learning multiple objects in the same scene, merging the different object detectors into a single network could improve performance by sharing intermediate features. And creating a large-scale data-set for this approach would be very valuable to develop it further. Taking a broader view, the presented approach takes a step towards unsupervised learning of object-based representations. While this paper used manually recorded videos, the method can also be applied to data collected by a robot similar to BID13 and BID28 to learn objects autonomously. Acquiring such object-based representations could build a bridge to geometric and symbolic reasoning and enable efficient learning, communication, prediction, and planning in object-based representations.

Learning to detect objects without image labels from 3 minutes of video

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.