Question: represent the article for finding a single-sentence tldr

Question: The backpropagation of error algorithm (BP) is often said to be impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired a number of proposals for understanding how the brain might learn across multiple layers, and hence how it might implement or approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks. Here we present the first on scaling up a biologically motivated model of deep learning to datasets which need deep networks with appropriate architectures to achieve good performance. We present on CIFAR-10 and ImageNet. For CIFAR-10 we show that our algorithm, a straightforward, weight-transport-free variant of difference target-propagation (DTP) modified to remove backpropagation from the penultimate layer, is competitive with BP in training deep networks with locally defined receptive fields that have untied weights. For ImageNet we find that both DTP and our algorithm perform significantly worse than BP, opening questions about whether different architectures or algorithms are required to scale these approaches. Our and implementation details help establish baselines for biologically motivated deep learning schemes going forward. The suitability of the backpropagation of error (BP) algorithm BID27 for explaining learning in the brain was questioned soon after its popularization BID8 BID5. Weaker objections included undesirable characteristics of artificial networks in general, such as their violation of Dale's Law, their lack of cell-type variability, and the need for the gradient signals to be both positive and negative. Much more serious objections were: The need for the feedback connections carrying the gradient to have the same weights as the corresponding feedforward connections and The need for a distinct form of information propagation (error propagation) that does not influence neural activity, and hence does not conform to known biological feedback mechanisms underlying neural communication. Researchers have long sought biologically plausible and empirically powerful learning algorithms that avoid some of these flaws BID1 BID25 BID0 BID23; BID12 BID14 BID10 BID21. A common theme of some of the most promising approaches -such as Contrastive Hebbian Learning BID22, and Generalized Recirculation BID23 -is to use feedback connections to influence neural activity, and to use differences in feedfoward-driven and feedback-driven activities or products of activities to locally approximate gradients BID0 BID26 BID23; BID30 ). Since these activity propagation methods don't require explicit propagation of gradients through the network, they go a long way towards answering the second serious objection noted above. However, many of these methods require long "positive" and "negative" settling phases for computing the activities or activity products whose differences provide the learning signal. Proposals for shortening the phases BID11 BID4 are not entirely satisfactory as they still fundamentally depend on a settling process, and, in general, any settling process will likely be too slow for a brain that needs to quickly compute hidden activities. Indeed, for the same reason, only a handful of the algorithms that require settling have ever been used on large scale problems in machine learning. Perhaps the most practical among this family of "activity propagation" algorithms is target propagation (TP) and its variants BID17 BID11 BID2. The intuition for TP is as follows: Suppose you have a feedforward neural network and have the capacity to compute perfect inverses backwards through the network (i.e., given the activities in layer h l+1, we can compute h l = f −1 (h l+1 ; θ l+1)). If we impose an output target (for a given input) on the output layer, then we can propagate activity backwards through the network to infer what the activities should be to produce the output target. These backwards propagated activities are denoted the layer targets, or simply targets. Then, when computing a feedfoward propagation through the network given some input, we can layer-wise compare the feedforward activations to what they should have been (i.e., the targets), and use the differences to compute weight changes. TP algorithms do not require settling dynamics, and thus can compute forward passes and updates quickly. As well, for one TP variant, it has been shown that weight changes that cause future feedforward activity to be nudged towards their targets approximate the weight changes computed by BP.While TP and its variants are promising as biologically-plausible algorithms, there are some lingering questions about their applicability to the brain. First, the only variant explored empirically -difference target propagation (DTP) -still depends on explicit gradient computation via backpropagation for learning the penultimate layer's outgoing synaptic weights (see Algorithm Box 1 in). Second, they have not been tested on datasets more difficult than MNIST. And third, they have not been incorporated into architectures more complicated than simple multi-layer perceptrons (MLPs).In this work we address each of these issues. Our contribution is threefold: We examine the learning and performance of a biologically-motivated algorithm, Difference Target-propagation (DTP), on MNIST, CIFAR, and ImageNet, We develop a variant of DTP called Simplified Difference Target Propagation (SDTP), which eliminates significant lingering biologically implausible features from DTP, and We investigate the role of weight-sharing convolutions, which are key to performance on difficult datasets in artificial neural networks, by testing the effectiveness of locally connected architectures trained with BP, DTP, and SDTP.Sharing the weights of locally connected units greatly reduces the number of free parameters and this has several very beneficial effects on computer simulations of large neural nets. It improves generalization and it drastically reduces both the amount of memory needed to store the parameters and the amount of communication required between replicas of the same model running on different subsets of the data on different processors. From a biological perspective we are interested in how STDP compares with BP without using weight sharing, so both our BP and our SDTP are considerably worse than convolutional neural nets and take far longer to produce. Consider the case of a feed-forward neural network with L layers {h l} L l=1, whose activations h l are computed by elementwise-applying a non-linear function σ l to an affine transformation of previous layer activations h l−1: DISPLAYFORM0 with input to the network denoted as h 0 = x and the last layer h L used as output. For example, in classification problems the output layer h L parametrizes a predicted distribution over possible labels p(y|h L), usually using the softmax function. The learning signal is then provided as a loss L(h L) incurred by making a prediction for an input x, which in the classification case can be cross-entropy between the ground-truth label distribution q(y|x) and the predicted one: DISPLAYFORM1 The goal of training is then to adjust the parameters Θ = {θ l} L l=1 in order to minimize a given loss over the training set of inputs. In BP and DTP, the final layer target is used to compute a loss, and the gradients from this loss are shuttled backwards (through all layers, in BP, or just one layer, in DTP) in error propagation steps that do not influence actual neural activity. SDTP never transports gradients using error propagation steps, unlike DTP and BP. Backpropagation BID27 ) was popularized as a method for learning in neural networks by computing gradients with respect to layer parameters using the chain rule: DISPLAYFORM0 Thus, gradients are obtained by first propagating activations forward to the output layer and then recursively applying these equations. These equations imply that gradients are propagated backwards through the network using weights symmetric to their feedforward counterparts. This is biologically problematic because it implies a mode of information propagation (error propagation) that does not influence neural activity, and that depends on an implausible network architecture (symmetric weight connectivity for feedforward and feedback directions, which is called the weight transport problem). In target propagation BID17 BID2 backwards communication induces neural activity, unlike in BP where backwards communication passes on gradients without inducing neural activity. The induced activities are those that layers should strive to match so as to produce the target output. After feedforward propagation given some input, the final output layer h L is trained directly to minimize the loss L, while all other layers are trained so as to match their associated targets. In general, good targets are those that minimize the loss computed in the output layer if they were actually realized in feedforward propagation. In networks with invertible layers one could generate such targets by first finding a loss-optimal output activationĥ L (e.g. the correct label distribution) and then propagating it back using inverse transformationsĥ l = f −1 (ĥ l+1 ; θ l+1). Since it is hard to maintain invertibility in a network, approximate inverse transformations (or decoders) can be learned DISPLAYFORM0 Note that this learning obviates the need for symmetric weight connectivity. The generic form of target propagation algorithms we consider in this paper can be summarized as a scheduled minimization of two kinds of losses for each layer: DISPLAYFORM1 2 2 used to train the approximate inverse that is parametrized similarly to the forward computation g(DISPLAYFORM2 where activations h l−1 are assumed to be propagated from the input. One can imagine other learning rules for the inverse, for example, the original DTP algorithm trained inverses on noise-corrupted versions of activations with the purpose of improved generalization. In our implementation we instead used the denoising criterion which we find more biologically plausible, see the appendix for details. The loss is applied for every layer except the first, since the first layer does not need to propagate target inverses backwards. 2 2 penalizes the layer parameters for producing activations different from their targets. Parameters of the last layer are trained to minimize the task's loss L directly. Under this framework both losses are local and involve only single layer's parameters, and implicit dependencies on other layer's parameters are ignored. Variants differ in the way targetsĥ l are computed. Target propagation "Vanilla" target propagation (TP) computes targets by propagating the higher layers' targets backwards through layer-wise inverses; i.e.ĥ l = g(ĥ l+1 ; λ l+1). For traditional categorization tasks the same 1-hot vector in the output will always map back to precisely the same hidden unit activities in a given layer. Thus, this kind of naive TP may have difficulties when different instances of the same class have very different appearances since it will be trying to make their representations identical even in the early layers. Also, there are no guarantees about how TP will behave when the inverses are imperfect. Difference target propagation Difference target propagation updates the output weights and biases using the standard gradient rule, but this is biologically unproblematic because it does not require weight transport BID23 BID21. For most other layers in the network, difference target propagation (DTP) computes targets asĥ l = h l + g(ĥ l+1 ; λ l+1) − g(h l+1 ; λ l+1). The extra terms provide a stabilizing linear correction for imprecise inverse functions. However, in the original work by the penultimate layer target, h L−1, was computed using gradients from the network's loss, rather than by target propagation. That is, DISPLAYFORM0 Though not stated explicitly, this approach was presumably taken to insure that the penultimate layer received reasonable and diverse targets despite the low-dimensional 1-hot targets at the output layer. When there are a small number of 1-hot targets (e.g. 10 classes), learning a good inverse mapping from these vectors back to the hidden activity of the penultimate hidden layer (e.g. 1000 units) might be problematic, since the inverse mapping cannot provide information that is both useful and unique to a particular sample. Using BP in the penultimate layer sidesteps this concern, but deviates from the intent of using these algorithms to avoid gradient computation and delivery. Simplified difference target propagation We introduce SDTP as a simple modification to DTP. In SDTP we compute the target for the penultimate layer DISPLAYFORM1. This completely removes biologically infeasible gradient communication (and hence weight-transport) from the algorithm. However, it is not clear whether targets for the penultimate layer will be diverse enough (given low entropy classification targets) or precise enough (given the inevitable poor performance of the learned inverse for this layer). This is a non-trivial change that requires empirical validation. Parallel and alternating training of inverses In the original implementation of DTP 1, the authors trained forward and inverse model parameters by alternating between their optimizations; in practice they trained one loss for one full epoch of the training set before switching to training the other loss. We considered a variant that simply optimizes both losses in parallel, which seems nominally more plausible in the brain since both forward and feedback connections are thought to undergo plasticity changes simultaneously. Though, it is possible that a kind of alternating learning schedule for forward and backward connections could be tied to wake/sleep cycles. Noise-preserving versus de-noising autoencoder training In the original DTP algorithm, autoencoder training is done via a noise-preserving loss, which may be a principled choice for the algorithm on a computer. But in the brain, autoencoder training is de-noising, since uncontrolled noise is necessarily added downstream of a given layer (e.g. by subsequent spiking activity and stochastic vesicle release). Therefore, in our experiments with TP we use de-noising autoencoder training. We also compared noise-preserving and de-noising losses in the context of DTP and SDTP and found that they performed roughly equivalently (see Appendix 4). Propagate activity forward: DISPLAYFORM0 Compute targets for lower layers: DISPLAYFORM1 Convolution-based architectures are critical for achieving state of the art in image recognition BID16. These architectures are biologically implausible, however, because of their extensive weight sharing. To implement convolutions in biology, many neurons would need to share the values of their weights precisely, which is unlikely. In the absence of weight sharing, the "locally connected" receptive field structure of convolutional neural networks is in fact very biologically realistic and may still offer a useful prior. Under this prior, neurons in the brain could sample from small areas of visual space, then pool together to create spatial maps of feature detectors. We assess the the degree to which BP-guided learning is enhanced by convolutions, and not BP per se, by evaluating learning methods (including BP) on networks with locally connected layers. Since the purpose of our study was not to establish state of the art , but rather to assess the limitations of biologically-motivated learning methods, we focused on evaluating architectures that were considered reasonable for a particular task or dataset. Thus, we did not perform an exhaustive architecture search beyond adjusting total number of training parameters to prevent overfitting. All experiments share the same straightforward methodology: a hyperparameter search was performed for a fixed architecture, for each learning algorithm. We then selected the best run from each hyperparameter search based on validation set accuracy across 5 consecutive training epochs (i.e. passes over training set) at the end of which we also measured accuracy on the test set. All locally-connected architectures consist of a stack of locally-connected layers specified as (receptive field size, number of output channels, stride, padding) followed by an output softmax layer. For padding, SAME denotes padding with zeros to ensure unchanged shape of the output with stride = 1 and VALID padding denotes no padding. For optimization we use Adam BID13, with different hyper-parameters for forward and inverse models in the case of target propagation. All layers are initialized using the method of BID7. In all networks we used the hyperbolic tangent as a nonlinearity between layers as it was previously found to work better with DTP than ReLUs. To compare to previously reported we began with the MNIST dataset, consisting of 60000 train and 10000 test 28 × 28 gray-scale images of hand-drawn digits, with 10000 images from the train test reserved for validation. For the evaluation of fully-connected architectures we chose a network from the original DTP paper, consisting of 7 hidden layers with 240 units per layer. While 7 hidden layers provide arguably excessive capacity for this task, this setup is well-suited for understanding how suitable the considered methods are for learning in relatively deep networks which are known to be prone to exploding or vanishing learning signals. The locally-connected architecture consisted of 4 hidden layers and has the following structure: (3 × 3, 32, 2, SAME), (3 × 3, 16, 1, SAME), (3 × 3, 16, 1, SAME), (3 × 3, 10, 1, VALID).Results are reported in table 1 and the learning dynamics is plotted on figure 4. Quite surprisingly, SDTP performed competitively with respect to both DTP and BP, even though it didn't use gradient propagation to assign targets for the penultimate hidden layer. This suggests that, at least for relatively simple learning tasks, the problem of finite number of targets may not be as serious as one might expect. Locally connected architectures performed well with all variants of target propagation, and about as well as with BP. Still, the ing test accuracy did not match previous known obtained with convolutional networks, which can produce less than 1% test error, see, e.g. BID19. However, the observed improvement in generalization in our experiments must have been solely caused by locally-connected layers, as none of the fully-connected networks with smaller number of hidden layers (and hence with less excessive numbers of parameters) performed similarly. We noticed that target propagation showed noisier and slower learning comparing to BP (see FIG2). Yet, with early stopping and hyper-parameter tuning it performed competitively. One can also see that with a fully-connected architecture BP achieved worse test error selected by our methodology. This is likely explained by the fact that BP overfits to the training set faster (in contrast, none of target propagation variants achieved 0% train error). These same phenomena were also observed in the locally-connected network. CIFAR-10 is a more challenging dataset introduced by BID15. It consists of 32 × 32 RGB images of 10 categories of objects in natural scenes, split into 50000 train and 10000 test images, where we also reserve 10000 train images for validation. In contrast to MNIST, classes in CIFAR-10 do not have a "canonical appearance" such as a "prototypical bird" or "prototypical truck" as opposed to "prototypical 7" or "prototypical 9". This makes them harder to classify with simple template matching, making depth imperative for achieving good performance. To our best knowledge, this is the first empirical study of biologically-motivated learning methods without weight transport on this dataset. We considered a fully-connected network with 3 hidden layers of 1024 units and a 5-layer network with locally-connected layers having the following structure: (3 × 3, 32, 2, SAME), (3 × 3, 32, 2, SAME), (3 × 3, 16, 1, SAME), (3 × 3, 16, 2, SAME), (1 × 1, 10, 1, SAME).Final can be found in table 2. One can see that with even on a more complex dataset different TP variants, including the most biologically-feasible SDTP performed similarly to BP. Clearly, the data augmentation employed (random crops and left-right flips) has been necessary for the locallyconnected network to demonstrate a significant improvement over the fully-connected network, otherwise LC models begin to overfit (see FIG2). At the same time, convolutional analog of the LC network has achieved 31.23% and 34.37% of train and test error correspondingly, without use of data augmentation. This quantitatively demonstrates the need of further advances in biologically-plausible architectures in order to match performance of modern convolutional networks. Table 3: Top-1 test error on ImageNet after 18 epochs. Finally, we assessed performance of the methods on the ImageNet dataset BID28, a large-scale benchmark that has propelled recent progress in deep learning. Again, to the best of our knowledge, this is the first empirical study of biologically-motivated methods and architectures conducted on a dataset of such scale and difficulty. ImageNet consists of 1271167 training examples from which 10000 were reserved for validation and 50000 for testing. It has 1000 object classes appearing in a variety of natural scenes and captured in high-resolution images (resized to 224 × 224).The locally-connected architecture we considered for this experiment was inspired by the ImageNet architecture used in BID31. Unfortunately, the naive replacement of convolutional layers with locally-connected layers would into a computationally-prohibitive architecture, so we decreased number of output channels in the layers and also removed layers with 1 × 1 filters. We also slightly decreased filters in the first layer, from 11 × 11 to 9 × 9. The ing network had the following architecture: (9 × 9, 48, 4, SAME), pooling, (5 × 5, 64, 1, SAME), pooling, (3 × 3, 96, 1, SAME), pooling, (3 × 3, 128, 1, SAME), spatial 6 × 6 average. Here every pooling layer is an average pooling with 3 × 3 receptive field. See the appendix for details of implementing locally-connected networks. To further reduce the amount of required computation, we included only parallel variants of DTP and SDTP in the evaluation, as these methods are more representative of the biological constraints, and are more straightforward to implement given the size of this dataset's epochs. Models were trained for 5 days, ing in 18 passes over training set. The final can be found in table 3. Unlike on MNIST and CIFAR, on ImageNet all variants performed quite poorly. Additionally, it is on this dataset where we first observed a striking difference between BP and the TP variants. A number of factors could contribute to this . One factor may be that deeper networks might require more careful hyperparameter tuning when using TP; for example, different learning rates or amount of noise injected for each layer. A second factor may be the difficulty with learning in the output layer, where a 1000-dimensional vector is predicted from just a 128-dimensional output from the final spatial average layer. Moreover, the inverse computation involves non-compressing learning, which has not been well studied in the context of TP. Unfortunately, preserving the original 1920 channels in the layer certainly presents a computational challenge. Addressing both of these factors could help improve performance, so it would be untimely to conclude on any principal inefficiencies of TP. Therefore, we leave the challenge of matching performance of BP on ImageNet to the future work. Historically, there has been significant disagreement about whether BP can tell us anything interesting about learning in the brain BID5 BID8. Indeed, from the mid 1990s to 2010, work on applying BP to the brain all but disappeared. Recent progress in machine learning has prompted a revival of this debate; where other approaches have failed, deep networks trained via BP have been key to achieving impressive performance on difficult datasets such as ImageNet. It is once again natural to wonder whether some approximation of BP might underlie learning in the brain. However, none of the algorithms proposed as approximations of BP have been tested on the datasets that were instrumental in convincing the machine learning and neuroscience communities to revisit these questions. Here we introduced a straightforward variant of difference target-propogation that completely removed gradient propagation and weight transport and tested it on the challenging task of classifying CIFAR and ImageNet images. We also investigated and reported on the use of local connectivity. We demonstrated that networks trained with SDTP without any weight sharing (i.e. weight transport in the backward pass or weight tying in convolutions) are generally able to compete with those trained with BP on difficult tasks such as CIFAR. However, BP significantly outperforms both DTP and SDTP on ImageNet, and more work is required to understand why this issue arises at scale. We note that although activity-propagation-based algorithms go a long way towards biological plausibility, there are still many biological constraints that we did not address here. For example, we've set aside the question of spiking neurons entirely to focus on asking whether variants of TP can scale up to solve difficult problems at all. The question of spiking networks is an important one BID29 BID9 ), but it is nevertheless possible to gain algorithmic insight to the brain without tackling all of the elements of biological complexity simultaneously. Similarly, we also ignore Dale's law in all of our experiments BID24. In general, we've aimed at the simplest models that allow us to address questions around weight sharing, and the form and function of feedback communication. Algorithms that contend for a role in helping us understand learning in cortex should be able to perform well on difficult domains without relying on weight transport or tying. Thus, our offer a new benchmark for future work looking to evaluate the effectiveness of potential biologically plausible algorithms in more powerful architectures and on more difficult datasets. Although locally-connected layers can be seen as a simple generalization of convolution layers, their implementation is not entirely straightforward. First, a locally-connected layer has many more trainable parameters than a convolutional layer with an equivalent specification (i.e. receptive field size, stride and number of output channels). This means that a simple replacement of every convolutional layer with a locally-connected layer can be computationally prohibitive for larger networks. Thus, one has to decrease the number of parameters in some way to run experiments using a reasonable amount of memory and compute. In our experiments we opted to decrease the number of output channels in each layer by a given factor. Obviously, this can have a negative effect on the ing performance and more work needs to be done to scale locally-connected architectures. Inverse operations When training locally-connected layers with target propagation, one also needs to implement the inverse computation in order to train the feedback weights. As in fully-connected layers, the forward computation implemented by both locally-connected and convolutional layers can be seen as a linear transformation y = W x + b, where the matrix W has a special, sparse structure (i.e., has a block of non-zero elements, and zero-elements elsewhere), and the dimensionality of y is not more than x. The inverse operation requires computation of the form x = V y + c, where matrix V has a similar sparse structure as W T. However, given this sparsity of V, computing the inverse of y using V would be highly inefficient BID6. We instead use an implementation trick often used in deconvolutional architectures. First, we define a forward computation z = Ax, where z and A are dummy activities and weights. We then define a transpose matrix as the gradient of this feedforward operation: DISPLAYFORM0 and thus DISPLAYFORM1 The gradient dz dx (and its multiplication with y) can be very quickly computed by the means of automatic differentiation in many popular deep learning frameworks. Note that this is strictly an implementation detail and does not introduce any additional use of gradients or weight sharing in learning. For DTP and SDTP we optimized over parameters of the model and inverse Adam optimizers, learning rate α used to compute targets for h L−1 in DTP, and the Gaussian noise magnitude σ used to train inverses. For backprop we optimized only the model Adam optimizer parameters. For all experiments the best hyperparameters were found by random searches over 60 random configurations drawn from the relevant ranges specified in table 4. As we mention in section 2.1.1, in our implementation of TP algorithms we use denoising training of model inverses which we find more biologically motivated than noise-preserving training used by. In particular, because downstream activity will always have noise applied to it (e.g., given that downstream neurons spike stochastically), one is always fundamentally in the denoising case in the brain. We did not observe a significant empirical difference between these two methods in practice for either DTP and SDTP. FIG6 shows the learning dynamics for parallel versions of DTP and SDTP with noise-preserving inverse losses to compare with figure 2c with denoising inverse loss. One can see that the considered methods converge to roughly same train and test errors with similar speed.

Represent the short summary for finding the corresponding scientific article (long)

Benchmarks for biologically plausible learning algorithms on complex datasets and architectures

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the scientific article to find a short TLDR

Question: Deep neural networks (DNNs) usually contain millions, maybe billions, of parameters/weights, making both storage and computation very expensive. This has motivated a large body of work to reduce the complexity of the neural network by using sparsity-inducing regularizers. Another well-known approach for controlling the complexity of DNNs is parameter sharing/tying, where certain sets of weights are forced to share a common value. Some forms of weight sharing are hard-wired to express certain in- variances, with a notable example being the shift-invariance of convolutional layers. However, there may be other groups of weights that may be tied together during the learning process, thus further re- ducing the complexity of the network. In this paper, we adopt a recently proposed sparsity-inducing regularizer, named GrOWL (group ordered weighted l1), which encourages sparsity and, simulta- neously, learns which groups of parameters should share a common value. GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates. Unlike standard sparsity-inducing regularizers (e.g., l1 a.k.a. Lasso), GrOWL not only eliminates unimportant neurons by setting all the corresponding weights to zero, but also explicitly identifies strongly correlated neurons by tying the corresponding weights to a common value. This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization in the training process to simultaneously identify significant neurons and groups of parameter that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e., keeping only the significant neurons and enforcing the learned tying structure. We evaluate the proposed approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization performance. Deep neural networks (DNNs) have recently revolutionized machine learning by dramatically advancing the state-of-the-art in several applications, ranging from speech and image recognition to playing video games BID20. A typical DNN consists of a sequence of concatenated layers, potentially involving millions or billions of parameters; by using very large training sets, DNNs are able to learn extremely complex non-linear mappings, features, and dependencies. A large amount of research has focused on the use of regularization in DNN learning BID20, as a means of reducing the generalization error. It has been shown that the parametrization of many DNNs is very redundant, with a large fraction of the parameters being predictable from the remaining ones, with no accuracy loss BID14. Several regularization methods have been proposed to tackle the potential over-fitting due to this redundancy. Arguably, the earliest and simplest choice is the classical 2 norm, known as weight decay in the early neural networks literature, and as ridge regression in statistics. In the past two decades, sparsity-inducing regularization based on the 1 norm (often known as Lasso) BID35, and variants thereof, became standard tools in statistics and machine learning, including in deep learning BID20. Recently, BID32 used group-Lasso (a variant of Lasso that assumes that parameters are organized in groups and encourages sparsity at the group level BID37) in deep learning. One of the effects of Lasso or group-Lasso regularization in learning a DNN is that many of the parameters may become exactly zero, thus reducing the amount of memory needed to store the model, and lowering the computational cost of applying it. Figure 1: A DNN is first trained with GrOWL regularization to simultaneously identify the sparse but significant connectivities and the correlated cluster information of the selected features. We then retrain the neural network only in terms of the selected connectivities while enforcing parameter sharing within each cluster. It has been pointed out by several authors that a major drawback of Lasso (or group-Lasso) regularization is that in the presence of groups of highly correlated covariates/features, it tends to select only one or an arbitrary convex combination of features from each group BID6 BID7 BID17 BID28 BID42. Moreover, the learning process tends to be unstable, in the sense that subsets of parameters that end up being selected may change dramatically with minor changes in the data or algorithmic procedure. In DNNs, it is almost unavoidable to encounter correlated features, not only due to the high dimensionality of the input to each layer, but also because neurons tend to co-adapt, yielding strongly correlated features that are passed as input to the subsequent layer BID34.In this work, we propose using, as a regularizer for learning DNNs, the group version of the ordered weighted 1 (OWL) norm BID17, termed group-OWL (GrOWL), which was recently proposed by BID28. In a linear regression context, GrOWL regularization has been shown to avoid the above mentioned deficiency of group-Lasso regularization. In addition to being a sparsity-inducing regularizer, GrOWL is able to explicitly identify groups of correlated features and set the corresponding parameters/weights to be very close or exactly equal to each other, thus taking advantage of correlated features, rather than being negatively affected by them. In deep learning parlance, this corresponds to adaptive parameter sharing/tying, where instead of having to define a priori which sets of parameters are forced to share a common value, these sets are learned during the training process. We exploit this ability of GrOWL regularization to encourage parameter sparsity and group-clustering in a two-stage procedure depicted in Fig. 1: we first use GrOWL to identify the significant parameters/weights of the network and, simultaneously, the correlated cluster information of the selected features; then, we retrain the network only in terms of the selected features, while enforcing the weights within the same cluster to share a common value. The experiments reported below confirm that using GrOWL regularization in learning DNNs encourages sparsity and also yields parameter sharing, by forcing groups of weights to share a common absolute value. We test the proposed approach on two benchmark datasets, MNIST and CIFAR-10, comparing it with weight decay and group-Lasso regularization, and exploring the accuracy-memory trade-off. Our indicate that GrOWL is able to reduce the number of free parameters in the network without degrading the accuracy, as compared to other approaches. In order to relieve the burden on both required memory and data for training and storing DNNs, a substantial amount of work has focused on reducing the number of free parameters to be estimated, namely by enforcing weight sharing. The classical instance of sharing is found in the convolutional layers of DNNs BID20. In fact, weight-sharing as a simplifying technique for NNs can be traced back to more than 30 years ago BID24 BID30.Recently, there has been a surge of interest in compressing the description of DNNs, with the aim of reducing their storage and communication costs. Various methods have been proposed to approximate or quantize the learned weights after the training process. BID15 have shown that, in some cases, it is possible to replace the original weight matrix with a low-rank approximation. Alternatively, BID1 propose retraining the network layer by layer, keeping the layer inputs and outputs close to the originally trained model, while seeking a sparse transform matrix, whereas BID19 propose using vector quantization to compress the parameters of DNNs. Network pruning is another relevant line of work. In early work, BID25 and BID22 use the information provided by the Hessian of the loss function to remove less important weights; however, this requires expensive computation of second order derivatives. Recently, BID21 reduce the number of parameters by up to an order of magnitude by alternating between learning the parameters and removing those below a certain threshold. propose to prune filters, which seeks sparsity with respect to neurons, rather than connections; that approach relieves the burden on requiring sparse libraries or special hardware to deploy the network. All those methods either require multiple training/retraining iterations or a careful choice of thresholds. There is a large body of work on sparsity-inducing regularization in deep learning. For example, BID12 exploit 1 and 0 regularization to encourage weight sparsity; however, the sparsity level achieved is typically modest, making that approach not competitive for DNN compression. Group-Lasso has also been used in training DNNs; it allows seeking sparsity in terms of neurons BID32 BID2 BID41 BID27 or other structures, e.g., filters, channels, filter shapes, and layer depth BID36. However, as mentioned above, both Lasso and group-Lasso can fail in the presence of strongly correlated features (as illustrated in Section 4, with both synthetic data and real data. A recent stream of work has focused on using further parameter sharing in convolutional DNNs. By tying weights in an appropriate way, BID16 obtain a convolutional DNN with rotation invariance. On the task of analyzing positions in the game Go, BID10 showed improved performance by constraining features to be invariant to reflections along the x-axis, y-axis, and diagonal-axis. Finally, BID9 used a hash function to randomly group the weights such that those in a hash bucket share the same value. In contrast, with GrOWL regularization, we aim to learn weight sharing from the data itself, rather than specifying it a priori. Dropout-type methods have been proposed to fight over-fitting and are very popular, arguably due to their simplicity of implementation BID34 . Dropout has been shown to effectively reduce over-fitting and prevent different neurons from co-adapting. Decorrelation is another popular technique in deep learning pipelines BID4 BID11 BID29 ; unlike sparsity-inducing regularizers, these methods try to make full use of the model's capacity by decorrelating the neurons. Although dropout and decorrelation can reduce over-fitting, they do not compress the network, hence do not address the issue of high memory cost. It should also be mentioned that our proposal can be seen as complementary to dropout and decorrelation: whereas dropout and decorrelation can reduce co-adaption of nodes during training, GrOWL regularization copes with co-adaptation by tying together the weights associated to co-adapted nodes. We start by recalling the definition of the group-OWL (GrOWL) regularizer and very briefly reviewing some of its relevant properties BID28. Definition 1. Given a matrix W ∈ R n×m, let w [i]· denote the row of W with the i-th largest 2 norm. Let λ ∈ R n +, with 0 < λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0. The GrOWL regularizer (which is a norm) DISPLAYFORM0 This is a group version of the OWL regularizer BID17, also known as WSL1 (weighted sorted 1 BID40) and SLOPE BID5, where the groups are the rows of its matrix argument. It is clear that GrOWL includes group-Lasso as a special case when λ 1 = λ n. As a regularizer for multiple/multi-task linear regression, each row of W contains the regression coefficients of a given feature, for the m tasks. It has been shown that by adding the GrOWL regularizer to a standard squared-error loss function, the ing estimate of W has the following property: rows associated with highly correlated covariates are very close or even exactly equal to each other BID28. In the linear case, GrOWL encourages correlated features to form predictive clusters corresponding to the groups of rows that are nearly or exactly equal. The rationale underlying this paper is that when used as a regularizer for DNN learning, GrOWL will induce both sparsity and parameters tying, as illustrated in Fig. 2 and explained below in detail. A typical feed-forward DNN with L layers can be treated as a function f of the following form: DISPLAYFORM0 L denotes the set of parameters of the network, and each f i is a componentwise nonlinear activation function, with the rectified linear unit (ReLU), the sigmoid, and the hyperbolic tangent being common choices for this function BID20. DISPLAYFORM1, DNN learning may be formalized as an optimization problem, DISPLAYFORM2 where L y,ŷ is the loss incurred when the DNN predictsŷ for y, and R is a regularizer. Here, we adopt as regularizer a sum of GrOWL penalties, each for each layer of the neural network, i.e., DISPLAYFORM3 where N l denotes the number of neurons in the l-th layer and 0 < λ DISPLAYFORM4.., b L, the biases are not regularized, as is common practice. As indicated in Eq., the number of groups in each GrOWL regularizer is the number of neurons in the previous layer, i.e., λ (l) ∈ R N l−1. In other words, we treat the weights associated with each input feature as a group. For fully connected layers, where W l ∈ R N l−1 ×N l, each group is a row of the weight matrix. In convolutional layers, where W l ∈ R Fw×F h ×N l−1 ×N l, with F w and F h denoting the width and height, respectively, of each filter, we first reshape W l to a 2-dimensional array, i.e., DISPLAYFORM5, and then apply GrOWL on the reshaped matrix. That is, if the l-th layer is convolutional, then DISPLAYFORM6 Each row of W 2D l represents the operation on an input channel. The rationale to apply the GrOWL regularizer to each row of the reshaped weight matrix is that GrOWL can select the relevant features of the network, while encouraging the coefficient rows of each layer associated with strongly correlated features from the previous layer to be nearly or exactly equal, as depicted in Fig. 2. The goal is to significantly reduce the complexity by: (i) pruning unimportant neurons of the previous layer that correspond to zero rows of the (reshaped) weight matrix of the current layer; (ii) grouping the rows associated with highly correlated features of the previous layer, thus encouraging the coefficient rows in each of these groups to be very close to each other. As a consequence, in the retraining process, we can further compress the neural network by enforcing the parameters within each neuron that belong to the same cluster to share same values. In the work of BID2, each group is predefined as the set of parameters associated to a neuron, and group-Lasso regularization is applied to seek group sparsity, which corresponds to zeroing out redundant neurons of each layer. In contrast, we treat the filters corresponding Figure 2: GrOWL's regularization effect on DNNs. Fully connected layers (Left): for layer l, GrOWL clusters the input features from the previous layer, l − 1, into different groups, e.g., blue and green. Within each neuron of layer l, the weights associated with the input features from the same cluster (input arrows marked with the same color) share the same parameter value. The neurons in layer l − 1 corresponding to zero-valued rows of W l have zero input to layer l, hence get removed automatically. Convolutional layers (right): each group (row) is predefined as the filters associated with the same input channel; parameter sharing is enforced among the filters within each neuron that corresponds with the same cluster (marked as blue with different effects) of input channels.to the same input channel as a group, and GrOWL is applied to prune the redundant groups and thus remove the associated unimportant neurons of the previous layer, while grouping associated parameters of the current layer that correspond with highly correlated input features to different clusters. Moreover, as shown in Section 4, group-Lasso can fail at selecting all relevant features of previous layers, and for the selected ones the corresponding coefficient groups are quite dissimilar from each other, making it impossible to further compress the DNN by enforcing parameter tying. To solve, we use a proximal gradient algorithm BID3, which has the following general form: at the t-th iteration, the parameter estimates are updated according to DISPLAYFORM0 where, for some convex function Q, prox Q denotes its proximity operator (or simply "prox") BID3, defined as prox DISPLAYFORM1 2 denotes the sum of the squares of the differences between the corresponding components of ν and ξ, regardless of their organization (here, a collection of matrices and vectors).Since R(θ), as defined in FORMULA4, is separable across the weight matrices of different layers and zero for b 1,..., b L, the corresponding prox is also separable, thus DISPLAYFORM2 DISPLAYFORM3 It was shown by BID28 that the prox of GrOWL can be computed as follows. For some matrix V ∈ R N ×M, let U = prox Ω λ (V), and v i and u i denote the corresponding i-th rows. Then, DISPLAYFORM4 DISPLAYFORM5 For vectors in R N (in which case GrOWL coincides with OWL), prox Ω λ (l) can be computed with O(n log n) cost, where the core computation is the socalled pool adjacent violators algorithm (PAVA BID13)) for isotonic regression. We provide one of the existing algorithms in Appendix A; for details, the reader is referred to the work of BID5 and BID39. In this paper, we apply the proximal gradient algorithm per epoch, which generally performs better. The training method is summarized in Algorithm 1. GrOWL is a family of regularizers, with different variants obtained by choosing different weight sequences λ 1,..., λ n. In this paper, we propose the following choice: DISPLAYFORM6 where p ∈ {1, ...n} is a parameter. The first p weights follow a linear decay, while the remaining ones are all equal to Λ 1. Notice that, if p = n, the above setting is equivalent to OSCAR BID6. Roughly speaking, Λ 1 controls the sparsifying strength of the regularizer, while Λ 2 controls the clustering property (correlation identification ability) of GrOWL BID28. Moreover, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefficients, i.e., only among relevant coefficient groups. Finding adequate choices for p, Λ 1, and Λ 2 is crucial for jointly selecting the relevant features and identifying the underlying correlations. In practice, we find that with properly chosen p, GrOWL is able to find more correlations than OSCAR. We explore different choices of p in Section 4.1. After the initial training phase, at each layer l, rows of W l that corresponds to highly correlated outputs of layer l − 1 have been made similar or even exactly equal. To further compress the DNN, we force rows that are close to each other to be identical. We first group the rows into different clusters 1 according to the pairwise similarity metric DISPLAYFORM0 where W l,i and W l,j denote the i-th and j-th rows of W l, respectively. With the cluster information obtained by using GrOWL, we enforce parameter sharing for the rows that belong to a same cluster by replacing their values with the averages (centroid) of the rows in that cluster. In the subsequent retraining process, let Gk denote the k-th cluster of the l-th layer, then centroid g DISPLAYFORM1 We assess the performance of the proposed method on two benchmark datasets: MNIST and CIFAR-10. We consider two different networks and compare GrOWL with group-Lasso and weight decay, in terms of the compression vs accuracy trade-off. For fair comparison, the training-retraining pipeline is used with the different regularizers. After the initial training phase, the rows that are close to each other are clustered together and forced to share common values in the retraining phase. We implement all models using Tensorflow BID0. We evaluate the effect of the different regularizers using the following quantities: sparsity = (#zero params)/(# total params), compression rate = (# total params)/(# unique params), and parameter sharing = (# nonzero params)/(# unique params). First, we consider a synthetic data matrix X with block-diagonal covariance matrix Σ, where each block corresponds to a cluster of correlated features, and there is a gap g between two blocks. Within each cluster, the covariance between two features X i and X j is cov(X i, X j) = 0.96 |i−j|, while features from different clusters are generated independently of each other. We set n = 784, K = 10, block size 50, and gap g = 28. We generate 10000 training and 1000 testing examples. FORMULA14 ).We train a NN with a single fully-connected layer of 300 hidden units. FIG0 the first 25000 entries of the sorted pairwise similarity matrices (Eq 10) obtained by applying GrOWL with different p (Eq 9) values. By setting the weights beyond index p to a common constant implies that clustering is only encouraged among the p largest coefficients, i.e., relevant coefficient groups; however, FIG0 shows that, with properly chosen p, GrOWL yields more parameter tying than OSCAR (p = n). On the other hand, smaller p values allow using large Λ 2, encouraging parameter tying among relatively loose correlations. In practice, we find that for p around the target fraction of nonzero parameters leads to good performance in general. The intuition is that we only need to identify correlations among the selected important features. FIG0 shows that weight decay (denoted as 2) also pushes parameters together, though the parameter-tying effect is not as clear as that of GrOWL. As has been observed in the literature BID6, weight decay often achieves better generalization than sparsity-inducing regularizers. It achieves this via parameter shrinkage, especially in the highly correlated region, but it does not yield sparse models. In the following section, we explore the compression performance of GrOWL by comparing it with both group-Lasso and weight decay. We also explore how to further improve the accuracy vs compression trade-off by using sparsity-inducing regularization together with weight decay. For each case, the baseline performance is provided as the best performance obtained by running the original neural network (without compression) after sweeping the hyper-parameter on the weight decay regularizer over a range of values. The MNIST dataset contains centered images of handwritten digits, of size 28×28 pixels FIG2 shows the (784 × 784) correlation matrix of the dataset (the margins are zero due to the redundant of the images). We use a network with a single fully connected layer of 300 hidden units. The network is trained for 300 epochs and then retrained for an additional 100 epochs, both with momentum. The initial learning rate is set to 0.001, for both training and retraining, and is reduced by a factor of 0.96 every 10 epochs. We set p = 0.5, and Λ 1, Λ 2 are selected by grid search. Pairwise similarities (see Eq. FORMULA15) between the rows of the weight matrices learned with different regularizers are shown in FIG2. As we can see, GrOWL (+ 2) identifies more correlations than group-Lasso (+ 2), and the similarity patterns in FIG2 are very close to that of the data FIG2 ). On the other hand, weight decay also identifies correlations between parameter rows, but it does not induce sparsity. Moreover, as shown in Table 1 FORMULA15 ) of the parameter rows obtained by training the neural network with GrOWL, GrOWL+ 2, group-Lasso, group-Lasso+ 2 and weight decay. Table 1: Sparsity, parameter sharing, and compression rate on MNIST. Baseline model is trained with weight decay and we do not enforce parameter sharing for baseline model. We train each model for 5 times and report the average values together with their standard deviations. Sparsity Parameter Sharing Compression ratio Accuracy none 0.0 ± 0% 1.0 ± 0 1.0 ± 0 98.3 ± 0.1% weight decay 0.0 ± 0% 1.6 ± 0 1.6 ± 0 98.4 ± 0.0% group-Lasso 87.6 ± 0.1% 1.9 ± 0.1 15.8 ± 1.0 98.1 ± 0.1% group-Lasso+ 2 93.2 ± 0.4%1.6 ± 0.1 23.7 ± 2.1 98.0 ± 0.1% GrOWL 80.4 ± 1.0% 3.2 ± 0.1 16.7 ± 1.3 98.1 ± 0.1% GrOWL+ 2 83.6 ± 0.5% 3.9 ± 0.1 24.1 ± 0.8 98.1 ± 0.1%The compression vs accuracy trade-off of the different regularizers is summarized in Table 1, where we see that applying 2 regularization together with group-Lasso or GrOWL leads to a higher compression ratio, with negligible effect on the accuracy. Table 1 also shows that, even with lower sparsity after the initial training phase, GrOWL (+ 2) compresses the network more than group-Lasso (+ 2), due to the significant amount of correlation it identifies; this also implies that group-Lasso only selects a subset of the correlated features, while GrOWL selects all of them. On the other hand, group-Lasso suffers from randomly selecting a subset of correlated features; this effect is illustrated in FIG4, which plots the indices of nonzero rows, showing that GrOWL (+ 2) stably selects relevant features while group-Lasso (+ 2) does not. The mean ratios of changed indices 2 are 11.09%, 0.59%, 32.07%, and 0.62% for group-Lasso, GrOWL, group-Lasso+ 2, and GrOWL+ 2, respectively. To evaluate the proposed method on large DNNs, we consider a VGG-like BID33 architecture proposed by BID38 on the CIFAR-10 dataset. The network architecture is summarized in Appendix C; comparing with the original VGG of BID33, their fully connected layers are replaced with two much smaller ones. A batch normalization layer is added after each convolutional layer and the first fully connected layer. Unlike BID38, we don't use dropout. We first train the network under different regularizers for 150 epochs, then retrain it for another 50 epochs, using the learning rate decay scheme described by He et al. 2 The mean ratio of changed indices is defined as: Table 2: Sparsity (S1) and Parameter Sharing (S2) of VGG-16 on CIFAR-10. Layers marked by * are regularized. We report the averaged over 5 runs. Weight Decay group-Lasso group-Lasso + 2 GrOWL GrOWL + 2 (S1, S2) (S1, S2) (S1, S2) (S1, S2) (S1, S2) conv1 0%, 1.: the initial rates for the training and retraining phases are set to 0.01 and 0.001, respectively; the learning rate is multiplied by 0.1 every 60 epochs of the training phase, and every 20 epochs of the retraining phase. For GrOWL (+ 2), we set p = 0.1 n (see Eq. FORMULA14) for all layers, where n denotes the number of rows of the (reshaped) weight matrices of each layer. The are summarized in Table 2. For all of the regularizers, we use the affinity propagation algorithm (with preference value 3 set to 0.8) to cluster the rows at the end of initial training process. Our experiments showed that it is hard to encourage parameter tying in the first 7 convolutional layers; this may be because the filters of these first 7 convolutional layers have comparatively large feature maps (from 32 × 32 to 8 × 8), which are only loosely correlated. We illustrate this reasoning in Fig. 6, showing the cosine similarity between the vectorized output channels of layers 1, 6, 10, and 11, at the end of the training phase; it can be seen that the outputs of layers 10 and 11 have many more significant similarities than that of layer 6. Although the output channels of layer 1 also Figure 6: Output channel cosine similarity histogram obtained with different regularizers. Labels: GO:GrOWL, GOL:GrOWL+ 2, GL:group-Lasso, GLL:group-Lasso+ 2, WD:weight decay.have certain similarities, as seen in Table 2, neither GrOWL (+ 2) nor weight decay tends to tie the associated weights. This may mean that the network is maintaining the diversity of the inputs in the first few convolutional layers. Although GrOWL and weight decay both encourage parameter tying in layers 9-13, weight decay does it with less intensity and does not yield a sparse model, thus it cannot significantly compress the network. propose to prune small weights after the initial training phase with weight decay, then retrain the reduced network; however, this type of method only achieves compression 4 ratios around 3. As mentioned by, layers 3-7 can be very sensitive to pruning; however, both GrOWL (+ 2) and group-Lasso (+ 2) effectively compress them, with minor accuracy loss. On the other hand, similar to what we observed by running the simple fully-connected network on MNIST, the accuracy-memory trade-off improves significantly by applying GrOWL or group-Lasso together with 2. However, Table 2 also shows that the trade-off achieved by GrOWL (+ 2) and group-Lasso (+ 2) are almost the same. We suspect that this is caused by the fact that CIFAR-10 is simple enough that one could still expect a good performance after strong network compression. We believe this gap in the compression vs accuracy trade-off can be further increased in larger networks on more complex datasets. We leave this question for future research. We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning. By leveraging on GrOWL's capability of simultaneously pruning redundant parameters and tying parameters associated with highly correlated features, we achieve significant reduction of model complexity, with a slight or even no loss in generalization accuracy. We evaluate the proposed method on both a fully connected neural network and a deep convolutional neural network. The show that GrOWL can compress large DNNs by factors ranging from 11.4 to 14.5, with negligible loss on accuracy. The correlation patterns identified by GrOWL are close to those of the input features to each layer. This may be important to reveal the structure of the features, contributing to the interpretability of deep learning models. On the other hand, by automatically tying together the parameters corresponding to highly correlated features, GrOWL alleviates the negative effect of strong correlations that might be induced by the noisy input or the co-adaption tendency of DNNs. The gap in the accuracy vs memory trade-off obtained by applying GrOWL and group-Lasso decreases as we move to large DNNs. Although we suspect this can be caused by running a much larger network on a simple dataset, it motivates us to explore different ways to apply GrOWL to compress neural networks. One possible approach is to apply GrOWL within each neuron by predefining each 2D convolutional filter as a group (instead all 2D convolutional filters corresponding to the same input features). By doing so, we encourage parameter sharing among much smaller units, which in turn would further improve the diversity vs parameter sharing trade-off. We leave this for future work. Various methods have been proposed to compute the proximal mapping of OWL (ProxOWL). It has been proven that the computation complexity of these methods is O(n log n) which is just slightly worse than the soft thresholding method for solving 1 norm regularization. In this paper, we use Algorithm 2 that was originally proposed in BID5.Unlike k-means or agglomerative algorithm, Affinity Propagation does not require the number of clusters as an input. We deem this as a desired property for enforcing parameter sharing in neural network compression because it's impossible to have the exact number of clusters as a prior information. In practice, the input preference of Affinity Propagation determines how likely each sample will be chosen as an exemplar and its value will influence the number of clusters created. APPENDIX C VGG-16 ON CIFAR-10 Table 4: VGG: Clustering rows over different preference values for running the affinity propagation algorithm (Algorithm 3). For each experiment, we report clustering accuracy (A), compression rate (C), and parameter sharing (S) of layers 9-14. For each regularizer, we use different preference values to run Algorithm 3 to cluster the rows at the end of initial training process. Then we retrain the neural network correspondingly. The are reported as the averages over 5 training and retraining runs. Preference Value 0.6 0.7 0.8 0.9 (A, C, S) (A, C, S) (A, C, S) (A, C, S) GrOWL 92.2%, 13.6, 3.5 92.2%, 12.5, 2.6 92.2%, 11.4, 2.1 92.2%, 10.9, 1.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning.

represent the tldr for finding the corresponding scientific article
The provided query could be "We explored how a novel method of compositional set embeddings can both perceive and represent not just a single class but an entire set of classes that is associated with the input data." and the positive "We explore the idea of compositional set embeddings that can be used to infer not just a single class, but the set of classes associated with the input data (e.g., image, video, audio signal). This can be useful, for example, in multi-object detection in images, or multi-speaker diarization (one-shot learning) in audio. In particular, we devise and implement two novel models consisting of an embedding function f trained jointly with a “composite” function g that computes set union opera- tions between the classes encoded in two embedding vectors; and embedding f trained jointly with a “query” function h that computes whether the classes en- coded in one embedding subsume the classes encoded in another embedding. In contrast to prior work, these models must both perceive the classes associated with the input examples, and also encode the relationships between different class label sets. In experiments conducted on simulated data, OmniGlot, and COCO datasets, the proposed composite embedding models outperform baselines based on traditional embedding approaches. Embeddings, especially as enabled by advances in deep learning, have found widespread use in natural language processing, object recognition, face identification and verification, speaker verification and diarization (i.e., who is speaking when ), and other areas. What embedding functions have in common is that they map their input into a fixed-length distributed representation (i.e., continuous space) that facilitates more efficient and accurate downstream analysis than simplistic representations such as one-of-k. Moreover, they are amenable to one-shot and few-shot learning since the set of classes that can be represented does not depend directly on the dimensionality of the embedding space. Previous research on embeddings has focused on cases where each example is associated with just one class (e.g., the image contains only one person's face). In contrast, we investigate the case where each example is associated with not just one, but an entire subset of classes from a universe S. The goal is to embed each example so that questions of two types can be answered (see Figure 1 (a)): Is the set of classes in example x a equal to the union of the classes in examples x b and x c? Does the set of classes in example x a subsume the set of classes in example x b? Importantly, we focus on settings in which the classes present in the example must be perceived automatically. We approach this problem using compositional set embeddings. Like traditional embeddings, we train a function f that maps each example x ∈ R n into an embedding space R m so that examples with the same classes are mapped close together and examples with different classes are mapped far apart. Unlike traditional embeddings, our function f is trained to represent the set of classes that is associated with each example, so that questions about set union and subsumption can be answered by comparing vectors in the embedding space. We do not assume that the mechanism by which examples (e.g., images, audio signals) are rendered from multiple classes is known. Rather, the rendering process must be learned from training data. We propose two models, whereby f is trained jointly with either a "composition" function g (Model I) that answers questions about set union, or a "query" function h (Model II) that answers question about subsumption (see Figure 1(a) ). Figure 1: (a): Overview of the paper: embedding function f is trained jointly with either the composition function g or the query function h. In particular, the goal is for g to "compose" the embeddings of two examples, containing classes T and U respectively, to approximate the embedding of an example containing classes T ∪ U. (b): 2-D projection of the embedding space from Experiment 1 on test classes and examples not seen during training (one-shot learning). Function g composes two embeddings (two arrow tails) and maps the back into the embedding space (arrow head). To substantial if imperfect degree, the embedding space is compositional as described in (a). To our knowledge, this computational problem is novel. We see at least two use-cases: Speaker recognition and diarization (i.e., infer who is talking within an audio signal) with multiple simultaneous speakers: Given an audio signal containing speakers who were not part of the training set and who may be speaking simultaneously, and given one example of each person speaking in isolation (one-shot learning), infer which set of speakers is talking. Multi-object recognition in images: Given just the embedding of an image x a, answer whether x a contains the object(s) in another image x b. Storing just the embeddings but not the pixels could potentially be more space-efficient. Because of the novelty of the problem, it was not obvious to what baselines we should compare. When evaluating our models, we sought to assess the unique contribution of the compositional embedding above and beyond what a "traditional" embedding could achieve. Hence, we created baselines by endowing a traditional embedding with some extra functionality to enable it to infer label sets. Modeling assumptions and notation: For generality, we refer to the data to be embedded (images, videos, audio signals, etc.) simply as "examples". Let the universe of classes be S. From any subset T ⊆ S, a ground-truth rendering function r: 2 S → R n "renders" an example, i.e., r(T) = x. Inversely, there is also a ground-truth classification function c: R n → 2 S that identifies the label set from the rendered example, i.e., c(x) = T. Neither r nor c is observed. We let e T represent the embedding (i.e., output of f) associated with some example containing classes T. Contribution: To our knowledge, this is the first paper to explore how embedding functions can be trained both to perceive multiple objects in the example and to represent the set of detected objects so that set operations can be conducted among embedded vectors. We instantiate this idea in two ways: Model I for set union (f & g) and Model II for set containment (f & h). By evaluating on synthetic data, OmniGlot handwritten image data , as well as the COCO dataset , we provide a proof-of-concept that "compositional set embeddings" can work. Embeddings: We distinguish between two types of embeddings: "Perceptual" embeddings such as for vision (Facenet ) and speech (x-vector ) where each class (e.g., person whose voice was recorded or face was photographed) may contain widely varying examples across speech content, facial expression, lighting, noise, etc. Word embeddings (word2vec , GloVe ) where each class contains only one exemplar by definition. Within the former, the task of the embedding function is to map examples from the same class close together and examples from other classes far apart. This often requires deep, non-linear transformations to be successful. With word embeddings, the class of each example is already clear and does not need to be inferred; instead, the goal is to give the embedded vectors geometric structure to reflect co-occurrence, similarity in meaning, etc. Compositional embeddings: Since at least 30 years, AI researchers, cognitive scientists, and computational neuroscientists have explored how the embeddings of multiple elements could be combined to reflect relationships between them or higher-level semantics. However, almost all this work was based on word embeddings, in which perception was not necessary. Some early work investigated how the grammatical structure and/or semantics of an input sentence can be represented in an efficient manner in neural networks, and how such a network could be trained . Given the advent of word embeddings, deep NLP architectures can combine the word-level semantics, as represented by the embeddings of the individual elements of an input sentence, to infer higher-level attributes, e.g., sentiment . Recent work has investigated to what extent contemporary recurrent neural networks can generalize to understand novel sentences consisting of known words. Also, in the NLP domain, developed compositional pairwise embeddings that model the co-occurrence relationships between two words given their common context. Probably the most algorithmically similar work to ours is by on compositional network embeddings: the goal is to predict whether two new nodes in a graph, which were not observed during training, are adjacent, using node-based features as predictors. In their approach, two embeddings are used: one to embed the node-based features, and another to aggregate these embedded features into a secondary embedding space. Structurally, their work differs from ours in that the two embedding spaces in their model do not represent the same universe of objects; the embeddings do not capture set relationships. Deep set representations: Our paper is also about how to encode a set of objects with a neural network. One issue is how to ensure invariance to the order in which examples are presented. proposed an approach based on permutation-invariant content-based attention. For producing sets as outputs, proposed a probabilistic model, within a supervised learning paradigm where all classes are known at training time, that predicts both the cardinality and the particular elements of the set. Given two examples x a and x b that are associated with singleton sets {s} and {t}, respectively, the hope is that, for some third example x c that is associated with both classes (i.e., {s, t}), we have Moreover, we hope that g can generalize to any number of classes within the set S. For example, if example x d is associated with a singleton set {u}, then we hope where x e is an example associated with {s, t, u}. There are two challenging tasks that f and g must solve cooperatively: f has to learn to perceive multiple objects that appear simultaneously and may possibly interact with each other -all without knowing the rendering process r of how examples are formed or how classes are combined. g has to define geometrical structure in the embedding space to support set union operations. One way to understand our computational problem is the following: If f is invertible, then ideally we would want g to compute g(e T, e U) = f (r(c(f −1 (e T)) ∪ c(f −1 (e U)))). In other words, one (though not necessarily the only) way that g can perform well is to learn to perform the following actions (without knowing r or c): invert each of the two input embeddings; classify the two corresponding label sets; render an example with the union of the two inferred label sets; and embed the . One-shot learning: Model I can be used for one-shot learning on a set of classes S not seen during training in the following way: We obtain k labeled examples x 1,..., x k from the user, where each {s i} = c(x i) is the singleton set formed from the ith element of S and |S| = k. We call these examples the reference examples. We then infer which set of classes is represented by a new example x using the following procedure: Compute the embedding of x, i.e., f (x). Use f to compute the embedding of each singleton example x i, i.e., e {i} = f (x i). From e {1},..., e {k}, estimate the embedding of every subset T = {s 1, . . ., s l} ⊆ S according to the recurrence relation: Finally, estimate the label of x as arg min Although the number of possible subsets is exponential in |S|, for speaker diarization the number of overlapping speakers is typically small, and thus the iteration is tractable. Functions f and g are trained jointly: For each example x associated with classes T, we compute e T from the singleton prototypes according to Equation 1. (To decide the order in which we apply the recursion, we define an arbitrary ordering over the elements of S and iterate accordingly.) We then compute a hinge loss: where is a small positive real number. In practice, for each example x, we randomly pick one element of T ∈ 2 S for comparison. See Appendix for a discussion of an alternative (but less effective) training procedure. To explore the viability of Model I, we first conducted a simulation using 1-D "audio" signals, in which each "speaker" s i ∈ S is modeled with a prototype p i ∈ R n consisting of the superposition of some randomly chosen frequencies and phases. Then, the simultaneous sound from multiple (up to 3) speakers is given by a rendering function r that computes a vector sum of slightly perturbed versions of the prototype vectors and then clips the . (See Figure 2 (a) for example waveforms, and the Appendix for more details.) We trained an embedding and composition functions f and g, as described in Section 3.1, on a set of 250 speakers. We then tested these functions on test speakers (and examples) not seen during training after providing just a single "reference" example of each speaker for one-shot learning. The goal is to identify the exact set of speakers who contributed to the formation of each audio waveform. Architecture: For function f we used a simple convolutional neural network that produces unitlength vectors in R 32: where Conv(k, f) is a 1-D convolutional layer with f filters of size k, stride 1, and zero-padding to preserve the spatial extent of the example; BN is batch normalization; MP(k) is a max-pooling layer with width and stride k; FC(n) is a fully-connected layer with n neurons; and L2Norm is a L 2 -normalization layer. We constructed the composition function (g DNN) to be symmetric: is a symmetric function (with parameter matrices W 1, W 2) of its two examples a, b ∈ R n that produces a vector in R k. Assessment: We evaluated accuracy (% correct) in multiple ways: the accuracy, over all examples, of identifying the exact set of speakers; the accuracy, separately for each set size, of identifying the exact set of speakers; and the accuracy, over all examples, in distinguishing just the number of speakers in the set rather than their specific identities. As a baseline, we used an oracle that returned the Most Frequently (MF) occurring label set in the test set (or a random selection among multiple most frequent elements). Results are shown in Figure 2 (b). Note that, for the rows that evaluate on k-sets, the embedding functions are still used to compare the test example to every possible subset (see Equation 2) -not just subsets of size k. In general, the compositional embedding method was far more accurate (48.1% for top-1 and 69.6% for top-3 on all subsets) compared to the baseline. The accuracy improvement was largest on singletons (1-sets), which made up 1/3 of the test data, and for which any non-compositional embedding functions (i.e., f trained without any g) would likely work well. Even on 2-sets and 3-sets, however, the compositional embedding approach was substantially more accurate than the baseline. Moreover, part of the task involves inferring the number of elements in the label set (set size determination); on this problem too the compositional embedding was nearly twice as accurate (61.9% vs. 33.3%) compared to the MF baseline. Geometry of the embedding space: We visualized (using PCA) the embedding space for a set of 3 classes not used for training (Figure 1(b) ). Each marker (the symbol/color combination distinguishes the different label sets) represents the embedding of a test example. The clusters of markers for each label set are clearly distinguished by the embedding f. From the examples from each label set, one is randomly chosen as the "reference" examples for 1-shot learning. Using composition function g DNN, their embeddings are combined to estimate where, in the embedding space, an example for the union of their two label sets would lie; the estimated location is shown at the head of the arrow whose two tails come from the two reference examples. Although the estimated and actual clusters do not align exactly, there is substantial agreement. For instance, the examples for speakers {1, 2, 3} (i.e., all 3 people talking simultaneously) are represented by the blue triangles pointing to the right, and the estimate, given by f and g DNN from reference examples of {3} and {1, 2} is shown at the head of the arrow with label {1, 2, 3} beside it. Not all compositions are accurate, e.g., the estimated location for label set {1, 3} is somewhat below the actual cluster location. We also evaluated our method on the OmniGlot dataset . OmniGlot contains handwritten characters from 50 different alphabets; in total it comprises 1623 symbols, each of which was drawn by 20 people and rendered as a 64x64 image. OmniGlot has been previously used in one-shot learning research (e.g., ;). In our experiment, the model is provided with one reference image for each singleton test class. Then, it uses f and g to select the subset of classes that most closely match the embedding of each test example (Equation 2). In this study we considered class label sets up to size 2 (i.e., singletons and 2-sets). The rendering function r randomly picks one of the 20 exemplars from each class; it randomly shifts, scales, and rotates it; and then it adds it to an image with Gaussian noise (see Figure 3(a) ). For images with multiple classes, the pixel-wise minimum across all classes is applied before adding the noise. (See Appendix for more details.) Similar to Experiment 1, the goal is to train f and g so that, on classes not seen during training, the exact set of classes contained in each test example can be inferred. During both training and testing, each of the 15 class label sets (5 singletons and 5 2 = 10 2-sets) occurred with equal frequency. All the embeddings were normalized to unit length. Architecture: For f, we used ResNet-18 Training: The process is similar to Experiment 1. See Appendix for details. Testing: Similar to training, the testing data are grouped into 5 randomly selected classes (not seen during training), and images from these classes are rendered using function r from either singleton or 2-set class label sets. We optimize Equation 2 to estimate the label set for each test example. Baselines: We compared to several baselines: • Most frequent (MF): Always guess the most frequent element in the test set. Since all classes occurred equally frequently, this was equivalent to random guessing. • Traditional f with simulated composite reference examples (SimRef): Given the reference examples of the singleton label sets (e.g., {1}, {3}), we can simulate reference examples for each composite label set (e.g., {1, 3}) and then use a traditional embedding approach to classify the test examples. We simulated imperfect knowledge of the rendering function r by performing pixel-wise minimum, but without shifting/scaling/rotation. In other words, this approach simply superimposes pairs of reference images on top of each other to simulate composite examples. To estimate the label set of each test example, we select the reference example (including the simulated composite references) with minimal L 2 distance. • Traditional f and average (Mean): f is trained as a traditional embedding function (only on singletons) using one-shot learning . Then the embedding of a composite image is computed using the mean of the embeddings of each class in its label set. • Traditional f trained jointly with g mean (g mean): f is trained on only singleton embeddings but is optimized jointly with composition g mean that computes the mean of its inputs. The difference with the previous baseline is that f can benefit from knowing how its embeddings are combined. Results: As shown in Figure 3 (b), the proposed f & g method outperforms the Mean and g mean baselines. Upon closer investigation, we discovered that, while Mean can distinguish each singleton from the other four singletons with high accuracy (0.979, not shown in table), it struggles when it must decide among all 15 possible label sets. The slightly more powerful f & g Mean method (i.e., the composition function has no trainable parameters) can achieve better accuracy on 2-set images, but the accuracy on 1-set images was much worse than other methods. We argue that f and g correspond to two different tasks, and the model is not able to do a good job on both tasks at the same time without optimizing g. If g is a learnable function, then a simple symmetric linear function (g Lin) achieves the best performance. After we stack more FC layers, the performance gets worse. The reason could be overfitting, and it may be possible to achieve better performance with regularization or more training data. In real life, it is very difficult to create a dataset that all singletons in a compositional set are labeled. Thus, we want to extend our compositional embedding from "composition" to "containing". Here we consider a second type of compositional embedding mechanism that tests whether the set of classes associated with one example subsumes the set of classes associated with another example. We implement this using a "query" function h that takes two embedded examples as inputs: In contrast to g, function h is not symmetric: its first and second arguments are the putative superset and subset, respectively. Also, h can be trained in an unsupervised manner w.r.t. the individual examples: it never needs to know which particular label set is associated with an example, but rather only pairwise information about which examples "subsume" other examples. For h, we tried several functions (h DNN, h Lin+FC, h Lin), analogous to the different implementations of g from Section 3.3. The final layers of all models have a 1-dimensional output. Ref. Functions f and h are trained jointly. Since h is not symmetric, its first layer is replaced with two weight matrices for the different input embeddings (see Appendix). In contrast to Model I, "reference" examples are not needed; only the subset relationships between label sets of pairs of examples are required. Model II can be trained on one set of classes and applied to a different set of classes (see Experiment 3), akin to zero-shot learning. It is also useful when the training and testing classes are the same because, in contrast to traditional supervised training of a detector for each possible class, no labels for individual examples are needed. To train f and h, we backpropagate a binary cross-entropy loss, based on correctly answering the query in Eq. 3, through both f and h. Here we assess Model II in a one-shot learning setting (OmniGlot), i.e., training and testing classes are different. As in Experiment 2, we consider class label sets of size up to 2, and we use the same rendering function r. Let f (x a) and f (x b) be the second arguments to h. For x a, we always choose examples associated with two classes, i.e., c(x a) = {s 1, s 2}. For x b, half of the positive examples (i.e., such that h(f (x a), f (x b)) = True) contain either s 1 or s 2, and half contain both classes. For the negative examples (h(f (x a), f (x b)) = False), x b is associated with some other singleton class s 3 ∈ {s 1, s 2}. See Appendix for more details. Baseline: We compared our proposed method with a traditional (non-compositional) embedding method (TradEmb) that is trained to separate examples according to their association with just a single class. In particular, for each composite example x a (i.e., |c(x a)| = 2), we picked one of the two classes arbitrarily (according to some fixed ordering on the elements of S); call this class s 1. Then, we chose both a positive example x b (such that c(x b) = {s 1}) and a negative example x c (such that c(x c) = {s 3} ⊂ c(x a)). We then compute a hinge loss so that the distance between f (x a) and f (x b) is smaller than the distance between f (x a) and f (x c), and backpropagate the loss through f. During testing, we use f to answer a query -does c(x a) contain c(x b)? -by thresholding the distance between f (x a) and f (x b) (threshold of 0.5). Results are shown in Table 1 (a). Compositional embeddings, as implemented with a combination of f trained jointly with either h DNN, h Lin+FC, or h Lin, slightly outperformed the TradEmb baseline, in terms of both % correct accuracy and AUC. Moreover, there was an advantage of deeper architectures for h. We trained and evaluated Model II on COCO . This is a highly challenging problem: in the example in Figure 4, f has to encode a backpack, a cell phone, and a person; then, given completely different images of these classes (and others), h has to decide which objects were present in the original image. Here, training and testing classes are the same (but testing examples are not used for training). In COCO, each image may contain objects from multiple classes, and each object has a bounding box. We used the bounding boxes to crop singleton objects from images. For training, we used the same strategy as Experiment 3, except that in positive queries (i.e., pairs of images x a, x b such that h(f (x a), f (x b)) = True), image x b was always associated with just a singleton label set. During testing, half of x b are contained in x a and half are not (see Appendix). Baseline: We compared to the TradEmb method: to decide the singleton label set for each image during training, we pick the largest object. Since in this problem the training and testing classes are the same, we also implemented another baseline (see Appendix) consisting of a ResNet classifier with multiple independent binary outputs (one for each class). We can then answer queries about the label sets of two images using the labels of the different classes estimated by the ResNet for each image. Results are shown in Table 1 (b). The proposed methods (f & one of the h functions) easily outperformed TradEmb. They also outperformed the ResNet baseline. One possible reason for the latter baseline's poor performance was that the data was highly imbalanced (since most images contain just a few images). As in Experiment 3, deeper architectures for h performed better. We proposed a new kind of embedding mechanism whereby the set of objects contained in the input data (e.g., image, video, audio) must be both perceived and then mapped into a space such that the set relationships -union (Model I) and subset (Model II) -between multiple embedded vectors can be inferred. Importantly, the ground-truth rendering process for how examples are rendered from their component classes is not known and must implicitly be learned. In our experiments, conducted on simulated data, OmniGlot, and COCO, the accuracy was far from perfect but outperformed several baselines, including one based on a traditional embedding approach. The provide a proof-of-concept of how an embedding function f, trained jointly with either the composition function g or the query function h, could be effectively optimized. One possible direction for further research to increase accuracy is to take better advantage of the statistical structure of class co-occurrence in a specific application domain (e.g., which objects tend to co-occur in the same image). A ALTERNATIVE TRAINING PROCEDURE We also tried another method of training f and g with the explicit goal of encouraging g to map e T and e U to be close to e T ∪U. This can be done by training f and g alternately, or by training them jointly in the same backpropagation. However, this approach yielded very poor . A possible explanation is that g could fulfill its goal by mapping all vectors to the same location (e.g., 0). Hence, a trade-off arises between g's goal and f's goal (separating examples with distinct label sets). Since it was initially not clear to us whether functions f and g even exist with the desired properties that could plausibly be implemented as a neural network, we first sought to design them analytically. Composition function g needs to map two vectors on a unit sphere back onto the same sphere, so that the location on the sphere uniquely identifies a set of classes. Here, we assume the set of classes is the same for training and testing. Our construction is inspired by basic group theory, in particular how each element of a permutation group can be represented as a rotation of Euclidean space and performed by multiplication with a permutation matrix. Let |S| = k. We model g after the permutation group of degree m = 2k consisting of all pairwise exchanges of coordinates 2i and 2i − 1 (i = 1, . . ., k), where the group action is composition. Since each such permutation is independent of all the others, the group is commutative and contains contains 2 k elements -just as we desire so that the range of g contains 2 |S| elements. Each member of the permutation group can be associated with a unique permutation matrix, where multiplication among these matrices is also commutative. Define a vector e ∅ ∈ R 2k whose length is 1 and whose components are all distinct. The first condition is standard when creating an embedding, and the second is important so that the permutations applied to the vector can be deduced unambiguously. We associate vector e ∅ with the empty set ∅. Next, to each singleton {s i}, where s i ∈ S, we associate the m-dimensional permutation matrix P {si} that swaps axes 2i and 2i − 1. We define f as a neural network: In its first layers, all the classes associated with example x are detected independently using k binary classifiers. Next, for all the detected classes, their associated permutation matrices P are multiplied together, which yields another commutative permutation matrix -the ant matrix identifies all the classes present in the example. Finally, the is multiplied by vector e ∅. This produces a vector with unit length in the embedding space R m. If the first few layers comprise k binary classifiers d i (z) ∈ {0, 1} (i = 1, . . ., m), then the whole f network computes: We then construct g so that, given two embedded vectors e T, e U, it computes the product of their associated permutation matrices and multiplies the by e ∅: g(e T, e U) = g(P T e ∅, P U e ∅). This construction of f and g enables perfect inference of the union of classes associated with any examples, provided the binary classifiers are perfect. This gave us hope that compositional set embeddings could succeed. Each speaker was represented by a "prototype" waveform comprising the sum of 8 sinusoids (range [−1, +1]) with randomly chosen frequencies and phases. To construct each training and testing example, the prototypes of all the speakers contained in the particular example are added together and then clipped at [−4, +4]. To simulate noise, we slightly perturbed the frequencies and phases of each prototype by random noise drawn uniformly from [−0.02, +0.02]. We also perturbed the superposition by a small amount (from [−0.02, +0.02]) prior to clipping. The training dataset contained a total of 250 simulated speakers. Each minibatch comprised 3072 examples from 5 unique speakers that were combined in randomly selected subsets T: 1024 of these contained individual speakers (singletons), 1024 contained combinations of 2 speakers, and 1024 contained combinations of 3 speakers. Functions f and g were trained jointly as described in Section 3.1. Training was conducted for 4000 epochs with a learning rate of 0.002 and the Adam algorithm. We set the hinge parameter = 0.1. At test time, one of the singleton examples was randomly chosen as the "reference" example for one-shot learning. Similar to training, each test set consists of 3072 examples comprising randomly chosen 1-sets, 2-sets and 3-sets of speakers; see Figure 2 (a) for example waveforms. To generate an example image from a class prototype, we applied random affine transformations consisting of shift up to 20%, scaling up to 10%, and rotation up to 10 •. Gaussian noise was added with mean 0.9 and variance 0.1. We set the hinge parameter = 0.1. Training is performed on 964 OmniGlot classes; from each of these, the 20 images are augmented using affine transformations (described above) to yield 50 images. Each mini-batch comprises 10000 examples from 5 classes, with two images randomly selected per class: one as a reference example, and one as a training example. The validation set consists of 20 OmniGlot classes not used for training. Training is performed using Adam (lr = .0003) to maximize the validation accuracy, up to a maximum of 100 epochs. Testing accuracy is computed over the 659 OmniGlot classes not used for training or validation. The dataset used in this experiment is the same as experiment 2 including the distribution of training set, validation set and test set. We also used the same ResNet-18 as for Experiment 2. Function h was constructed the same as g except that the first layer was relaxed to be asymmetric: W 1 e T + W 2 e U, and the output layer is logistic sigmoid. For training and testing, the proportion of positive and negative queries was fixed to be equal. During training, each mini-batch contains 128 query pairs and each query pair is composed by a positive query and a negative query. We use binary cross entropy loss and Adam optimizer (lr = .0003) to optimize the models. During evaluation, 0.5 is used for thresholding. COCO's training set is used for training, and COCO's labels of subclasses are used as training labels. A small part of images are taken as validation set. For ResNet baseline, We modified ResNet's last's dimension to 80 and applied logistic sigmoid to all of them. Thus, the output layer can be used as 80 classifiers and each one corresponds to one possible subclass. During training, the loss is composed by binary cross entropy loss from all 80 binary classifiers. For every subclass in all 80 possibilities, if it is in the set of subclasses of one image, then the label is 1, otherwise 0. Adam optimizer ((lr) =.0003) is used for optimization. During evaluation, for each query we assume the class labels of subclass images are already known. ResNet is not optimized to answer queries about image pairs. Instead, it tries to encode each image into an n-bit string (for n classes). While this representation can account for all 2 n possible label sets, it may not be the most effective or efficient representation for the task, especially since some objects are very unlikely to co-occur with others. The proposed f &h embedding method can harness the co-occurrence structure to answer queries more accurately, whereas a ResNet trained to recognize all the individual classes does not harness it. For other models singleton images are also required, which are cropped according to COCO's labels of bounding boxes. All images were padded with zeros to be square and then downscaled to 128x128. We used the same architecture as for Experiment 3, except that the input layer was 3 channels instead of 1. Testing was the same as Experiment 3 (likewise with a distance threshold of 0.5). In both training and testing, the number of positive queries and negative queries are balanced. In order to explore how the number of subclasses affects h function, we also plot the accuracy and AUC with different number of subclasses in Figure 5. We can see a decreasing trend of performance when the images contain more kinds of labeled objects."

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.