Question: Represent the scientific article to find a short TLDR

Question: Architecture search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption;most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform different-sized NASNets, MobileNets, MobileNets V2 and Wide Residual Networks on CIFAR-10 and ImageNet64x64 within only one week on eight GPUs, which is about 20-40x less compute power than previous architecture search methods that yield state-of-the-art performance. Deep learning has enabled remarkable progress on a variety of perceptual tasks, such as image recognition BID12, speech recognition, and machine translation BID0. One crucial aspect for this progress are novel neural architectures BID25; BID7. Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automatic architecture search methods . Some of the architectures found in an automated way have already outperformed the best manually-designed ones; however, algorithms such as by BID32;; BID20 BID36 for finding these architectures require enormous computational resources often in the range of thousands of GPU days. Prior work on architecture search has typically framed the problem as a single-objective optimization problem. However, most applications of deep learning do not only require high predictive performance on unseen data but also low resource-consumption in terms of, e.g., inference time, model size or energy consumption. Moreover, there is typically an implicit trade-off between predictive performance and consumption of resources. Recently, several architectures have been manually designed that aim at reducing resource-consumption while retaining high predictive performance BID8 BID22. Automatically found neural architectures have also been down-scaled to reduce resource consumption. However, very little previous work has taken the trade-off between resource-consumption and predictive performance into account during automatic architecture search. In this work, we make the following two main contributions:1. To overcome the need for thousands of GPU days BID32 BID21, we make use of operators acting on the space of neural network architectures that preserve the function a network represents, dubbed network morphisms (; BID27, obviating training from scratch and thereby substantially reducing the required training time per network. This mechanism can be interpreted as Lamarckian inheritance in the context of evolutionary algorithms, where Lamarckism refers to a mechanism which allows passing skills acquired during an individual's lifetime (e.g., by means of learning), on to children by means of inheritance. Since network morphisms are limited to solely increasing a network's size (and therefore likely also resource consumption), we introduce approximate network morphisms (Section 3.2) to also allow shrinking networks, which is essential in the context of multi-objective search. The proposed Lamarckian inheritance mechanism could in principle be combined with any evolutionary algorithm for architecture search, or any other method using (a combination of) localized changes in architecture space.2. We propose a Lamarckian Evolutionary algorithm for Multi-Objective Neural Architecture DEsign, dubbed LEMONADE, Section 4, which is suited for the joint optimization of several objectives, such as predictive performance, inference time, or number of parameters. LEMONADE maintains a population of networks on an approximation of the Pareto front of the multiple objectives. In contrast to generic multi-objective algorithms, LEMONADE exploits that evaluating certain objectives (such as an architecture's number of parameters) is cheap while evaluating the predictive performance on validation data is expensive (since it requires training the model first). Thus, LEMONADE handles its various objectives differently: it first selects a subset of architectures, assigning higher probability to architectures that would fill gaps on the Pareto front for the "cheap" objectives; then, it trains and evaluates only this subset, further reducing the computational resource requirements during architecture search. In contrast to other multi-objective architecture search methods, LEMONADE (i) does not require to define a trade-off between performance and other objectives a-priori (e.g., by weighting objectives when using scalarization methods) but rather returns a set of architectures, which allows the user to select a suitable model a-posteriori; (ii) LEMONADE does not require to be initialized with well performing architectures; it can be initialized with trivial architectures and hence requires less prior knowledge. Also, LEMONADE can handle various search spaces, including complex topologies with multiple branches and skip connections. We evaluate LEMONADE for up to five objectives on two different search spaces for image classification: (i) non-modularized architectures and (ii) cells that are used as repeatable building blocks within an architecture BID31 and also allow transfer to other data sets. LEMONADE returns a population of CNNs covering architectures with 10 000 to 10 000 000 parameters. Within only 5 days on 16 GPUs, LEMONADE discovers architectures that are competitive in terms of predictive performance and resource consumption with hand-designed networks, such as MobileNet V2 BID22, as well as architectures that were automatically designed using 40x greater resources and other multi-objective methods . Multi-objective Optimization Multi-objective optimization BID17 ) deals with problems that have multiple, complementary objective functions f 1,..., f n. Let N be the space of feasible solutions N (in our case the space of feasible neural architectures). In general, multi-objective optimization deals with finding N * ∈ N that minimizes the objectives f 1,..., f n. However, typically there is no single N * that minimizes all objectives at the same time. In contrast, there are multiple Pareto-optimal solutions that are optimal in the sense that one cannot reduce any f i without increasing at least one f j. More formally, a solution N Pareto-dominates another solution DISPLAYFORM0 ). The Pareto-optimal solutions N was recently proposed to frame NAS as a reinforcement learning (RL) problem, where the reward of the RL agent is based on the validation performance of the trained architecture BID1 BID32 BID31 BID19. BID32 use a recurrent neural network to generate a string representing the neural architecture. In a follow-up work, search for cells, which are repeated according to a fixed macro architecture to generate the eventual architecture. Defining the architecture based on a cell simplifies the search space. An alternative to using RL are neuro-evolutionary approaches that use genetic algorithms for optimizing the neural architecture BID24 BID14 BID21 BID18 BID28. In contrast to these works, our proposed method is applicable for multi-objective optimization and employs Lamarckian inheritance, i.e, learned parameters are passed on to a network's offspring. A related approach to our Lamarckian evolution is population-based training BID9, which, however, focuses on hyperparameter optimization and not on the specific properties of the optimization of neural architectures. We note that it would be possible to also include the evolution of hyperparameters in our work. Unfortunately, most of the aforementioned approaches require vast computational resources since they need to train and validate thousands of neural architectures; e.g., BID32 trained over 10.000 neural architectures, requiring thousands of GPU days. One way of speeding up evaluation is to predict performance of a (partially) trained model (; b; BID11 BID13 . Works on performance prediction are complementary to our work and could be incorporated in the future. One-Shot Architecture Search is another promising approach for speeding up performance estimation, which treats all architectures as different subgraphs of a supergraph (the one-shot model) and shares weights between architectures BID23; BID19 BID15 ). Only the weights of a single one-shot model need to be trained, and architectures (which are just subgraphs of the one-shot model) can then be evaluated without any separate training. However, a general limitation of one-shot NAS is that the supergraph defined a-priori restricts the search space to its subgraphs. Moreover, approaches which require that the entire supergraph resides in GPU memory during architecture search will be restricted to relatively small supergraphs. It is also not obvious how one-shot models could be employed for multi-objective optimization as all subgraphs of the one-shot models are of roughly the same size and it is not clear if weight sharing would work for very different-sized architectures. LEMONADE does not suffer from any of these disadvantages; it can handle arbitrary large, unconstrained search spaces while still being efficient.; Cai et al. (2018a) proposed to employ the concept of network morphisms (see Section 3.1). The basic idea is to initialize weights of newly generated neural architectures based on weights of similar, already trained architectures so that they have the same accuracy. This pretrained initialization allows reducing the large cost of training all architectures from scratch. Our work extends this approach by introducing approximate network morphisms, making the use of such operators suitable for multi-objective optimization. Multi-objective Neural Architecture Search Very recently, there has also been some work on multi-objective neural architecture search BID10; BID26 with the goal of not solely optimizing the accuracy on a given task but also considering resource consumption. BID10 parameterize an architecture by a fixed-length vector description, which limits the architecture search space drastically. In parallel, independent work to ours, extend PNAS BID13 by considering multiple objective during the model selection step. However, they employ CondenseNet BID6 as a base network and solely optimize building blocks within the network which makes the search less interesting as (i) the base network is by default already well performing and (ii) the search space is again limited. BID26 use a weighted product method to obtain a scalarized objective. However, this scalarization comes with the drawback of weighting the objectives a-priori, which might not be suitable for certain applications. In contrast to all mentioned work, LEMONADE (i) does not require a complex macro architecture but rather can start from trivial initial networks, (ii) can handle arbitrary search spaces, (iii) does not require to define hard constraints or weights on objectives a-priori. Let N (X) denote a space of neural networks, where each element N ∈ N (X) is a mapping from X ⊂ R n to some other space, e.g., mapping images to labels. A network operator T: DISPLAYFORM0 We now discuss two specific classes of network operators, namely network morphisms and approximate network morphisms. Operators from these two classes will later on serve as mutations in our evolutionary algorithm. Chen et al. FORMULA6 introduced two function-preserving operators for deepening and widening a neural network. BID27 built upon this work, dubbing function-preserving operators on neural networks network morphisms. Formally, a network morphism is a network operator satisfying N w (x) = (T N)w(x) for every x ∈ X, i.e., N w and (T N)w represent the same function. This can be achieved by properly initializingw. We now describe the operators used in LEMONADE and how they can be formulated as a network morphism. We refer to Appendix A.1.1 for details.1. Inserting a Conv-BatchNorm-ReLU block. We initialize the convolution to be an identity mapping, as done by Chen et al. FORMULA6 ("Net2DeeperNet"). Offset and scale of BatchNormalization are initialized to be the (moving) batch mean and (moving) batch variance, hence initially again an identity mapping. Since the ReLU activation is idempotent, i.e., ReLU (ReLU (x)) = ReLU (x), we can add it on top of the previous two operations without any further changes, assuming that the block will be added on top of a ReLU layer.2. Increase the number of filters of a convolution. This operator requires the layer to be changed to have a subsequent convolutional layer, whose parameters are padded with 0's. Alternatively, one could use the "Net2WiderNet" operator by.3. Add a skip connection. We allow skip connection either by concatenation BID7 or by addition . In the former case, we again use zero-padding in sub-sequential convolutional layers. In the latter case, we do not simply add two outputs x and y but rather use a convex combination (1 − λ)x + λy, with a learnable parameter λ initialized as 0 (assuming x is the original output and y the output of an earlier layer). One common property of all network morphisms is that they can only increase the capacity of a network 1. This may be a reasonable property if one solely aims at finding a neural architectures with maximal accuracy, but not if one also aims at neural architectures with low resource requirements. Also, decisions once made can not be reverted. Operators like removing a layer could considerably decrease the resources required by the model while (potentially) preserving its performance. Hence, we now generalize the concept of network morphisms to also cover operators that reduce the capacity of a neural architecture. We say an operator T is an approximate network morphism (ANM) with respect to a neural network N w with parameters w if N w (x) ≈ (T N)w(x) for every x ∈ X. We refer to Appendix A.1.2 for a formal definition. In practice we simply determinew so thatÑ approximates N by using knowledge distillation BID3.In our experiments, we employ the following ANM's: (i) remove a randomly chosen layer or a skip connection, (ii) prune a randomly chosen convolutional layer (i.e., remove 1/2 or 1/4 of its filters), and (iii) substitute a randomly chosen convolution by a depthwise separable convolution. Note that these operators could easily be extended by sophisticated methods for compressing neural networks BID8 BID31. LEMONADE maintains a population of trained networks that constitute a Pareto front in the multi-objective space. Parents are selected from the population inversely proportional to their density. Children are generated by mutation operators with Lamarckian inheritance that are realized by network morphisms and approximate network morphisms. NM operators generate children with the same initial error as their parent. In contrast, children generated with ANM operators may incur a (small) increase in error compared to their parent. However, their initial error is typically still very small. (Right) Only a subset of the generated children is accepted for training. After training, the performance of the children is evaluated and the population is updated to be the Pareto front. Algorithm 1 LEMONADE 1: input: P 0, f, n gen, n pc, n ac 2: P ← P 0 3: for i ← 1,..., n gen do 4: DISPLAYFORM0 Compute parent distribution p P (Eq. 1) BID38: DISPLAYFORM1 Compute children distribution p child (Eq. 2) 8: DISPLAYFORM2 Evaluate f exp for N c ∈ N c ac 10:P ← P aretoF ront(P ∪ N c ac, f) 11: end for 12: return P In this section, we propose a Lamarckian Evolutionary algorithm for MultiObjective Neural Architecture DEsign, dubbed LEMONADE. We refer to FIG1 for an illustration as well as Algorithm 1 for pseudo code. LEMONADE aims at minimizing multiple objectives DISPLAYFORM3 denote expensive-to-evaluate objectives (such as the validation error or some measure only be obtainable by expensive simulation) and its other components f cheap (N) ∈ R n denote cheap-to-evaluate objectives (such as model size) that one also tries to minimize. LEMONADE maintains a population P of parent networks, which we choose to comprise all non-dominated networks with respect to f, i.e., the current approximation of the Pareto front 2. In every iteration of LEMONADE, we first sample parent networks with respect to some probability distribution based on the cheap objectives and generate child networks by applying network operators (described in Section 3). In a second sampling stage, we sample a subset of children, again based on cheap objectives, and solely this subset is evaluated on the expensive objectives. Hence, we exploit that f cheap is cheap to evaluate in order to bias both sampling processes towards areas of f cheap that are sparsely populated. We thereby evaluate f cheap many times in order to end up with a diverse set of children in sparsely populated regions of the objective space, but evaluate f exp only a few times. More specifically, LEMONADE first computes a density estimator p KDE (e.g., in our case, a kernel density estimator) on the cheap objective values of the current population, {f cheap (N)|N ∈ P}. Note that we explicitly only compute the KDE with respect to f cheap rather than f as this allows to evaluate p KDE (f cheap (N)) very quickly. Then, larger number n pc of proposed children N c pc = {N c 1, . . ., N c npc} is generated by applying network operators, where the parent N for each child is sampled according to a distribution inversely proportional to p KDE, DISPLAYFORM4 with a normalization constant c = N ∈P 1/p KDE (f cheap (N)) −1. Since children have similar objective values as their parents (network morphisms do not change architectures drastically), this sampling distribution of the parents is more likely to also generate children in less dense regions of f cheap. Afterwards, we again employ p KDE to sample a subset of n ac accepted children N c ac ⊂ N c pc. The probability of a child being accepted is DISPLAYFORM5 withĉ being another normalization constant. Only these accepted children are evaluated according to f exp. By this two-staged sampling strategy we generate and evaluate more children that have the potential to fill gaps in f. We refer to the ablation study in Appendix A.2.2 for an empirical comparison of this sampling strategy to uniform sampling. Finally, LEMONADE computes the Pareto front from the current generation and the generated children, yielding the next generation. The described procedure is repeated for a prespecified number of generations (100 in our experiments). We present for LEMONADE on searching neural architectures for CIFAR-10. We ran LEMONADE with three different settings: (i) we optimize 5 objectives and search for entire architectures (Section 5.1), (ii) we optimize 2 objectives and search for entire architectures (Appendix A.2), and (iii) we optimize 2 objectives and search for cells (Section 5.2, Appendix A.2). We also transfer the discovered cells from the last setting to ImageNet (Section 5.4) and its down-scaled version ImageNet64x64 (Section 5.3). All experimental details, such as a description of the search spaces and hyperparameters can be found in Appendix A.3.The progress of LEMONADE for setting (ii) is visualized in FIG2. The Pareto front improves over time, reducing the validation error while covering a wide regime of, e.g., model parameters, ranging from 10 000 to 10 000 000. We aim at solving the following multi-objective problem: minimize the five objectives (i) performance on CIFAR-10 (expensive objective), (ii) performance on CIFAR-100 (expensive), (iii) number of parameters (cheap), (iv) number of multiply-add operations (cheap) and (v) inference time 3 (cheap). We think having five objectives is a realistic scenario for most NAS applications. Note that one could easily use other, more sophisticated measures for resource efficiency. In this experiment, we search for entire neural network architectures (denoted as Search Space I, see Appendix A.3.2 for details) instead of convolutional cells (which we will do in a later experiment). LEMONADE natively handles this unconstrained, arbitrarily large search space, whereas other methods are by design a-priori restricted to relatively small search spaces (; BID15 . Also, LEMONADE is initialized with trivial architectures (see Appendix A.3.2) rather than networks that already yield state-of-the-art performance (b;). The set of operators to generate child networks we consider in our experiments are the three network morphism operators (insert convolution, insert skip connection, increase number of filters), as well as the three approximate network morphism operators (remove layer, prune filters, replace layer) described in Section 3. The operators are sampled uniformly at random to generate children. The experiment ran for approximately 5 days on 16 GPUs in parallel. The ing Pareto front consists of approximately 300 neural network architectures. We compare against different-sized NASNets and MobileNets V2 BID22. In order to ensure that differences in test error are actually caused by differences in the discovered architectures rather than different training conditions, we retrained all architectures from scratch using exactly the same optimization pipeline with the same hyperparameters. We do not use stochastic regularization techniques, such as Shake-Shake or ScheduledDropPath in this experiment as they are not applicable to all networks out of the box. The are visualized in FIG3. As one would expect, the performance on CIFAR-10 and CIFAR-100 is highly correlated, hence the ing Pareto fronts only consist of a few elements and differences are rather small (top left). When considering the performance on CIFAR-10 versus the number of parameters (top right) or multiply-add operations (bottom left), LEMONADE is on par with NASNets and MobileNets V2 for resource-intensive models while it outperforms them in the area of very efficient models (e.g., less than 100,000 parameters). In terms of inference time (bottom right), LEMONADE clearly finds models superior to the baselines. We highlight that this has been achieved based on using only 80 GPU days for LEMONADE compared to 2000 in and with a significantly more complex Search Space I (since the entire architecture was optimized and not only a convolutional cell).We refer to Appendix A.2 for an experiment with additional baselines (e.g., random search) and an ablation study. Above, we compared different models when trained with the exact same data augmentation and training pipeline. We now also briefly compare LEMONADE's performance to reported in the literature. We apply two widely used methods to improve over the training pipeline used above: (i) instead of searching for entire architectures, we search for cells that are employed within a hand-crafted macro architecture, meaning one replaces repeating building blocks in the architecture with discovered cells (b;) and (ii) using stochastic regularization techniques, such as ScheduledDropPath during training BID19 b). In our case, we run LEMONADE to search for cells within the ShakeShake macro architecture (i.e., we replace basic convolutional blocks with cells) and also use ShakeShake regularization . Table 1. LEMONADE is on par or outperforms DPP-Net across all parameter regimes. As all other methods solely optimize for accuracy, they do not evaluate models with few parameters. However, also for larger models, LEMONADE is competitive to methods that require significantly more computational resources or start their search with non-trivial architectures (b;). To study the transferability of the discovered cells to a different dataset (without having to run architecture search itself on the target dataset), we built architectures suited for ImageNet64x64 based on five cells discovered on CIFAR-10. We vary the number of cells per block and the number of filters in the last block to obtain different architectures for a single cell (as done by for NASNets). We compare against different sized MobileNets V2, NASNets and Wide Residual Networks (WRNs) BID29. For direct comparability, we again train all architectures in the same way. In FIG4, we plot the Pareto Front from all cells combined, as well as the Pareto Front from a single cell, Cell 2, against the baselines. Both clearly dominate NASNets, WRNs and MobileNets V2 over the entire parameter range, showing that a multi-objective search again is beneficial. We also evaluated one discovered cell, Cell 2, on the regular ImageNet benchmark for the "mobile setting" (i.e., networks with 4M to 6M parameters and less than 600M multiply-add operations). The cell found by LEMONADE achieved a top-1 error of 28.3% and a top-5 error of 9.6%; this is slightly worse than published for, e.g., NASNet (26% and 8.4%, respectively) but still competitive, especially seeing that (due to time and resource constraints), we used an off-the-shelf training pipeline, on a single GPU (for four weeks), and did not alter any hyperparameters. We believe that our cell could perform substantially better with a better optimization pipeline and properly tuned hyperparameters (as in many other NAS papers by authors with more compute resources). We have proposed LEMONADE, a multi-objective evolutionary algorithm for architecture search. The algorithm employs a Lamarckian inheritance mechanism based on (approximate) network morphism operators to speed up the training of novel architectures. Moreover, LEMONADE exploits the fact that evaluating several objectives, such as the performance of a neural network, is orders of magnitude more expensive than evaluating, e.g., a model's number of parameters. Experiments on CIFAR-10 and ImageNet64x64 show that LEMONADE is able to find competitive models and cells both in terms of accuracy and of resource efficiency. We believe that using more sophisticated concepts from the multi-objective evolutionary algorithms literature and using other network operators (e.g., crossovers and advanced compression methods) could further improve LEMONADE's performance in the future. In the following two subsections we give some detailed information on the network morphisms and approximate network morphisms employed in our work. A network morphism is a network operator satisfying the network morphism equation: DISPLAYFORM0 withw i = (w i, A, b). The network morphism equation FORMULA8 then holds for A = 1, b = 0. This morphism can be used to add a fully-connected or convolutional layer, as these layers are simply linear mappings. dubbed this morphism "Net2DeeperNet". Alternatively to the above replacement, one could also choosẽ DISPLAYFORM1 DISPLAYFORM2 A, b are fixed, non-learnable. In this case, network morphism Equation (DISPLAYFORM3 with an arbitrary functionh wh (x). The new parameters arew i = (w i, wh,Ã). Again, Equation can trivially be satisfied by settingÃ = 0. We can think of two modifications of a neural network that can be expressed by this morphism: firstly, a layer can be widened (i.e., increasing the number of units in a fully connected layer or the number of channels in a CNN -the Net2WiderNet transformation of Chen et al. FORMULA6). Let h(x) be the layer to be widened. For example, we can then seth = h to simply double the width. Secondly, skip-connections by concatenation as used by BID5 can also be expressed. If h(x) itself is a sequence of layers, h(x) = h n (x) • · · · • h 0 (x), then one could chooseh(x) = x to realize a skip from h 0 to the layer subsequent to h n.Network morphism Type III. By definition, every idempotent function N wi i can simply be replaced by DISPLAYFORM4 with the initializationw i = w i. This trivially also holds for idempotent functions without weights, e.g., ReLU.Network morphism Type IV. Every layer N wi i is replaceable bỹ Nw DISPLAYFORM5 with an arbitrary function h and Equation FORMULA8 holds if the learnable parameter λ is initialized as 1. This morphism can be used to incorporate any function, especially any non-linearity. For example, BID27 use a special case of this operator to deal with non-linear, non-idempotent activation functions. Another example would be the insertion of an additive skip connection, which were proposed by to the layer subsequent to N wi n in. Note that every combination of network morphisms again yields a network morphism. Hence, one could, for example, add a block "Conv-BatchNorm-ReLU" subsequent to a ReLU layer by using Equations FORMULA9, FORMULA10 and. LEMONADE essentially consists of three components: (i) additionally using approximate network morphism operators to also allow shrinking architectures, (ii) using Lamarckism, i.e., (approximate) network morphisms, to avoid training from scratch, and (iii) the two-staged sampling strategy. In Figure 6, we present for deactivating each of these components one at a time. The shows that all three components improve LEMONADE's performance. In this section we list all the experimental details. Search Space I corresponds to searching for an entire architecture (rather than cells). LEMONADE's Pareto front was initialized to contain four simple convolutional networks with relatively large validation errors of 30 − 50%. All four initial networks had the following structure: three Conv- Figure 6: Ablation study on CIFAR-10. We deactivate different components of LEMONADE and investigate the impact. LEMONADE default: Performance of LEMONADE as proposed in this work. LEMONADE no ANM: we deactivated the approximate network morphisms operators, i.e., networks can only grow in size. LEMONADE no Lamarckism: all networks are initialized from scratch instead by means of (approximate) network morphisms. LEMONADE no KDE: we deactivate the proposed sampling strategy and use uniform sampling of parents and children instead. BatchNorm-ReLU blocks with intermittent Max-Pooling, followed by a global average pooling and a fully-connected layer with softmax activation. The networks differ in the number of channels in the convolutions, and for further diversity two of them used depthwise-separable convolutions. The models had 15 000, 50 000, 100 000 and 400 000 parameters, respectively. For generating children in LEMONADE, we chose the number of operators that are applied to parents uniformly from {1,2,3}.LEMONADE natively handles this unconstrained, arbitrary large search space, whereas other methods are by design restricted a-priori to relatively small search spaces (; BID15 .We restricted the space of neural architectures such that every architecture must contain at least 3 (depthwise separable) convolutions with a minimum number of filters, which lead to a lower bound on the number of parameters of approximately 10 000.The network operators implicitly define the search space, we do not limit the size of discovered architectures. Search Space II consists of convolutional cells that are used within some macro architecture to build the neural network. In the experiments in Section 5, we use cells within the macro architecture of the Shake-Shake architecture , whereas in the baseline experiment in the appendix (Section A.2), we rely on a simpler scheme as in as in BID13, i.e., sequentially stacking cells. We only choose a single operator to generate children, but the operator is applied to all occurrences of the cell in the architecture. The Pareto Front was again initialized with four trivial cells: the first two cells consist of a single convolutional layer (followed by BatchNorm and ReLU) with F = 128 and F = 256 filters in the last block, respectively. The other two cells consist of a single depthwise separable convolution (followed by BatchNorm and ReLU), again with either F = 128 or F = 256 filters. To classify CIFAR-10 with MobileNets V1 and V2, we replaced three blocks with stride 2 with identical blocks with stride 1 to adapt the networks to the lower spatial resolution of the input. We chose the replaced blocks so that there are the same number of stride 1 blocks between all stride 2 blocks. We varied the size of MobileNets V1 and V2 by varying the width multiplier α ∈ {0.1, 0.2, . . ., 1.2} and NASNets by varying the number of cell per block (∈ {2, 4, 6, 8}) and number of filters (∈ {96, 192, 384, 768, 1536}) in the last block. We apply the standard data augmentation scheme described by BID16, as well as the recently proposed methods mixup BID30 and Cutout . The training set is split up in a training (45.000) and a validation (5.000) set for the purpose of architecture search. We use weight decay (5 · 10 −4) for all models. We use batch size 64 throughout all experiments. During architecture search as well as for generating the random search baseline, all models are trained for 20 epochs using SGD with cosine annealing BID16, decaying the learning rate from 0.01 to 0. For evaluating the test performance, all models are trained from scratch on the training and validation set with the same setup as described above except for 1) we train for 600 epochs and 2) the initial learning rate is set to 0.025. While searching for convolutional cells on CIFAR-10, LEMONADE ran for approximately 56 GPU days. However, there were no significant changes in the Pareto front after approximately 24 GPU days. The training setup (both during architecture search and final evaluation) is exactly the same as before. The training setup on ImageNet64x64 is identical to. Below we list some additional figures.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose a method for efficient Multi-Objective Neural Architecture Search based on Lamarckian inheritance and evolutionary algorithms.

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the article for finding a single-sentence TLDR

Question: We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P(wordjcontext): No Language Structure in Context: Probabilistic topic models ignore word order by summarizing a given context as a “bag-of-word” and consequently the semantics of words in the context is lost. In this work, we incorporate language structure by combining a neural autoregressive topic model (TM) with a LSTM based language model (LSTM-LM) in a single probabilistic framework. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns, while the TM simultaneously learns a latent representation from the entire document. In addition, the LSTM-LM models complex characteristics of language (e.g., syntax and semantics), while the TM discovers the underlying thematic structure in a collection of documents. We unite two complementary paradigms of learning the meaning of word occurrences by combining a topic model and a language model in a unified probabilistic framework, named as ctx-DocNADE. Limited Context and/or Smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the wordtopic mapping on a smaller and/or short-text corpus. The proposed DocNADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural language models and embeddings priors that consistently outperform state-of-theart generative topic models in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 6 long-text and 8 short-text datasets from diverse domains. Probabilistic topic models, such as LDA BID1, Replicated Softmax (RSM) and Document Neural Autoregressive Distribution Estimator (DocNADE) variants BID12 BID34 BID15 BID8 are often used to extract topics from text collections, and predict the probabilities of each word in a given document belonging to each topic. Subsequently, they learn latent document representations that can be used to perform natural language processing (NLP) tasks such as information retrieval (IR), document classification or summarization. However, such probabilistic topic models ignore word order and represent a given context as a bag of its words, thereby disregarding semantic information. To motivate our first task of extending probabilistic topic models to incorporate word order and language structure, assume that we conduct topic analysis on the following two sentences: When estimating the probability of a word in a given context (here: P ("bear"|context)), traditional topic models do not account for language structure since they ignore word order within the context and are based on "bag-of-words" (BoWs) only. In this particular setting, the two sentences have the same unigram statistics, but are about different topics. On deciding which topic generated the word "bear" in the second sentence, the preceding words "market falls" make it more likely that it was generated by a topic that assigns a high probability to words related to stock market trading, where "bear territory" is a colloquial expression in the domain. In addition, the language structure (e.g., syntax and semantics) is also ignored. For instance, the word "bear" in the first sentence is a proper noun and subject while it is an object in the second. In practice, topic models also ignore functional words such as "into", which may not be appropriate in some scenarios. Recently, BID23 have shown that a deep contextualized LSTM-based language model (LSTM-LM) is able to capture different language concepts in a layer-wise fashion, e.g., the lowest layer captures language syntax and topmost layer captures semantics. However, in LSTM-LMs the probability of a word is a function of its sentence only and word occurrences are modeled in a fine granularity. Consequently, LSTM-LMs do not capture semantics at a document level. To this end, recent studies such as TDLM BID14, Topic-RNN and TCNLM BID32 have integrated the merits of latent topic and neural language models (LMs); however, they have focused on improving LMs with global (semantics) dependencies using latent topics. Similarly, while bi-gram LDA based topic models BID31 BID33 and n-gram based topic learning BID15 can capture word order in short contexts, they are unable to capture long term dependencies and language concepts. In contrast, DocNADE variants BID12 BID8 ) learns word occurrences across documents i.e., coarse granularity (in the sense that the topic assigned to a given word occurrence equally depends on all the other words appearing in the same document); however since it is based on the BoW assumption all language structure is ignored. In language modeling, BID17 have shown that recurrent neural networks in a significant reduction of perplexity over standard n-gram models. Contribution 1: We introduce language structure into neural autoregressive topic models via a LSTM-LM, thereby accounting for word ordering (or semantic regularities), language concepts and long-range dependencies. This allows for the accurate prediction of words, where the probability of each word is a function of global and local (semantics) contexts, modeled via DocNADE and LSTM-LM, respectively. The proposed neural topic model is named as contextualized-Document Neural Autoregressive Distribution Estimator (ctx-DocNADE) and offers learning complementary semantics by combining joint word and latent topic learning in a unified neural autoregressive framework. For instance, FIG0 (left and middle) shows the complementary topic and word semantics, based on TM and LM representations of the term "fall". Observe that the topic captures the usage of "fall" in the context of stock market trading, attributed to the global (semantic) view. While this is a powerful approach for incorporating language structure and word order in particular for long texts and corpora with many documents, learning from contextual information remains challenging in settings with short texts and few documents, since limited word co-occurrences or little context significant word non-overlap in such short texts and small training corpus of documents lead to little evidence for learning word co-occurrences. However, distributional word representations (i.e. word embeddings) BID22 have shown to capture both the semantic and syntactic relatedness in words and demonstrated impressive performance in NLP tasks. For example, assume that we conduct topic analysis over the two short text fragments: Deal with stock index falls and Brace for market share drops. Traditional topic models with "BoW" assumption will not be able to infer relatedness between word pairs such as (falls, drops) due to the lack of word-overlap and small context in the two phrases. However, in the distributed embedding space, the word pairs are semantically related as shown in FIG0 (left). DISPLAYFORM0 Related work such as BID26 employed web search to improve the information in short texts and BID24 introduced word similarity via thesauri and dictionaries into LDA. BID5 and BID20 integrated word embeddings into LDA and Dirichlet Multinomial Mixture (DMM) BID21 models. Recently, BID8 extends DocNADE by introducing pre-trained word embeddings in topic learning. However, they ignore the underlying language structure, e.g., word ordering, syntax, etc. In addition, DocNADE and its extensions outperform LDA and RSM topic models in terms of perplexity and IR.Contribution 2: We incorporate distributed compositional priors in DocNADE: we use pre-trained word embeddings via LSTM-LM to supplement the multinomial topic model (i.e., DocNADE) in learning latent topic and textual representations on a smaller corpus and/or short texts. Knowing similarities in a distributed space and integrating this complementary information via a LSTM-LM, a topic representation is much more likely and coherent. Taken together, we combine the advantages of complementary learning and external knowledge, and couple topic-and language models with pre-trained word embeddings to model short and long text documents in a unified neural autoregressive framework, named as ctx-DocNADEe. Our approach learns better textual representations, which we quantify via generalizability (e.g., perplexity), interpretability (e.g., topic extraction and coherence) and applicability (e.g., IR and classification).To illustrate our two contributions, we apply our modeling approaches to 7 long-text and 8 short-text datasets from diverse domains and demonstrate that our approach consistently outperforms state-ofthe-art generative topic models. Our learned representations, in a gain of: 4.6% (.790 vs .755) in topic coherence, 6.5% (.615 vs .577) in precision at retrieval fraction 0.02, and 4.4% (.662 vs .634) in F 1 for text classification, averaged over 6 long-text and 8 short-text datasets. When applied to short-text and long-text documents, our proposed modeling approaches generate contextualized topic vectors, which we name textTOvec. The code is available at https: //github.com/pgcool/textTOvec. Generative models are based on estimating the probability distribution of multidimensional data, implicitly requiring modeling complex dependencies. Restricted Boltzmann Machine (RBM) BID9 and its variants BID11 are probabilistic undirected models of binary data. RSM and its variants BID7 are generalization of the RBM, that are used to model word counts. However, estimating the complex probability distribution of the underlying high-dimensional observations is intractable. To address this challenge, NADE BID13 decomposes the joint distribution of binary observations into autoregressive conditional distributions, each modeled using a feed-forward network. Unlike for RBM/RSM, this leads to tractable gradients of the data negative log-likelihood. An extension of NADE and RSM, DocNADE BID12 ) models collections of documents as orderless bags of words (BoW approach), thereby disregarding any language structure. In other words, it is trained to learn word representations reflecting the underlying topics of the documents only, ignoring syntactical and semantic features as those encoded in word embeddings BID0 BID18 BID22 BID23. DocNADE BID15 DISPLAYFORM0, where each autoregressive conditional p(v i |v <i) for the word observation v i is computed using the preceding observations v <i ∈ {v 1, ..., v i−1} in a feed-forward neural network for i ∈ {1, ...D}, DISPLAYFORM1 where g(·) is an activation function, U ∈ R K×H is a weight matrix connecting hidden to output, e ∈ R H and b ∈ R K are bias vectors, W ∈ R H×K is a word representation matrix in which a column W:,vi is a vector representation of the word v i in the vocabulary, and H is the number of hidden units (topics). The log-likelihood of any document v of any arbitrary length is given by: DISPLAYFORM2 Note that the past word observations v <i are orderless due to BoWs, and may not correspond to the words preceding the ith word in the document itself. Input: A training document v Input: Word embedding matrix E Output: log p(v) 1: a ← e 2: q(v) = 1 3: for i from 1 to D do 4:compute hi and p(vi|v<i) 5: Table 1: Computation of hi and p(vi|v<i) in DocNADE, ctx-DocNADE and ctx-DocNADEe models, correspondingly used in estimating log p(v) (Algorithm 1). DISPLAYFORM0 We propose two extensions of the DocNADE model: ctx-DocNADE: introducing language structure via LSTM-LM and ctx-DocNADEe: incorporating external knowledge via pre-trained word embeddings E, to model short and long texts. The unified network(s) account for the ordering of words, syntactical and semantic structures in a language, long and short term dependencies, as well as external knowledge, thereby circumventing the major drawbacks of BoW-based representations. Similar to DocNADE, ctx-DocNADE models each document v as a sequence of multinomial observations. Let [x 1, x 2, ..., x N] be a sequence of N words in a given document, where x i is represented by an embedding vector of dimension, dim. Further, for each element v i ∈ v, let c i = [x 1, x 2, ..., x i−1] be the context (preceding words) of ith word in the document. Unlike in DocNADE, the conditional probability of the word v i in ctx-DocNADE (or ctx-DocNADEe) is a function of two hidden vectors: LSTM-based components of ctx-DocNADE, respectively: DISPLAYFORM0 DISPLAYFORM1 where h In the weight matrix W of DocNADE BID12, each row vector W j,: encodes topic information for the jth hidden topic feature and each column vector W:,vi is a vector for the word v i. To obtain complementary semantics, we exploit this property and expose W to both global and local influences by sharing W in the DocNADE and LSTM-LM componenents. Thus, the embedding layer of LSTM-LM component represents the column vectors. DISPLAYFORM2 ctx-DocNADE, in this realization of the unified network the embedding layer in the LSTM component is randomly initialized. This extends DocNADE by accounting for the ordering of words and language concepts via context-dependent representations for each word in the document.ctx-DocNADEe, the second version extends ctx-DocNADE with distributional priors, where the embedding layer in the LSTM component is initialized by the sum of a pre-trained embedding matrix E and the weight matrix W. Note that W is a model parameter; however E is a static prior. Algorithm 1 and Table 1 show the log p(v) for a document v in three different settings: Doc-NADE, ctx-DocNADE and ctx-DocNADEe. In the DocNADE component, since the weights in the matrix W are tied, the linear activation a can be re-used in every hidden layer and computational complexity reduces to O(HD), where H is the size of each hidden layer. In every epoch, we run an LSTM over the sequence of words in the document and extract hidden vectors h LM i, corresponding to c i for every target word v i. Therefore, the computational complexity in ctx-DocNADE or ctx-DocNADEe is O(HD + N), where N is the total number of edges in the LSTM network BID10 BID27. The trained models can be used to extract a textTOvec representation, i.e., h(v DISPLAYFORM3 ctx-DeepDNEe: DocNADE and LSTM can be extended to a deep, multiple hidden layer architecture by adding new hidden layers as in a regular deep feed-forward neural network, allowing for improved performance. In the deep version, the first hidden layer is computed in an analogous fashion to DocNADE variants (equation 1 or 2). Subsequent hidden layers are computed as: DISPLAYFORM4 where n is the total number of hidden layers (i.e., depth) in the deep feed-forward and LSTM networks. For d=1, the hidden vectors h DN i,1 and h LM i,1 correspond to equations 1 and 2. The conditional p(v i = w|v <i) is computed using the last layer n, i.e., h i,n = h TAB10: State-of-the-art comparison: IR (i.e, IR-precision at 0.02 fraction) and classification F 1 for short texts, where Avg: average over the row values, the bold and underline: the maximum for IR and F1, respectively. DISPLAYFORM5 We apply our modeling approaches (in improving topic models, i.e, DocNADE using language concepts from LSTM-LM) to 8 short-text and 7 long-text datasets of varying size with single/multiclass labeled documents from public as well as industrial corpora. We present four quantitative measures in evaluating topic models: generalization (perplexity), topic coherence, text retrieval and categorization. See the appendices for the data description and example texts. TAB1 shows the data statistics, where 20NS and R21578 signify 20NewsGroups and Reuters21578, respectively. Baselines: While, we evaluate our multi-fold contributions on four tasks: generalization (perplexity), topic coherence, text retrieval and categorization, we compare performance of our proposed models ctx-DocNADE and ctx-DocNADEe with related baselines based on: word representation: glove BID22, where a document is represented by summing the embedding vectors of it's words, document representation: doc2vec , LDA based BoW TMs: ProdLDA and SCHOLAR 1 BID3 ) neural BoW TMs: DocNADE and NTM BID2 and, TMs, including pre-trained word embeddings: Gauss-LDA (GaussianLDA) BID5, and glove-DMM, glove-LDA BID20. jointly 2 trained topic and language models: TDLM , Topic-RNN and TCNLM BID32. Experimental Setup: DocNADE is often trained on a reduced vocabulary (RV) after pre-processing (e.g., ignoring functional words, etc.); however, we also investigate training it on full text/vocabulary (FV) (TAB1) and compute document representations to perform different evaluation tasks. The FV setting preserves the language structure, required by LSTM-LM, and allows a fair comparison of DocNADE+FV and ctx-DocNADE variants. We use the glove embedding of 200 dimensions. All the baselines and proposed models (ctx-DocNADE, ctx-DocNADEe and ctx-DeepDNEe) were run in the FV setting over 200 topics to quantify the quality of the learned representations. To better initialize the complementary learning in ctx-DocNADEs, we perform a pre-training for 10 epochs with λ set to 0. See the appendices for the experimental setup and hyperparameters for the following tasks, including the ablation over λ on validation set. BID14 for all the short-text datasets to evaluate the quality of representations learned in the spare data setting. For a fair comparison, we set 200 topics and hidden size, and initialize with the same pre-trained word embeddings (i.e., glove) as used in the ctx-DocNADEe. DISPLAYFORM0 To evaluate the generative performance of the topic models, we estimate the log-probabilities for the test documents and compute the average held-out perplexity (P P L) per word as, P P L = exp − 1 z z t=1 DISPLAYFORM0, where z and |v t | are the total number of documents and words in a document v t. For DocNADE, the log-probability log p(v t) is computed using L DN (v); however, we ignore the mixture coefficient, i.e., λ=0 (equation 2) to compute the exact log-likelihood in ctx-DocNADE versions. The optimal λ is determined based on the validation set. TAB5 quantitatively shows the PPL scores, where the complementary learning with λ = 0.01 (optimal) in ctx-DocNADE achieves lower perplexity than the baseline DocNADE for both short and long texts, e.g., (822 vs 846) and (1358 vs 1375) on AGnewstitle and 20NS 4 datasets, respectively in the FV setting. We compute topic coherence BID4 BID19 BID7 to assess the meaningfulness of the underlying topics captured. We choose the coherence measure proposed by BID25, which identifies context features for each topic word using a sliding window over the reference corpus. Higher scores imply more coherent topics. We use the gensim module (radimrehurek.com/gensim/models/coherencemodel.html, coherence type = c v) to estimate coherence for each of the 200 topics (top 10 and 20 words). TAB7 shows average coherence over 200 topics, where the higher scores in ctx-DocNADE compared to DocNADE (.772 vs .755) suggest that the contextual information and language structure help in generating more coherent topics. The introduction of embeddings in ctx-DocNADEe boosts the topic coherence, leading to a gain of 4.6% (.790 vs .755) on average over 11 datasets. Note that the proposed models also outperform the baselines methods glove-DMM and glove-LDA. Qualitatively, Table 8 illustrates an example topic from the 20NSshort text dataset for DocNADE, ctx-DocNADE and ctx-DocNADEe, where the inclusion of embeddings in a more coherent topic. Additional Baslines: We further compare our proposed models to other approaches that combining topic and language models, such as TDLM , Topic-RNN and TCNLM BID32. However, the related studies focus on improving language models using topic models: in contrast, the focus of our work is on improving topic models for textual representations (short-text or long-text documents) by incorporating language concepts (e.g., word ordering, syntax, semantics, etc.) and external knowledge (e.g., word embeddings) via neural language models, as discussed in section 1.To this end, we follow the experimental setup of the most recent work, TCNLM and quantitatively compare the performance of our models (i.e., ctx-DocNADE and ctx-DocNADEe) in terms of topic coherence (NPMI) on BNC dataset. The sliding window is one of the hyper-parameters for computing topic coherence BID25 BID32. A sliding window of 20 is used in TCNLM; in addition we also present for a window of size 110. λ is the mixture weight of the LM component in the topic modeling process, and (s) and (l) indicate small and large model, respectively. The symbol'-' indicates no , since word embeddings of 150 dimensions are not available from glove vectors. (Right): The top 5 words of seven learnt topics from our models and TCNLM. The asterisk (*) indicates our proposed models and (#) taken from TCNLM BID32.ues of λ illustrates the relevance of the LM component for topic coherence (DocNADE corresponds to λ=0). Similarly, the inclusion of word embeddings (i.e., ctx-DocNADEe) in more coherent topics than the baseline DocNADE. Importantly, while ctx-DocNADEe is motivated by sparse data settings, the BNC dataset is neither a collection of short-text nor a corpus of few documents. Consequently, ctx-DocNADEe does not show improvements in topic coherence over ctx-DocNADE.In TAB8 (right), we further qualitatively show the top 5 words of seven topics (topic name summarized by BID32) from TCNML and our models. Observe that ctx-DocNADE captures a topic expression that is a collection of only verbs in the past participle. Since the BNC dataset is unlabeled, we are here restricted to comparing model performance in terms of topic coherence only. Text Retrieval: We perform a document retrieval task using the short-text and long-text documents with label information. We follow the experimental setup similar to BID15, where all test documents are treated as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity measure between their textTOvec representations (section 2.2). To compute retrieval precision for each fraction (e.g., 0.0001, 0.005, 0.01, 0.02, 0.05, etc.), we average the number of retrieved training documents with the same label as the query. For multi-label datasets, we average the precision scores over multiple labels for each query. Since, BID28 and BID15 have shown that RSM and DocNADE strictly outperform LDA on this task, we solely compare DocNADE with our proposed extensions. TAB10 and 4 show the retrieval precision scores for the short-text and long-text datasets, respectively at retrieval fraction 0.02. Observe that the introduction of both pre-trained embeddings and language/contextual information leads to improved performance on the IR task noticeably for short texts. We also investigate topic modeling without pre-processing and filtering certain words, i.e. the FV setting and find that the DocNADE(FV) or glove(FV) improves IR precision over the baseline RV setting. Therefore, we opt for the FV in the proposed extensions. On an average over the 8 shorttext and 6 long-text datasets, ctx-DocNADEe reports a gain of 7.1% (.630 vs .588) (in IR-precision. In addition, the deep variant (d=3) with embeddings, i.e., ctx-DeepDNEe shows competitive performance on TREC6 and Subjectivity datasets. FIG4, 3d, 3e and 3f) illustrate the average precision for the retrieval task on 6 datasets. Observe that the ctx-DocNADEe outperforms DocNADE(RV) at all the fractions and demonstrates a gain of 6.5% (.615 vs .577) in precision at fraction 0.02, averaged over 14 datasets. Additionally, our proposed models outperform TDLM and ProdLDA 6 (for 20NS) by noticeable margins. We perform text categorization to measure the quality of our textTovec representations. We consider the same experimental setup as in the retrieval task and extract textTOvec of 200 dimension for each document, learned during the training of ctx-DocNADE variants. To perform text categorization, we employ a logistic regression classifier with L2 regularization. While, ctx-DocNADEe and ctx-DeepDNEe make use of glove embeddings, they are evaluated against the topic model baselines with embeddings. For the short texts TAB10, the glove leads DocNADE in classification performance, suggesting a need for distributional priors in the topic model. Therefore, the ctx-DocNADEe reports a gain of 4.8% (.705 vs .673) and 3.6%(.618 vs .596) in F 1, compared to DocNADE(RV) on an average over the short TAB10 and long TAB4 texts, respectively. In , a gain of 4.4% (.662 vs .634) overall. In terms of classification accuracy on 20NS dataset, the scores are: DocNADE (0.734), ctxDocNADE (0.744), ctx-DocNADEe (0.751), NTM (0.72) and SCHOLAR (0.71). While, our proposed models, i.e., ctx-DocNADE and ctx-DocNADEe outperform both NTM ( taken from BID2, FIG1) and SCHOLAR ( taken from BID3, TAB1), the DocNADE establishes itself as a strong neural topic model baseline. To further interpret the topic models, we analyze the meaningful semantics captured via topic extraction. Table 8 shows a topic extracted using 20NS dataset that could be interpreted as computers, which are (sub)categories in the data, confirming that meaningful topics are captured. Observe that the ctx-DocNADEe extracts a more coherent topic due to embedding priors. To qualitatively inspect the contribution of word embeddings and textTOvec representations in topic models, we analyse the text retrieved for each query using the representations learned from DocNADE and ctxDoocNADEe models. Table 9 illustrates the retrieval of the top 3 texts for an input query, selected from TMNtitle dataset, where #match is YES if the query and retrievals have the same class label. Observe that ctx-DocNADEe retrieves the top 3 texts, each with no unigram overlap with the query. vga, screen, computer, color, svga, graphics computer, sell, screen, offer, bar, macintosh, color, powerbook, vga, card, san, windows, sold, cars, terminal, forsale, utility, monitor, svga, offer gov, vesa computer, processor.554.624.667 Table 8: A topic of 20NS dataset with coherence -DocNADEe Query:: "emerging economies move ahead nuclear plans" #match ctx-#IR1:: imf sign lifting japan yen YES #IR2:: japan recovery takes hold debt downgrade looms YES #IR3:: japan ministers confident treasuries move YES DocNADE #IR1:: nuclear regulator back power plans NO #IR2:: defiant iran plans big rise nuclear NO #IR3:: japan banks billion nuclear operator sources YES Table 9: Illustration of the top-3 retrievals for an input query Additionally, we show the quality of representations learned at different fractions (20%, 40%, 60%, 80%, 100%) of training set from TMNtitle data and use the same experimental setup for the IR and classification tasks, as in section 3.3. In FIG5, we quantify the quality of representations learned and demonstrate improvements due to the proposed models, i.e., ctx-DocNADE and ctx-DocNADEe over DocNADE at different fractions of the training data. Observe that the gains in both the tasks are large for smaller fractions of the datasets. For instance, one of the proposed models, i.e., ctxDocNADEe (vs DocNADE) reports: a precision (at 0.02 fraction) of 0.580 vs 0.444 at 20% and 0.595 vs 0.525 at 100% of the training set, and an F1 of 0.711 vs 0.615 at 20% and 0.726 vs 0.688 at 100% of the training set. Therefore, the findings conform to our second contribution of improving topic models with word embeddings, especially in the sparse data setting. DISPLAYFORM0 In this work, we have shown that accounting for language concepts such as word ordering, syntactic and semantic information in neural autoregressive topic models helps to better estimate the probability of a word in a given context. To this end, we have combined a neural autoregressive topic-(i.e., DocNADE) and a neural language (e.g., LSTM-LM) model in a single probabilistic framework with an aim to introduce language concepts in each of the autoregressive steps of the topic model. This facilitates learning a latent representation from the entire document whilst accounting for the local dynamics of the collocation patterns, encoded in the internal states of LSTM-LM. We further augment this complementary learning with external knowledge by introducing word embeddings. Our experimental show that our proposed modeling approaches consistently outperform stateof-the-art generative topic models, quantified by generalization (perplexity), topic interpretability (coherence), and applicability (text retrieval and categorization) on 15 datasets. Label: training Instructors shall have tertiary education and experience in the operation and maintenance of the equipment or sub-system of Plant. They shall be proficient in the use of the English language both written and oral. They shall be able to deliver instructions clearly and systematically. The curriculum vitae of the instructors shall be submitted for acceptance by the Engineer at least 8 weeks before the commencement of any training. Label: maintenance The Contractor shall provide experienced staff for 24 hours per Day, 7 Days per week, throughout the Year, for call out to carry out On-call Maintenance for the Signalling System. Label: cables Unless otherwise specified, this standard is applicable to all cables which include single and multi-core cables and wires, Local Area Network (LAN) cables and Fibre Optic (FO) cables. Label: installation The Contractor shall provide and permanently install the asset labels onto all equipment supplied under this Contract. The Contractor shall liaise and co-ordinate with the Engineer for the format and the content of the labels. The Contractor shall submit the final format and size of the labels as well as the installation layout of the labels on the respective equipment, to the Engineer for acceptance. Label: operations, interlocking It shall be possible to switch any station Interlocking capable of reversing the service into "Auto-Turnaround Operation". This facility once selected shall automatically route Trains into and out of these stations, independently of the ATS system. At stations where multiple platforms can be used to reverse the service it shall be possible to select one or both platforms for the service reversal. TAB10: Perplexity scores for different λ in Generalization task: Ablation over validation set labels are not used during training. The class labels are only used to check if the retrieved documents have the same class label as the query document. To perform document retrieval, we use the same train/development/test split of documents discussed in data statistics (experimental section) for all the datasets during learning. See TAB1 for the hyperparameters in the document retrieval task. We used gensim (https://github.com/RaRe-Technologies/gensim) to train Doc2Vec models for 12 datasets. Models were trained with distributed bag of words, for 1000 iterations using a window size of 5 and a vector size of 500. We used the same split in training/development/test as for training the Doc2Vec models (also same split as in IR task) and trained a regularized logistic regression classifier on the inferred document vectors to predict class labels. In the case of multilabel datasets (R21578,R21578title, RCV1V2), we used a one-vs-all approach. Models were trained with a liblinear solver using L2 regularization and accuracy and macro-averaged F1 score were computed on the test set to quantify predictive power. We used LFTM (https://github.com/datquocnguyen/LFTM) to train glove-DMM and glove-LDA models. Models were trained for 200 iterations with 2000 initial iterations using 200 topics. For short texts we set the hyperparameter beta to 0.1, for long texts to 0.01; the mixture parameter lambda was set to 0.6 for all datasets. The setup for the classification task was the same as for doc2vec; classification was performed using relative topic proportions as input (i.e. we inferred the topic distribution of the training and test documents and used the relative distribution as input Topic coherence (NPMI) using 20 topics: DocNADE (.18) and SCHOLAR (.35), i.e., SCHOLAR BID3 generates more coherence topics than DocNADE, though worse in PPL and text classification (see section 3.3) than DocNADE, ctx-DocNADE and ctx-DocNADEe. IR tasks: Since, SCHOLAR BID3 without meta-data equates to ProdLDA and we have shown in section 3.3 that ProdLDA is worse on IR tasks than our proposed models, therefore one can infer the performance of SCHOLAR on IR task. The experimental above suggest that the DocNADE is better than SCHOLAR in generating good representations for downstream tasks such as information retrieval or classification, however falls behind SCHOLAR in interpretability. The investigation opens up an interesting direction for future research.

Represent the short summary for finding the corresponding scientific article (long)

Unified neural model of topic and language modeling to introduce language structure  in topic models for contextualized topic vectors

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The large memory requirements of deep neural networks strain the capabilities of many devices, limiting their deployment and adoption. Model compression methods effectively reduce the memory requirements of these models, usually through applying transformations such as weight pruning or quantization. In this paper, we present a novel scheme for lossy weight encoding which complements conventional compression techniques. The encoding is based on the Bloomier filter, a probabilistic data structure that can save space at the cost of introducing random errors. Leveraging the ability of neural networks to tolerate these imperfections and by re-training around the errors, the proposed technique, Weightless, can compress DNN weights by up to 496x; with the same model accuracy, this in up to a 1.51x improvement over the state-of-the-art. The continued success of deep neural networks (DNNs) comes with increasing demands on compute, memory, and networking resources. Moreover, the correlation between model size and accuracy suggests that tomorrow's networks will only grow larger. This growth presents a challenge for resource-constrained platforms such as mobile phones and wireless sensors. As new hardware now enables executing DNN inferences on these devices BID0 ), a practical issue that remains is reducing the burden of distributing the latest models especially in regions of the world not using high-bandwidth networks. For instance, it is estimated that, globally, 800 million users will be using 2G networks by 2020 BID11, which can take up to 30 minutes to download just 20MB of data. By contrast, today's DNNs are on the order of tens to hundreds of MBs, making them difficult to distribute. In addition to network bandwidth, storage capacity on resource-constrained devices is limited, as more applications look to leverage DNNs. Thus, in order to support state-of-the-art deep learning methods on edge devices, methods to reduce the size of DNN models without sacrificing model accuracy are needed. Model compression is a popular solution for this problem. A variety of compression algorithms have been proposed in recent years and many exploit the intrinsic redundancy in model weights. Broadly speaking, the majority of this work has focused on ways of simplifying or eliminating weight values (e.g., through weight pruning and quantization), while comparatively little effort has been spent on devising techniques for encoding and compressing. In this paper we propose a novel lossy encoding method, Weightless, based on Bloomier filters, a probabilistic data structure BID5. Bloomier filters inexactly store a function map, and by adjusting the filter parameters, we can elect to use less storage space at the cost of an increasing chance of erroneous values. We use this data structure to compactly encode the weights of a neural network, exploiting redundancy in the weights to tolerate some errors. In conjunction with existing weight simplification techniques, namely pruning and clustering, our approach dramatically reduces the memory and bandwidth requirements of DNNs for over the wire transmission and ondevice storage. Weightless demonstrates compression rates of up to 496× without loss of accuracy, improving on the state of the art by up to 1.51×. Furthermore, we show that Weightless scales better with increasing sparsity, which means more sophisticated pruning methods yield even more benefits. This work demonstrates the efficacy of compressing DNNs with lossy encoding using probabilistic data structures. Even after the same aggressive lossy simplification steps of weight pruning and clustering (see Section 2), there is still sufficient extraneous information left in model weights to allow an approximate encoding scheme to substantially reduce the memory footprint without loss of model accuracy. Section 3 reviews Bloomier filters and details Weightless. State-of-the-art compression using Weightless are presented in Section 4. Finally, in Section 4.3 shows that Weightless scales better as networks become more sparse compared to the previous best solution. Our goal is to minimize the static memory footprint of a neural network without compromising accuracy. Deep neural network weights exhibit ample redundancy, and a wide variety of techniques have been proposed to exploit this attribute. We group these techniques into two categories: methods that modify the loss function or structure of a network to reduce free parameters and methods that compress a given network by removing unnecessary information. The first class of methods aim to directly train a network with a small memory footprint by introducing specialized structure or loss. Examples of specialized structure include low-rank, structured matrices of BID25 and randomly-tied weights of BID6. Examples of specialized loss include teacher-student training for knowledge distillation BID3 BID16 and diversity-density penalties BID27. These methods can achieve significant space savings, but also typically require modification of the network structure and full retraining of the parameters. An alternative approach, which is the focus of this work, is to compress an existing, trained model. This exploits the fact that most neural networks contain far more information than is necessary for accurate inference BID9. This extraneous information can be removed to save memory. Much prior work has explored this opportunity, generally by applying a two-step process of first simplifying weight matrices and then encoding them in a more compact form. Simplification changes the number or characteristic of weight values to reduce the information needed to represent them. For example, pruning by selectively zeroing weight values BID18 BID12 can, in some cases, eliminate over 99% of the values without penalty. Similarly, most models do not need many bits of information to represent each weight. Quantization collapses weights to a smaller set of unique values, for instance via reduction to fixed-point binary representations BID13 or clustering techniques BID10.Simplifying weight matrices can further enable the use of more compact encoding schemes, improving compression. For example, two recent works BID14; BID7 encode pruned and quantized DNNs with sparse matrix representations. In both works, however, the encoding step is a lossless transformation, applied on top of lossy simplification. Weightless is a lossy encoding scheme based around Bloomier filters. We begin by describing what a Bloomier filter is, how to construct one, and how to retrieve values from it. We then show how we encode neural network weights using this data structure and propose a set of augmentations to make it an effective compression strategy for deep neural networks. A Bloomier filter generalizes the idea of a Bloom filter BID1, which are data structures that answer queries about set membership. Given a subset S of a universe U, a Bloom filter answers queries of the form, "Is v ∈ S?". If v is in S, the answer is always yes; if v is not in S, there is some probability of a false positive, which depends on the size of the filter, as size is proportional to encoding strength. By allowing false positives, Bloom filters can dramatically reduce the space needed to represent the set. A Bloomier filter BID5 ) is a similar data structure but instead encodes a function. For each v in a domain S, the function has an associated value f (v) in the range R = [0, 2 r). Given an input v, a Bloomier filter always returns f (v) when v is in S. When v is not in S, the Bloomier filter returns a null value ⊥, except that some fraction of the time there is a "false positive", and the Bloomier filter returns an incorrect, non-null value in the range R. W is an inexact reconstruction of W from a compressed projection X. To retrieve the value w i,j, we hash its location and exclusive-or the corresponding entries of X together with a computed mask M. If the ing value falls within the range [0, 2 r−1), it is used for w i,j, otherwise, it is treated as a zero. The red path on the left shows a false positive due to collisions in X and random M value. Decoding Let S be the subset of values in U to store, with |S| = n. A Bloomier filter uses a small number of hash functions (typically four), and a hash table X of m = cn cells for some constant c (1.25 in this paper), each holding t > r bits. For hash functions DISPLAYFORM0 Hence, to find the value of f (v), hash v four times, perform three table lookups, and exclusive-or together the four values. Like the Bloom filter, querying a Bloomier filter runs in O time. For u / ∈ S, the , DISPLAYFORM1 will be uniform over all t-bit values. If this is not in [0, 2 r), then ⊥ is returned and if it happens to land in [0, 2 r), a false positive occurs and a is (incorrectly) returned. An incorrect value is therefore returned with probability 2 r−t.Encoding Constructing a Bloomier filter involves finding values for X such that the relationship above holds for all values in S. There is no known efficient way to do so directly. All published approaches involve searching for configurations with randomized algorithms. In their paper introducing Bloomier filters, BID5 give a greedy algorithm which takes O(n log n) time and produces a table of size cn t bits with high probability. BID4 provide two slightly better constructions. First, they give a method with identical space requirements but runs in O(n) time. They also show a separate O(n log n)-time algorithm for producing a smaller table with c closer to 1. Using a more sophisticated algorithm for construction should allow for a more compact table and, by extension, improve the overall compression rate. However, we leave this for future work and use the method of BID5 for simplicity. While construction can be expensive, it is a one-time cost. Moreover, the absolute runtime is small compared to the time it takes to train a deep neural network. In the case of VGG-16, our unoptimized Python code built a Bloomier filter within an hour. We see this as a small worthwhile overhead given the savings offered and in contrast to the days it can take to fully train a network. We propose using the Bloomier filter to compactly store weights in a neural network. The function f encodes the mapping between indices of nonzero weights to their corresponding values. Given a weight matrix W, define the domain S to be the set of indices {i, j | w i,j = 0}. Likewise, the range R is [−2 a−1, 2 a−1) − {0} for a such that all values of W fall within the interval. Due to weight value clustering (see below) this range is remapped to [0, 2 r−1) and encodes the cluster indices. A null response from the filter means the weight has a value of zero. Once f is encoded in a filter, an approximation W of the original weight matrix is reconstructed by querying it with all indices. The original nonzero elements of W are preserved in the approximation, as are most of the zero elements. A small subset of zero-valued weights in W will take on nonzero values as a of random collisions in X, possibly changing the model's output. FIG0 illustrates the operation of this scheme: An original nonzero is correctly recalled from the filter on the right and a false positive is created by an erroneous match on the left (red).Complementing Bloomier filters with simplification Because the space used by a Bloomier filter is O(nt), they are especially useful under two conditions: The stored function is sparse (small n, with respect to |U |) and It has a restricted range of output values (small r, since t > r). To improve overall compression, we pair approximate encoding with weight transformations. Pruning networks to enforce sparsity (condition 1) has been studied extensively BID15 BID18. In this paper, we consider two different pruning techniques: (i) magnitude threshold plus retraining and (ii) dynamic network surgery (DNS) BID12. Magnitude pruning with retraining as straightforward to use and offers good . DNS is a recently proposed technique that prunes the network during training. We were able to acquire two sets of models, LeNet-300-100 and LeNet5, that were pruned using DNS and include them in our evaluation; as no reference was published for VGG-16 only magnitude pruning is used. Regardless of how it is accomplished, improving sparsity will reduce the overall encoding size linearly with the number of non-zeros with no effect on the false positive rate (which depends only on r and t). The reason for using two methods is to demonstrate the benefits of Weightless as networks increase in sparsity, the DNS networks are notably more sparse than the same networks using magnitude pruning. Reducing r (condition 2) amounts to restricting the range of the stored function or minimizing the number of bits required to represent weight values. Though many solutions to discretize weights exist (e.g., limited binary precision and advanced quantization techniques BID7), we use k-means clustering. After clustering the weight values, the k centroids are saved into an auxiliary table and the elements of W are replaced with indices into this table. This style of indirect encoding is especially well-suited to Bloomier filters, as these indices represent a small, contiguous set of integers. Another benefit of using Bloomier filters is that k does not have to be a power of 2. When decoding Bloomier filters, the of the XORs can be checked with an inequality, rather than a bitmask. This allows Bloomier filters to use k exactly, reducing the false positive rate by a factor of 1 − k 2 r. In other methods, like that of BID14, there is no benefit, as any k not equal to a power of two strictly wastes space. Tuning the t hyperparameter The use of Bloomier filters introduces an additional hyperparameter t, the number of bits per cell in the Bloomier filter. t trades off the Bloomier filter's size and the false positive rate which, in turn, effects model accuracy. While t needs to be tuned, we find it far easier to Figure 3: Weightless operates layer-by-layer, alternating between simplification and lossy encoding. Once a Bloomier filter is constructed for a weight matrix, that layer is frozen and the subsequent layers are briefly retrained (only a few epochs are needed).reason about than other DNN hyperparameters. Because we encode k clusters, t must be greater than log 2 k, and each additional t bit reduces the number of false positives by a factor of 2. This limits the number of reasonable values for t: when t is too low the networks experience substantial accuracy loss, but also do not benefit from high values of t because they have enough implicit resilience to handle low error rates (see FIG1). Experimentally we find that t typically falls in the range of 6 to 9 for our models. Retraining to mitigate the effects of false positives We encode each layer's weights sequentially. Because the weights are fixed, the Bloomier filter's false positives are deterministic. This allows for the retraining of deeper network layers to compensate for errors. It is important to note that encoded layers are not retrained. The randomness of the Bloomier filter would sacrifice all benefits of mitigating the effects of errors. If the encoded layer was retrained, a new encoding would have to be constructed (because S changes) and the indices of weights that in false positives would differ after every iteration of retraining. Instead, we find retraining all subsequent layers to be effective, typically allowing us to reduce t by one or two bits. The process of retraining around faults is relatively cheap, requiring on the order of tens of epochs to converge. The entire optimization pipeline is shown in Figure 3.Compressing Bloomier filters When sending weight matrices over a network or saving them to disk, it is not necessary to retain the ability to access weight values as they are being sent, so it is advantageous to add another layer of compression for transmission. We use arithmetic coding, an entropy-optimal stream code which exploits the distribution of values in the table BID20. Because the nonzero entries in a Bloomier filter are, by design, uniformly distributed values in [1, 2 t − 1), improvements from this stage largely come from the prevalence of zero entries. We evaluate Weightless on three deep neural networks commonly used to study compression: LeNet-300-100, LeNet5 , and VGG-16 BID24. The LeNet networks use MNIST (Lecun & Cortes) and VGG-16 uses ImageNet BID23. The networks are trained and tested in Keras BID8. The Bloomier filter was implemented in house and uses a Mersenne Twister pseudorandom number generator for uniform hash functions. To reduce the cost of constructing the filters for VGG-16, we shard the non-zero weights into ten separate filters, which are built in parallel to reduce construction time. Sharding does not significantly affect compression or false positive rate as long as the number of shards is small BID2.We focus on applying Weightless to the largest layers in each model, as shown in TAB1. This corresponds to the first two fully-connected layers of LeNet-300-100 and VGG-16. For LeNet5, the second convolutional layer and the first fully-connected layer are the largest. These layers account for 99.6%, 99.7%, and 86% of the weights in LeNet5, LeNet-300-100, and VGG-16, respectively. (The DNS version is slightly different than magnitude pruning, however the trend is the same.) While other layers could be compressed, they offer diminishing returns. The below present both the absolute compression ratio and the improvements Weightless achieves relative to Deep Compression BID14, which represents the current state-of-the-art. The absolute compression ratio is with respect the original standard 32-bit weight size of the models. This baseline is commonly used to report in publications and, while larger than many models used in practice, it provides the complete picture for readers to draw their own . For comparison to a more aggressive baseline, we reimplemented Deep Compression in Keras. Deep Compression implements a lossless optimization pipeline where pruned and clustered weights are encoded using compressed sparse row encoding (CSR) and then compresses CSR encoding tables with Huffman coding. The compression achieved by Deep Compression we use as a baseline is notably better than the original publication (e.g., VGG-16 FC-0 went from 91× to 119×).Error baseline Because Weightless performs lossy compression, it is important to bound the impact of the loss. We establish this bound as the error of the trained network after the simplification steps (i.e., post pruning and clustering). In doing so, the test errors from compressing with Weightless and Deep Compression are the same (shown as Baseline Error % in TAB1). Weightless is sometimes slightly better due to training fluctuations, but never worse. While Weightless does provide a tradeoff between compression and model accuracy, we do not consider it here. Instead, we feel the strongest case for this method is to compare against a lossless technique with iso-accuracy and note compression ratio will only improve in any use case where degradation in model accuracy is tolerable. Given a simplified baseline model, we first evaluate the how well Bloomier filters encode sparse weights. Results for Bloomier encoding are presented in TAB2, and show that the filters work exceptionally well. In the extreme case, the large fully connected layer in LeNet5 is compressed by 445×. With encoding alone and demonstrates a 1.99× improvement over CSR, the alternative encoding strategy used in Deep Compression. Recall that the size of a Bloomier filter is proportional to mt, and so sparsity and clustering determine how compact they can be. Our suggest that sparsity is more important than the number of clusters for reducing the encoding filter size. This can be seen by comparing each LeNet models' magnitude pruning to those of dynamic network surgery-while DNS needs additional clusters, the increased sparsity in a substantial size reduction. We suspect this is due to the ability of DNNs to tolerate a high false positive rate. The t value used here is already on the exponential part of the false positive curve (see FIG1 . At this point, even if k could be reduced, it is unlikely t can be since the additional encoding strength saved by reducing k does little to protect against the doubling of false positives when in this range. For VGG-16 FC-0, there are more false positives in the reconstructed weight matrix than there are non-zero weights originally; using t = 6 in over 6.2 million false positives while after simplification there are only 3.07 million weights. Before recovered with retraining, Bloomier filter encoding increased the top-1 error by 2.0 percentage points. This is why we see Bloomier filters work so well here-most applications cannot function with this level of approximation, nor do they have an analogous retrain mechanism to mitigate the errors' effects. BID14 propose using Huffman coding for their, and we use arithmetic coding, as described in Section 3.2. The in TAB3 show that while Deep Compression gets relatively more benefit from a final compression stage, Weightless remains a substantially better scheme overall. Prior work by BID21 on regular Bloom filters has shown that they can be optimized Figure 4: Weightless exploits sparsity more effectively than Deep Compression. By setting pruning thresholds to produce specific nonzero ratios, we can quantify sparsity scaling. There is no loss of accuracy at any point in this plot.for better post-compression size. We believe a similar method could be used on Bloomier filters, but we leave this for future work. Recent work continues to demonstrate better ways to extract sparsity from DNNs BID12 BID26 BID22, so it is useful to quantify how different encoding techniques scale with increasing sparsity. As a proxy for improved pruning techniques, we set the threshold for magnitude pruning to produce varying ratios of nonzero values for LeNet5 FC-0. We then perform retraining and clustering as usual and compare encoding with Weightless and Deep Compression (all without loss of accuracy). Figure 4 shows that as sparsity increases, Weightless delivers far better compression ratios. Because the false positive rate of Bloomier filters is controlled independent of the number of nonzero entries and addresses are hashed not stored, Weightless tends to scale very well with sparsity. On the other hand, as the total number of entries in CSR decreases, the magnitude of every index grows slightly, offsetting some of the benefits. This paper demonstrates a novel lossy encoding scheme, called Weightless, for compressing sparse weights in deep neural networks. The lossy property of Weightless stems from its use of the Bloomier filter, a probabilistic data structure for approximately encoding functions. By first simplifying a model with weight pruning and clustering, we transform its weights to best align with the properties of the Bloomier filter to maximize compression. Combined, Weightless achieves compression of up to 496×, improving the previous state-of-the-art by 1.51×.We also see avenues for continuing this line of research. First, as better mechanisms for pruning model weights are discovered, end-to-end compression with Weightless will improve commensurately. Second, the theory community has already developed more advanced-albeit more complicatedconstruction algorithms for Bloomier filters, which promise asymptotically better space utilization compared to the method used in this paper. Finally, by demonstrating the opportunity for using lossy encoding schemes for model compression, we hope we have opened the door for more research on encoding algorithms and novel uses of probabilistic data structures. Given a set of key value pairs, in this case weight addresses and cluster indexes, a Bloomier filter is constructed as follows. The task of construction is to find a unique location in the table for each key such that the value can be encoded there. For each to be encoded key a neighborhood of 3 hash digests are generated, indexes of these 3 are named iotas. To begin, the neighborhoods of all keys are computed and the unique digests are identified. Keys with unique neighbors are removed from the list of keys to be encoded. When a key is associated with a unique location, its iota value (i.e., the unique neighbor index into the neighborhood of hashs), is saved in an ordered list along with the key itself. The process is repeated for the remaining keys and continues until either all keys identify a unique location or none can be found. In the case that this search fails a different hash function can be tried or a larger table is required (increasing m).Once the unique locations and ordering is known, the encoding can begin. Values are encoded into the filter in the reverse order in which the unique locations are found during the search described above. This is done such that as non-unique neighbors of keys collide, they can still resolve the correct encoded values. For each key, the unique neighbor (indexed by the saved iota value) is set as the XOR of all neighbor filter entries (each a t-bit length vector), a random mask M, and the key's value. As the earlier found unique key locations are populated in the filter, it is likely that the neighbor values will be non-zero. By XORing together them in reverse order the encoding scheme guarantees that the correct value is retrieved. (The XORs can be thought of cancelling out all the erroneous information except that of the true value.)An implementation of the Bloomier filter will be released along with publication. Validation Error (%) Figure 5: Plot of retraining (as described in Section 3.2) VGG16's FC-1 after encoding FC-0 in a Bloomier filter. The baseline validation accuracy (on 25000 samples) is 30% and after encoding FC-0, with a Bloomier filter, this increases to 33%. As the plots hows, after a few epochs pass the error introduced by the lossy encoding is recovered. In comparison, test accuracy after encoding FC-0 and before retraining FC-1 is 38.0% and goes down by 2% to 36.0% which maintains the model's baseline performance. Figure 6: Sensitivity analysis of t with respect to the number of weight clusters, k, for FC-0 in LeNet-300-100. As per FIG1, model error decreases with the number of false positives, which are determined by the difference between t and k. This is emphasized here as k = 8 performs strictly better than higher k values as it in the strongest encoding given t. Figure 7: Demonstrating iso-accuracy comparisons between previous state-of-the-art DNN compression techniques and Weightless for FC-0 in LeNet5. In comparison to TAB3, we see that with respect to accuracy, Weightless' compression ratio scales significantly better than Deep Compression. This is advantageous in applications that are amenable to reasonable performance loss. To generate this plot we swept the pruning threshold for both Weightless and Deep Compression, and t from 6 to 9 for Weightless.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a new way to compress neural networks using probabilistic data structures.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.