Question: Represent the scientific article to find a short TLDR

Question: Neural networks can converge faster with help from a smarter batch selection strategy. In this regard, we propose Ada-Boundary, a novel adaptive-batch selection algorithm that constructs an effective mini-batch according to the learning progress of the model. Our key idea is to present confusing samples what the true label is. Thus, the samples near the current decision boundary are considered as the most effective to expedite convergence. Taking advantage of our design, Ada-Boundary maintains its dominance in various degrees of training difficulty. We demonstrate the advantage of Ada-Boundary by extensive experiments using two convolutional neural networks for three benchmark data sets. The experiment show that Ada-Boundary improves the training time by up to 31.7% compared with the state-of-the-art strategy and by up to 33.5% compared with the baseline strategy. Deep neural networks (DNNs) have achieved remarkable performance in many fields, especially, in computer vision and natural language processing BID13 BID5. Nevertheless, as the size of data grows very rapidly, the training step via stochastic gradient descent (SGD) based on mini-batches suffers from extremely high computational cost, which is mainly due to slow convergence. The common approaches for expediting convergence include some SGD variants BID28 BID11 that maintain individual learning rates for parameters and batch normalization BID9 that stabilizes gradient variance. Recently, in favor of the fact that not all samples have an equal impact on training, many studies have attempted to design sampling schemes based on the sample importance BID25 BID3 at the training accuracy of 60%. An easy data set (MNIST) does not have "too hard" sample but "moderately hard" samples colored in gray, whereas a relatively hard data set (CIFAR-10) has many "too hard" samples colored in black. (b) shows the of SGD on a hard batch. The moderately hard samples are informative to update a model, but the too hard samples make the model overfit to themselves. et al., 2017; BID10. Curriculum learning BID0 ) inspired by human's learning is one of the representative methods to speed up the training step by gradually increasing the difficulty level of training samples. In contrast, deep learning studies focus on giving higher weights to harder samples during the entire training process. When the model requires a lot of epochs for convergence, it is known to converge faster with the batches of hard samples rather than randomly selected batches BID21 BID17 BID4. There are various criteria for judging the hardness of a sample, e.g., the rank of the loss computed from previous epochs BID17.Here, a natural question arises: Does the "hard" batch selection always speed up DNN training? Our answer is partially yes: it is helpful only when training an easy data set. According to our indepth analysis, as demonstrated in FIG1 (a), the hardest samples in a hard data set (e.g., CIFAR-10) were too hard to learn. They are highly likely to make the decision boundary bias towards themselves, as shown in FIG1 (b). On the other hand, in an easy data set (e.g., MNIST), the hardest samples, though they are just moderately hard, provide useful information for training. In practice, it was reported that hard batch selection succeeded to speed up only when training the easy MNIST data set BID17 BID4, and our experiments in Section 4.4 also confirmed the previous findings. This limitation calls for a new sampling scheme that supports both easy and hard data sets. In this paper, we propose a novel adaptive batch selection strategy, called Ada-Boundary, that accelerates training and is better generalized to hard data sets. As opposed to existing hard batch selection, Ada-Boundary picks up the samples with the most appropriate difficulty, considering the learning progress of the model. The samples near the current decision boundary are selected with high probability, as shown in FIG3 (a). Intuitively speaking, the samples far from the decision boundary are not that helpful since they are either too hard or too easy: those on the incorrect (or correct) side are too hard (or easy). This is the reason why we regard the samples around the decision boundary, which are moderately hard, as having the appropriate difficulty at the moment. Overall, the key idea of Ada-Boundary is to use the distance of a sample to the decision boundary for the hardness of the sample. The beauty of this design is not to require human intervention. The current decision boundary should be directly influenced by the learning progress of the model. The decision boundary of a DNN moves towards eliminating the incorrect samples as the training step progresses, so the difficulty of the samples near the decision boundary gradually increases as the model is learned. Then, the decision boundary keeps updated to identify the confusing samples in the middle of SGD, as illustrated in FIG3 (b). This approach is able to accelerate the convergence speed by providing the samples suited to the model at every SGD iteration, while it is less prone to incur an overfitting issue. We have conducted extensive experiments to demonstrate the superiority of Ada-Boundary. Two popular convolutional neural network (CNN) 1 models are trained using three benchmark data sets. Compared to random batch selection, Ada-Boundary significantly reduces the execution time by 14.0-33.5%. At the same time, it provides a relative improvement of test error by 7.34-14.8% in the final epoch. Moreover, compared to the state-of-the-art hard batch selection BID17, Ada-Boundary achieves the execution time smaller by 18.0% and the test error smaller by 13.7% in the CIFAR-10 data set.2 Ada-Boundary COMPONENTS The main challenge for Ada-Boundary is to evaluate how close a sample is to the decision boundary. In this section, we introduce a novel distance measure and present a method of computing the sampling probability based on the measure. To evaluate the sample's distance to the decision boundary, we note that the softmax distribution, which is the output of the softmax layer in neural networks, clearly distinguishes how confidently the learner predicts and whether the prediction is right or wrong, as demonstrated in FIG5. If the prediction probability of the true label is the highest, the prediction is correct; otherwise, incorrect. If the highest probability dominates the distribution, the model's confidence is strong; otherwise, weak. Let h(y|x i ; θ t) be the softmax distribution of a given sample x i over y ∈ {1, 2, . . ., k} labels, where θ t is the parameter of a neural network at time t. Then, the distance from a sample x i with the true label y i to the decision boundary of the neural network with θ t is defined by the directional distance function in Eq.. More specifically, the function consists of two terms related to the direction and magnitude of the distance, determined by the model's correctness and confidence, respectively. The correctness is determined by verifying whether the label with the highest probability matches the true label y i, and the confidence is computed by the standard deviation of the softmax distribution. Intuitively, the standard deviation is a nice indicator of the confidence because the value gets closer to zero when the learner confuses. DISPLAYFORM0 One might argue that the cross-entropy loss, H(p, q) = −p(x i) log(q(x i)) where p(x i) and q(x i) are the true and softmax distributions for x i, can be adopted for the distance function. However, because p(x i) is formulated as a one-hot true label vector, the cross-entropy loss cannot capture the prediction probability for false labels, which is an important factor of confusing samples. Another advantage is that our distance function is bounded as opposed to the loss. For k labels, the maximum value of std (h(y|x i DISPLAYFORM1 The rank-based approach introduced by BID17 is a common way to make the sampling probability of being selected for the next mini-batch. This approach sorts the samples by a certain importance measure in descending order, and exponentially decays the sampling probability of a given sample according to its rank. Let N denote the total number of samples. Then, each r-th ranked sample is selected with the probability p(r) which drops by a factor of exp (log(s e)/N ).Here, s e is the selection pressure parameter that affects the probability gap between the most and the least important samples. When normalized to sum up to 1.0, the probability of the r-th ranked sample's being selected is defined by Eq.. DISPLAYFORM2 In the existing rank-based approach, the rank of a sample is determined by |dist(x i, y i ; θ t)| in ascending order, because it is inversely proportional to the sample importance. However, if the mass of the true sample distribution is skewed to one side (e.g., easy side) as shown in FIG6, the mini-batch samples are selected with high probability from the skewed side rather than around the decision boundary where |dist(x i, y i ; θ t)| is very small. This problem was attributed to unconditionally fixed probability to a given rank. In other words, the samples with similar ranks are selected with similar probabilities regardless of the magnitude of the distance values. To incorporate the impact of the distance into batch selection, we adopt the quantization method BID6 BID2 ) and use the quantization index q instead of the rank r. Let Δ be the quantization step size and d be the output of the function dist(x i, y i ; θ t) of a given sample x i. Then, the index q is obtained by the quantizer Q(d) as in Eq.. The quantization index gets larger as a sample moves away from the decision boundary. In addition, the difference between two indexes reflects the difference in the actual distances. DISPLAYFORM3 In Eq., we set Δ to be k −1 √ k − 1/N such that the index q is bounded to N (the total number of samples) by Eq.. The sampling probability of a given sample x i with the true label y i is defined as Eq.. As shown in FIG6, our quantization-based method provides a well-balanced distribution, even if the true sample distribution is skewed. DISPLAYFORM4 3 Ada-Boundary ALGORITHM 3.1 MAIN PROPOSED ALGORITHM Algorithm 1 describes the overall procedure of Ada-Boundary. The input to the algorithm consists of the samples of size N (i.e., training data set), the mini-batch size b, the selection pressure s e, and the threshold γ used to decide the warm-up period. In the early stages of training, since the quantization index for each sample is not confirmed yet, the algorithm requires the warm-up period during γ epochs. Randomly selected mini-batch samples are used to warm-up (Lines 6-7), and their quantization indexes are updated (Lines 11-16). After the warm-up epochs, the algorithm computes the sampling probability of each sample by Eq. and selects mini-batch samples based on the probability (Lines 8-10). Then, the quantization indexes are updated in the same way (Lines 11-16). Here, we compute the indexes using the model with θ t+1 after every SGD step rather than every epoch, in order to reflect the latest state of the model; besides, we asynchronously update the indexes of the samples only included in the mini-batch, to avoid the forward propagation of the entire samples which induces a high computational cost. For a more sophisticated analysis of sampling strategies, we modify a few lines of Algorithm 1 to present three heuristic sampling strategies, which are detailed in Appendix A. (i) Ada-Easy is designed to show the effect of easy samples on training, so it focuses on the samples far from the decision boundary to the positive direction. DISPLAYFORM5 (ii) Ada-Hard is similar to the existing hard batch strategy BID17, but it uses our distance function instead of the loss. That is, Ada-Hard focuses on the samples far from the decision boundary to the negative direction, which is the opposite of Ada-Easy. (iii) Ada-Uniform is designed to select the samples for a wide range of difficulty, so it samples uniformly over the distance range regardless of the sample distribution. FIG7 shows the distributions of mini-batch samples drawn by these three variants. The distribution of Ada-Easy is skewed to the easy side, that of Ada-Hard is skewed to the hard side, and that of Ada-Uniform tends to be uniform. To avoid additional inference steps of Ada-Boundary (Line 14 in Algorithm 1), we present a historybased variant, called Ada-Boundary(History). It updates the qunatization indexes using the previous model with θ t. See Appendix B for the detailed algorithm and experiment . In this section, all the experiments were performed on three benchmark data sets: MNIST 2 of handwritten digits with 60,000 training and 10,000 testing images; Fashion-MNIST 3 of various clothing BID26 with 60,000 training and 10,000 testing images; and CIFAR-10 4 of a subset of 80 million categorical images BID12 with 50,000 training and 10,000 testing images. We did not apply any data augmentation and pre-processing procedures. A simple model LeNet-5 was used for two easy data sets, MNIST and Fasion-MNIST. A complex model WideResNet-16-8 was used for a relatively difficult data set, CIFAR-10. Batch normalization BID9 was applied to both models. As for hyper-parameters, we used a learning rate of 0.01 and a batch size of 128; the training epoch was set to be 50 for LeNet-5 and 70 for WideResNet-16-8, which is early stopping to clearly show the difference in convergence speed. Regarding those specific to our algorithm, we set the selection pressure s e to be 100, which is the best value found from s e = {10, 100, 1000} on the three data sets, and set the warm-up threshold γ to be 10. Technically, a small γ was enough to warm-up, but to reduce the performance variance caused by randomly initialized parameters, we used the larger γ and shared model parameters for all strategies during the warm-up period. Due to the lack of space, the experimental using DenseNet (L = 25, k = 12) BID8 on two hard data sets, CIFAR-100 4 and Tiny-ImageNet 5, are discussed in Appendix C together with the impact of the selection pressure s e. We compared Ada-Boundary with not only random batch selection but also four different adaptive batch selections. Random batch selection selects the next batch uniformly at random from the entire data set. One of four adaptive selections is the state-of-the-art strategy that selects hard samples based on the loss-rank, which is called online batch selection BID17, and the remainders, Ada-Easy, Ada-Hard, and Ada-Uniform, are the three variants introduced in Section 3.2. All the algorithms were implemented using TensorFlow 6 and executed using a single NVIDIA Tesla V100 GPU on DGX-1. For reproducibility, we provide the source code at https://github. com/anonymized. To measure the performance gain over the baseline (random batch selection) as well as the state-ofart (online batch selection), we used the following three metrics. We repeated every test five times for robustness and reported the average. The wall-clock training time is discussed in Appendix D. (ii) Gain epo: Reduction in number of epochs to obtain the same error (%). In FIG9 (a), the test error of 1.014 · 10 2 achieved at the 50th epoch by random batch selection can be achieved only at the 29th epoch by Ada-Boundary. Thus, Gain err was (50 − 29)/50 × 100 = 42.0%.(iii) Gain tim: Reduction in running time to obtain the same error (%). In FIG9 FIG9 shows the convergence curves of training loss and test error for five batch selection strategies on three data sets, when we used the SGD optimizer for training. In order to improve legibility, only the curves for the baseline and proposed strategies are dark colored; thus, the three metrics in the figure were calculated against the baseline strategy, random batch selection. Owing to the lack of space, we discuss the with the momentum optimizer in Appendix E. Ada-Easy was excluded in FIG9 because its convergence speed was much slower than other strategies. That is, easy samples did not contribute to expedite training. We conduct convergence analysis of the five batch selection strategies for the same number of epochs, as follows: • MNIST FIG9 (a)): All adaptive batch selections achieved faster convergence speed compared with random batch selection. Ada-Boundary, Ada-Hard, and online batch selection showed similar performance. Ada-Uniform was the fastest at the beginning, but its training loss and test error increased sharply in the middle of the training or testing procedures.• Fashion-MNIST FIG9 ): Ada-Boundary showed the fastest convergence speed in both training loss and test error. In contrast, after warm-up epochs, the training loss of the other adaptive batch selections increased temporarily, and their test error at the final epoch became similar to that of random batch selection.• CIFAR-10 FIG9 ): Ada-Boundary and Ada-Hard showed the fastest convergence on training loss, but in test error, the convergence speed of Ada-Hard was much slower than that of Ada-Boundary. This means that focusing on hard samples in the overfitting to "too hard" samples, which is indicated by a larger difference between the converged training loss (error) and the converged test error. Also, the slow convergence speed of online batch selection in test error is explained by the same reason. In summary, in the easiest MNIST data set, all adaptive batch selections accelerated their convergence speed compared with random batch selection. However, as the training difficulty (complexity) increased from MNIST to Fashion-MNIST and further to CIFAR-10, only Ada-Boundary converged significantly (by Gain err) faster than random batch selection. We clarify the quantitative performance gains of Ada-Boundary over random batch and online batch selections in TAB3. Ada-Boundary significantly outperforms both strategies, as already shown in FIG9. There is only one exception in MNIST, because online batch selection is known to work well with an easy data set BID17. The noticeable advantage of AdaBoundary is to reduce the training time significantly by up to around 30%, which is really important for huge, complex data sets. There have been numerous attempts to understand which samples contribute the most during training. Curriculum learning BID0, inspired by the perceived way that humans and animals learn, first takes easy samples and then gradually increases the difficulty of samples in a manual manner. Self-paced learning BID14 uses the prediction error to determine the easiness of samples in order to alleviate the limitation of curriculum learning. They regard that the importance is determined by how easy the samples are. However, easiness is not sufficient to decide when a sample should be introduced to a learner BID4.Recently, BID24 used Bayesian optimization to optimize a curriculum for training dense, distributed word representations. BID20 emphasized that the right curriculum not only has to arrange data samples in the order of difficulty, but also introduces a small number of samples that are dissimilar to the previously seen samples. BID23 proposed a hard-example mining algorithm to eliminate several heuristics and hyper-parameters commonly used to select hard examples. However, these algorithms are designed to support only a designated task, such as natural language processing or region-based object detection. The neural data filter proposed by BID3 is orthogonal to our work because it aims at filtering the redundant samples from streaming data. As mentioned earlier, Ada-Boundary in general follows the philosophy of curriculum learning. More closely related to the adaptive batch selection, BID17 keep the history of losses for previously seen samples, and compute the sampling probability based on the loss rank. The sample probability to be selected for the next mini-batch is exponentially decayed with its rank. This allows the samples with low ranks (i.e., high losses) are considered more frequently for the next mini-batch.'s work is similar to BID17's work except that gradient norms are used instead of losses to compute the probability. In contrast to curriculum learning, both methods focus on only hard samples for training. Also, they ignore the difference in actual losses or gradient norms by transforming the values to ranks. We have empirically verified that Ada-Boundary outperforms online batch selection BID17, which is regarded as the state-of-the-art of this category. Similar to our work, BID1 claimed that the uncertain samples should be preferred during training, but their main contribution lies on training more accurate and robust model by choosing samples with high prediction variances. In contrast, our main contribution lies on training faster using confusing samples near the decision boundary. For the completeness of the survey, we mention the work to accelerate the optimization process of conventional algorithms based on importance sampling. BID19 re-weight the obtained gradients by the inverses of their sampling probabilities to reduce the variance. BID22 biased the sampling to the Lipschitz constant to quickly find the solution of a strongly-convex optimization problem arising from the training of conditional random fields. In this paper, we proposed a novel adaptive batch selection algorithm, Ada-Boundary, that presents the most appropriate samples according to the learning progress of the model. Toward this goal, we defined the distance from a sample to the decision boundary and introduced a quantization method for selecting the samples near the boundary with high probability. We performed extensive experiments using two CNN models for three benchmark data sets. The showed that Ada-Boundary significantly accelerated the training process as well as was better generalized in hard data sets. When training an easy data set, Ada-Boundary showed a fast convergence comparable to that of the state-of-the-art algorithm; when training relatively hard data sets, only Ada-Boundary converged significantly faster than random batch selection. The most exciting benefit of Ada-Boundary is to save the time needed for the training of a DNN. It becomes more important as the size and complexity of data becomes higher, and can be boosted with recent advance of hardware technologies. Our immediate future work is to apply Ada-Boundary to other types of DNNs such as the recurrent neural networks (RNN) BID18 and the long short-term memory (LSTM) BID7, which have a neural structure completely different from the CNN. In addition, we plan to investigate the relationship between the power of a DNN and the improvement of Ada-Boundary. For Ada-Easy which prefers easy samples to hard samples, q should be small for the sample located deep in the positive direction. For Ada-Hard, q should be small for the sample located deep in the negative direction. Thus, Ada-Easy and Ada-Hard can be implemented by modifying the quantizers Q(d) in Line 16 of Algorithm 1. When we set Δ = k −1 √ k − 1/N to make the index q bound to N, the quantizers of Ada-Easy and Ada-Hard are defined as Eqs. and, respectively. DISPLAYFORM0 Ada-Uniform can be implemented by using F −1 (x) to compute the sampling probability in Line 9 of Algorithm 1, where F (x) is the empirical sample distribution according to the sample's distance to the decision boundary. C Ada-Boundary ON TWO HARD DATA SETS As a practical paper, we include the experimental on two more challenging data sets: CIFAR-100 composed of 100 image classes with 50, 000 training and 10, 000 testing images; Tiny-ImageNet composed of 200 image classes with 100, 000 training and 10, 000 testing images. All images in Tiny-ImageNet were resized to 32 × 32 images. One of the state-of-the-art model DenseNet (L=25, k=12) BID8 was used for two hard data sets with momentum optimizer. Regarding algorithm parameters, we used a learning rate of 0.1 and a batch size of 128; The training epoch and warm-up threshold γ were set to be 90 and 10, respectively. We repeated every test five times for robustness and reported the average. The selection pressure s e determines how strongly the boundary samples are selected. The greater the s e, the greater the sampling probability of the boundary sample, so more boundary samples were chosen for the next mini-batch. On the other hand, the less s e makes Ada-Boundary closer to random batch selection. FIG14 shows the convergence curves of Ada-Boundary with varying s e on two hard data sets. To clearly analyze the impact of the selection pressure, we plotted the minimum of training loss and test error with a given epochs. Overall, the convergence speed of training loss was accelerated as the s e increased from 2 to 16, but that of test error was faster only when the s e was less than a certain value. The convergence speed of test error was faster than random batch selection, when s e was less than or equal to 4 (CIFAR-100) and 2 (Tiny-ImageNet). Surprisingly, the overexposure to the boundary samples using the large s e incurred the overfitting issue in hard data sets, whereas the large s e = 100 worked well for our easy or relatively hard data sets as discussed in Section 4. That is, the selection pressure s e should be chosen more carefully considering the difficulty of the given data set. We leave this challenge as our future work. TAB4 shows the performance gains of Ada-Boundary over random batch selection on two hard data sets. We only quantify the gains of Ada-Boundary(s e = 2) because its performance was the best as shown in FIG14. Ada-Boundary(s e = 2) always outperforms random batch selection. Especially, it reduces the training time significantly by up to around 20%. Table 3 shows the wall-clock training time for the same number of parameter updates on two hard data sets FIG14 ). Ada-Boundary(s e = 2) with momentum was 15.2%-16.0% slower than random batch selection. However, it reduced the running time by 18.0%-21.4% (by Gain tim) to obtain the same test error of random batch selection. D WALL-CLOCK TRAINING TIME The procedures for recomputing sampling probabilities and updating quantization indexes make Ada-Boundary slower than random batch selection. Table 4 shows the wall-clock training time for the same number of parameter updates (i.e., the same number of epochs) with SGD (FIG9) and momentum (FIG1). Ada-Boundary with SGD was 12.8%-14.7% and 6.06%-12.2% slower than random batch and online batch selections, respectively. Ada-Boundary with momentum was 13.1%-14.7% and 6.67%-12.2% slower than random batch and online batch selections, respectively. Although Ada-Boundary took longer for the same number of updates, Ada-Boundary achieved significant reduction in running time by 7.96%-33.5% (by Gain tim) to obtain the same test error of random batch selection due to the fast convergence. TAB3 E EXPERIMENT USING MOMENTUM OPTIMIZER E.1 CONVERGENCE ANALYSIS FIG1 shows the convergence curves of training loss and test error for five batch selection strategies on three data sets, when we used the momentum optimizer with setting the momentum to be 0.9. In the MNIST data set, we limited the number of epochs to be 30 because both training loss and test error were fully converged after 30 epochs. We repeat the convergence analysis, as follows:• MNIST FIG1 ): Except Ada-Uniform, all adaptive batch selections converged faster than random batch selection. Online batch selection showed much faster convergence speed than other adaptive batch selections in training loss, but converged similarly with the others in test error owing to the overfitting to hard samples.• Fashion-MNIST FIG1 ): Ada-Boundary showed the fastest convergence speed in test error, although it did not converge faster than online batch selection in training loss. In contrast, online batch selection was the fastest in training loss, but its convergence in test error was slightly slower than that of random batch selection. This emphasizes the need to consider the samples with appropriate difficulty rather than hard samples. The convergence speeds of Ada-Hard and Ada-Uniform in test error were slower than that of random batch selection.• CIFAR-10 FIG1 ): In both training loss and test error, Ada-Boundary and Ada-Hard showed slightly faster convergence speed than random batch selection. On the other hand, online batch selection converged slightly slower than random batch selection in both cases. In summary, in the easiest MNIST data set, most of adaptive batch selections accelerated their convergence speed compared with random batch selection. However, in Fashion-MNIST data set, only Ada-Boundary converged faster than random batch selection. In a relatively difficult CIFAR-10 data set, Ada-Boundary and Ada-Hard showed comparable convergence speed and then converged faster than random batch selection. We quantify the performance gains of Ada-Boundary over random batch and online batch selections in TAB7. Ada-Boundary always outperforms both strategies, as already shown in FIG1. Compared with TAB3, Gain tim over random batch selection tends to become smaller, whereas Gain tim over online batch selection tends to become larger.

Represent the short summary for finding the corresponding scientific article (long)!

We suggest a smart batch selection technique called Ada-Boundary.

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: State of the art sound event classification relies in neural networks to learn the associations between class labels and audio recordings within a dataset. These datasets typically define an ontology to create a structure that relates these sound classes with more abstract super classes. Hence, the ontology serves as a source of domain knowledge representation of sounds. However, the ontology information is rarely considered, and specially under explored to model neural network architectures. We propose two ontology-based neural network architectures for sound event classification. We defined a framework to design simple network architectures that preserve an ontological structure. The networks are trained and evaluated using two of the most common sound event classification datasets. Results show an improvement in classification performance demonstrating the benefits of including the ontological information. Humans can identify a large number of sounds in their environments e.g., a baby crying, a wailing ambulance siren, microwave bell. These sounds can be related to more abstract categories that aid interpretation e.g., humans, emergency vehicles, home. These relations and structures can be represented by ontologies BID0, which are defined for most of the available datasets for sound event classification (SEC). However, sound event classification rarely exploits this additional available information. Moreover, although neural networks are the state of the art for SEC BID1 BID2 BID3, they are rarely designed considering such ontologies. An ontology is a formal representation of domain knowledge through categories and relationships that can provide structure to the training data and the neural network architecture. The most common type of ontologies are based on abstraction hierarchies defined by linguistics, where a super category represents its subcategories. Generally, the taxonomies are defined by either nouns or verbs e.g., animal contains dog and cat, dog contains dog barking and dog howling. Examples of datasets are ESC-50 BID4, UrbanSounds BID5, DCASE BID6, AudioSet BID7. Another taxonomy can be defined by interactions between objects and materials, actions and descriptors e.g., contains Scraping, which contains Scraping Rapidly and Scraping a Board BID8 BID9 BID10. Another example of this type is given by physical properties, such as frequency and time patterns BID11 BID12 BID13. There are multiple benefits of considering hierarchical relations in sound event classifiers. They can allow the classifier to back-off to more general categories when encountering ambiguity among subcategories. They can disambiguate classes that are acoustically similar, but not semantically. They can be used to penalize classification differently, where miss classifying sounds from different super classes is worse than within the same super class. Lastly, they can be used as domain knowledge to model neural networks. In fact, ontological information has been evaluated in computer vision BID14 and music BID15, but has rarely been used for sound event classification. Ontology-based network architectures have showed improvement in performance along with other benefits. Authors in BID16 proposed an ontology-based deep restricted Boltzmann machine for textual topic classification. The architecture replicates the tree-like structure adding intermediate layers to model the transformation from a super class to its sub classes. Authors showed improved performance and reduced overfitting in training data. Another example used a perceptron for each node of the hierarchy, which classified whether an image corresponded to such class or not BID17. Authors showed an improvement in performance due to the ability of class disambiguation by comparing predictions of classes and sub classes. Motivated by these approaches and by the flexibility to adapt structures in a deep learning model we propose our ontology-based networks detailed in the following section. In this section we present a framework to deal with ontological information using deep learning architectures. First, we describe a set of assumptions we consider along this paper. In particular, we describe the type of ontologies we work and some of their implications. Later, we present a Feed-forward model that includes the discussed constraints, defining our proposed ontological layer. Second, in order to preserve an embedding space consistent with the ontological structure, we extended the learning model to compute ontology-based embeddings using Siamese Neural Networks. The framework is defined to make use of the ontology structure and to model the neural network architectures. It should be noted that we considered ontologies with two levels, which are the most common in sound event datasets. Nevertheless, the presented framework can be easily generalized to more levels. In our framework, we considered the training data {(x 1, y 1),..., (x n, y n)}, where x i ∈ X is an audio representation, which is associated to a set of labels given by the ontology DISPLAYFORM0 In this case, C i is the set of possible classes at i-level. Assuming a hierarchical relation, we can consider that each possible class in C i is mapped to one element in C i+1. The higher the value of i, the higher the level in the ontology. For example, consider the illustration of an ontology in FIG0. In this case k = 2, C 1 = {cat, dog, breathing, eating, sneezing, violin, drums, piano, beep, boing, train, siren} and C 2 = {nature, human, music, effects, urban}. As the figure shows, every element in C 1 is related to one element in C 2; e.g., cat belongs to nature, or drums belongs to music. Furthermore, for a given representation x ∈ X, if we know the corresponding label y 1 in C 1, we can infer its label in C 2. This intuition can be formalized using a probabilistic formulation, where it is Net linear + softmax Figure 2: Architecture of the Feed-forward Network with Ontological Layer. The blue column represents the acoustic feature vector, the red columns are the output probabilities for both levels.straightforward to see that, assuming p(y 2 |y 1, x) = p(y 2 |y 1), the following is satisfied: DISPLAYFORM1 Therefore, if we want to estimate p(y 2 |x) using a model, we just need to compute the estimation of p(y 1 |x) and sum the values corresponding to the children of y 2. This case is valid for inference time, however, it is not clear that using the representation and label (x, y 1) should be enough to train the model. If at training time we can make use of knowledge to relate the different classes in y 1, it should improve the performance of the model, specially at making predictions for classes y 2.In the following sections we take our proposed framework and use it to design ontology-based neural network architectures. In this section, we describe how we use our proposed framework to design the architecture. Also, we introduce the ontological layer, which makes use of the ontology structure. The Feed-forward Network (FFN) with Ontological Layer consists of a base network (Net), an intermediate vector z, and two outputs, one for each ontology level. The base network weights are learned at every parameter update. The base network utilizes an input vector of audio features x and generates a vector z. This vector is used to generate two outputs, p(y 1 |x) a probability vector for C 1 and p(y 2 |x) a probability vector for C 2. First, the vector z is passed to a softmax layer of the size of C 1. Then, this output is multiplied by the ontological layer M and generates a layer of size of C 2. Once the FFN is trained, it can be used to predict any class in C 1 and C 2 for any input x. The ontological layer reflects the relation between super classes and sub classes given by the ontology. To describe how we used this layer, we refer to Equation 3, where p(y 2 |x) is the sum of all the values of p(y 1 |x) corresponding to the children of y 2. If we consider this equation as a directed graph where M is the |C 2 | × |C 1 | incidence matrix, then, it is clear that Equation 3 can be rewritten as, DISPLAYFORM0 Note that the ontological layer M defines the weights of a standard layer connection. Although we do not consider that these weights are trainable, they are part of our training data. In order to train this model, we simply propose to apply gradient-based method to minimize the loss function L, which is a convex combination between two categorical cross-entropy functions; L 1 the categorical cross entropy corresponding to p(y 1 |x) and L 2 corresponding to p(y 2 |x). Formally, DISPLAYFORM1 Hence, we consider λ ∈ as a hyper parameter to be tuned. Note that, when λ = 1, we are reducing the problem to train a standard classifier just using the information from the first level of the ontology. In this section, we describe how we learned the ontology-based embeddings. Our goal is to create embeddings that preserved the ontological structure. We used a Siamese neural network (SNN), which enforces samples of the same class to be closer, while separating samples of different classes. If two samples belong to different subclasses, but they belong to the same super class, they are closer than two samples that belong to different super classes. The architecture of the SNN with the Feed-forward Network with Ontological Layer is shown in FIG1. The blue rows represent the acoustic feature vectors of two different samples; they can be from the same subclass, different subclass but same super class, or different super class. Then, the twin networks have the same base architecture (Net) with shared weights. The weights are learned simultaneously at every parameter update. The white rows represent the ontological embeddings used to compute a Similarity metric (Euclidean Distance), where the distance of the embeddings z 1 and z 2 should indicate how different x 1 and x 2 are with respect to the ontology. For this work, we imposed that the distance between z 1 and z 2 is close to 0 if the samples are from the same subclass, close to 5 if they are from different sub classes, but the same super class, and close to 10 if they are from different super classes. Finally, the red rows are the output probabilities for both levels, p(y 1 |x 1), p(y 1 |x 2), p(y 2 |x 1) and p(y 2 |x 2).To train the Feed-forward Model with Ontological layer using Ontology-based embeddings, we provided the three types of pairs of audio examples and applied a gradient-based method to minimize DISPLAYFORM0 In this section, we evaluate the sound event classification performance of the ontological-based neural network architectures. We present the datasets and its ontologies, the baseline and proposed architectures, and the classification performance at different levels of the hierarchy. Making Sense of Sounds Challenge 2 -MSoS: The dataset is designed for a challenge which objective is to classify the most abstract classes or highest level in its taxonomy. The ontology, illustrated in FIG0 has two levels, the lowest level 1, has 97 classes and the highest level 2, has 5 classes. The audio files were taken from Freesound data base, the ESC-50 dataset and the Cambridge-MT Multitrack Download Library. The development dataset consists of 1500 audio files divided into the five categories, each containing 300 files. The number of different sound types within each category is not balanced. The evaluation dataset consists of 500 audio files, 100 files per category. All files have an identical format: single-channel 44.1 kHz, 16-bit.wav files. We randomly partitioned the set in 80% for training and tuning parameters and 10% for testing. All files are exactly 5 seconds long, but may feature periods of silence. The official blind evaluation set of the challenge consisted on 500 files distributed among the 5 classes. Urban Sounds -US8K: The dataset is designed to evaluate classification of urban sounds, which are organized using a taxonomy with more nodes than the annotated number of classes. Due to this reason, we adjusted the taxonomy to avoid redundant levels with only one annotated child. The ing ontology is illustrated in FIG2, with two levels, the lowest level 1, has 10 classes and the highest level 2, has 4 classes. The audio files were taken from Freesound data base and corresponded to real field recordings. All files have an identical format: single-channel 44.1 kHz, 16-bit.wav files. The dataset contains 8,732 audio files divided into 10 stratified subsets. We used 9 folds to train and tune parameters and one fold for testing. We used state-of-the-art Walnet features BID1 to represent audio recordings. For each audio, we computed a 128-dimensional logmel-spectrogram vector and transformed it via a convolutional neural network (CNN) that was trained separately on the balanced set of AudioSet. The architecture of the base network (Net) considered in this experiment, shown in Fig. 2, is a feed-forward multi-layer perceptron network. It consists of 4 layers: the input layer of dimensionality 1024, which takes audio feature vectors, 2 dense layers of dimensionality 512 and 256, respectively, and the output layer of dimensionality 128, which is the dimensionality of the vector z. The dense layers utilize Batch Normalization, a dropout rate of 0.5 and the ReLU activation function; max(0, x), where x is input to the function. We tuned the parameters in the Net box as well as the parameters that transformed z into p(y 1 |x). We considered baseline models in both level 1 and 2 for the different data sets. In this case, the baseline models did not consider any ontological information, hence the models consist of the Base Network Architecture with the addition of an output layer that was either for level 1 or level 2.Note that for level 1 this is equivalent to training the Feed-forward model with Ontological Layer using λ = 1. Indeed, with λ = 1 the loss function associated to level 2 is not considered. For level 2, the baseline model is different from the Feed-forward model with λ = 0, because in the baseline model there is no layer corresponding to the prediction of y 1. Table 1 shows the of baseline models for both, MSoS and US8K data set in level 1 and level 2.The baseline performance of the development set in the MSoS challenge was reported to be 0.81 for level 2 and no baseline was provided for level 1. To validate the architecture presented in Section 2.2 and analyze the utility of the ontological layer, we trained models taking different values of λ. FIG3 shows the effect of λ in both data sets. In general, we observe that considering values different from 0 and 1 helps to increase the performance. Note that the classification in both level is affected by the ontological layer. In the case of MSoS data set, the best performance was obtained using λ = 0.8, getting 0.74 and 0.913 of accuracy in level 1 and 2 respectively. Thus, using the ontological structure we can get an absolute improvement of 5.4% and 6% respect baseline models. Running the same experiment on the US8K data set, we observe a smaller improvement. The best performance was obtained using λ = 0.7, being the accuracy of 0.82 and 0.86 for level 1 and 2 respectively. This means an improvement of 2.5% and 0.2% only, respect baseline models. Best is achieved using λ = 0.8 (Right) Results in US8K data set. Best is achieved using λ = 0.7 Figure 6: The MSoS t-SNE plots of the samples in classes from level 2 (1st and 3rd) and level 1 (2nd and 4th). The first two boxes are from the base network vectors and the second two boxes are the ontology-based embeddings. We observe in 1st and 3rd, the groups of classes in level 2 and in 2nd and 4th the same level 2 groups, but using the level 1 class samples. The ontology-based embeddings in tighter and better defined clusters. We tested the architecture described in Section 2.3 to evaluate the performance of the ontology-based embeddings for sound event classification. Additionally, we include t-SNE plots, to illustrate how the embeddings cluster at different levels. We processed the Walnet audio features and chose different super and sub class pairs to train the Siamese neural network to produce the ontology-based embeddings. The embeddings are passed to the architecture of the base network (Net), which is the same as the one used in the previous section. We trained the SNN for 50 epochs using the Adam algorithm. We also tuned the hyper-parameters of the SNN to achieve good performance with the input features that are described in the next section. We also tried different number of pairs for the input training data, from 100 to 1,000,000 pairs and found that 100,000 yielded the best performance. For the loss function we used the values derived in the previous experiment. We used the value of 0.8 for the lambda of the classifiers of level 2 and 0.2 for the classifiers in level 1, and 0.2 for the similarity metric. Modifying the lambdas in the loss function affected the overall performance. The in Table 1 show that the accuracy performance of MSoS and US8K were respectively as follows, in level 1 0.736 and 0.818, and in level 2 0.886 and 0.856. Based on these we made the following . The performance of this architecture is better than the baseline, but slightly under performed the method without the embeddings. Nevertheless, the ontology-based embeddings have the benefit of better grouping as illustrated in Figure6. We took the MSoS data and created the t-SNE plots (perplexity=30) of the classes in level 2 and level 1. We observed that the FF + Ontology vectors and the ontology-based embeddings provided clustered groups of level 2 classes. However, the ontology-based embeddings have tighter and better defined clusters. In the case of the US8K data set performance was limited. We think this was because the number of sub classes was similar to the number of super classes. We had 10 sub classes for 4 classes unlike the MSoS data set, where we had 97 sub classes and 5 classes. It seems when the ratio between the number of sub classes and the number of classes is not large, the contribution of the ontology is negligible. Both approaches were used to compete in the Making Sense of Sounds Challenge. The baseline for the blind evaluation set was 0.80 accuracy for level 2. The Feed-forward Network with Ontological Layer achieved 0.88 while using the ontological-embeddings achieved 0.89. Again, both architectures outperformed significantly the baseline. In this paper we proposed a framework to design neural networks for sound event classification using hierarchical ontologies. We have shown two methods to add such structure into deep learning models in a simple manner without adding more learnable parameters. We used a Feed-forward Network with an ontological layer to relate predictions of different levels in the hierarchy. Additionally, we proposed a Siamese neural Network to compute ontology-based embeddings to preserve the ontology in an embedding space. The embeddings plots showed clusters of super classes containing different sub classes. Our in the datasets and MSoS challenge improved over the baselines. We expect that our pave the path to further explore ontologies and other relations, which is fundamental for sound event classification due to wide acoustic diversity and limited lexicalized terms to describe sounds.

Represent the TLDR for finding the corresponding scientific article

We present ontology-based neural network architectures for sound event classification.

represent this text to find a much much longer article about it with say hundreds-thousands of words
------
To give you a sense - "We proposed a universal method which can be used in the data preprocessing stage to generate the more meaningful topic that better represents the given document" should be close to "The topic modeling discovers the latent topic probability of given the text documents. To generate the more meaningful topic that better represents the given document, we proposed a universal method which can be used in the data preprocessing stage. The method consists of three steps. First, it generates the word/word-pair from every single document. Second, it applies a two way parallel TF-IDF algorithm to word/word-pair for semantic filtering. Third, it uses the k-means algorithm to merge the word pairs that have the similar semantic meaning. Experiments are carried out on the Open Movie Database (OMDb), Reuters Dataset and 20NewsGroup Dataset and use the mean Average Precision score as the evaluation metric. Comparing our with other state-of-the-art topic models, such as Latent Dirichlet allocation and traditional Restricted Boltzmann Machines. Our proposed data preprocessing can improve the generated topic accuracy by up to 12.99\%. How the number of clusters and the number of word pairs should be adjusted for different type of text document is also discussed. After millennium, most collective information are digitized to form an immense database distributed across the Internet. Among all, text-based knowledge is dominant because of its vast availability and numerous forms of existence. For example, news, articles, or even Twitter posts are various kinds of text documents. For the human, it is difficult to locate one's searching target in the sea of countless texts without a well-defined computational model to organize the information. On the other hand, in this big data era, the e-commerce industry takes huge advantages of machine learning techniques to discover customers' preference. For example, notifying a customer of the release of "Star Wars: The Last Jedi" if he/she has ever purchased the tickets for "Star Trek Beyond"; recommending a reader "A Brief History of Time" from Stephen Hawking in case there is a "Relativity: The Special and General Theory" from Albert Einstein in the shopping cart on Amazon. The content based recommendation is achieved by analyzing the theme of the items extracted from its text description. Topic modeling is a collection of algorithms that aim to discover and annotate large archives of documents with thematic information BID0. Usually, general topic modeling algorithms do not require any prior annotations or labeling of the document while the abstraction is the output of the algorithms. Topic modeling enables us to convert a collection of large documents into a set of topic vectors. Each entry in this concise representation is a probability of the latent topic distribution. By comparing the topic distributions, we can easily calculate the similarity between two different documents. BID25 Some topic modeling algorithms are highly frequently used in text-mining BID13, preference recommendation BID27 and computer vision BID28. BID0 Many of the traditional topic models focus on latent semantic analysis with unsupervised learning. Latent Semantic Indexing (LSI) BID11 applies Singular-Value Decomposition (SVD) BID6 to transform the term-document matrix to a lower dimension where semantically similar terms are merged. It can be used to report the semantic distance between two documents, however, it does not explicitly provide the topic information. The Probabilistic Latent Semantic Analysis (PLSA) BID9 model uses maximum likelihood estimation to extract latent topics and topic word distribution, while the Latent Dirichlet Allocation (LDA) BID1 model performs iterative sampling and characterization to search for the same information. The availability of many manually categorized online documents, such as Internet Movie Database (IMDb) movie review Inc., Wikipedia articles, makes the training and testing of topics models possible. All of the existing workds are based on the bag-of-words model, where a document is considered as a collection of words. The semantic information of words and interaction among objects are assumed to be unknown during the model construction. Such simple representation can be improved by recent research advances in natural language processing and word embedding. In this paper, we will explore the existing knowledge and build a topic model using explicit semantic analysis. The work studies the best data processing and feature extraction algorithms for topic modeling and information retrieval. We investigate how the available semantic knowledge, which can be obtained from language analysis or from existing dictionary such as WordNet, can assist in the topic modeling. Our main contributions are:• We redesign a new topic model which combines two types of text features to be the model input.• We apply the numerical statistic algorithm to determine the key elements for each document dynamically.• We apply a vector quantization method to merge and filter text unit based on the semantic meaning.• We significantly improve the accuracy of the prediction using our proposed model. The rest of the paper is structured as follows: In Section 2, we review the existing methods, from which we got the inspirations. This is followed in Section 3 by details about our topic models. Section 4 describes our experimental steps and evaluate the . Finally, Section 5 concludes this work. Many topic models have been proposed in the past decades. This includes LDA, Latent Semantic Analysis(LSA), word2vec, and Restricted Boltzmann Machine (RBM), etc. In this section, we will compare the pros and cons of these topic models. LDA was one of the most widely used topic models. LDA introduces sparse Dirichlet prior distributions over document-topic and topic-word distributions, encoding the intuition that documents cover a small number of topics and that topics often use a small number of words. BID1 LSA was another topic modeling technique which is frequently used in information retrieval. LSA learned latent topics by performing a matrix decomposition (SVD) on the term-document matrix. BID4 In practice, training the LSA model is faster than training the LDA model, but the LDA model is more accurate than the LSA model. Traditional topic models did not consider the semantic meaning of each word and cannot represent the relationship between different words. Word2vec can be used for learning high-quality word vectors from huge data sets with billions of words, and with millions of words in the vocabulary. BID14 During the training, the model generated word-context pairs by applying a sliding window to scan through a text corpus. Then the word2vec model trained word embeddings using word-context pairs by using the continuous bag of words (CBOW) model and the skip-gram model. BID15 The generated word vectors can be summed together to form a semantically meaningful combination of both words. Moreover, there were a lot of extensions of the word2vec model, such as paragraph2vec and LDA2vec. Paragraph2vec was an unsupervised framework that learned continuous distributed vector representations for pieces of texts. The training of the paragraph2vec model is based on the similar idea as the word2vec. One of the advantages of the generated paragraph vector was that they take into consideration the word order. BID12 LDA2vec focused on utilizing document-wide feature vectors while simultaneously learning continuous document weights loading onto topic vectors. LDA2vec embedded both words and document vectors into the same space and train both representations simultaneously. BID18 RBM was proposed to extract low-dimensional latent semantic representations from a large collection of documents BID8. The architecture of the RBMs is an undirected bipartite graphic, in which word-count vectors were modeled as Softmax input units and the output units were usually binary units. The RBM model can be generalized much better than LDA in terms of both log-probability on the document and the retrieval accuracy. A deeper structure of neural network was developed based on stacked RBMs, which was the Deep Belief Network (DBN). In BID7, the input layer was the same as RBM mentioned above, other layers are all binary units. 3.1 WORD PAIR BASED RBM MODEL Current RBM model for topic modeling uses Bag of Words approach. Each visible neuron represents the number of appearance of a dictionary word. We believe that the order of the words also exhibits rich information, which is captured by the bag of words approach. Our hypothesis is that including word pairs (with specific dependencies) helps to improve topic modeling. In this work, Stanford natural language parser BID3 BID19 is used to analyze sentences in both training and testing texts, and extract word pairs. For example, if we have the following sentence, by applying the word pair extraction, we will get 38 word pairs and one root pair as the FIG0.The strongest rain ever recorded in India shut down the financial hub of Mumbai, snapped communication lines, closed airports and forced thousands of people to sleep in their offices or walk home during the night, officials said today. The first part in each word pair segment represents the relationship between two words, such as det, nsubj, advmod, etc. And the number after each word gives its position of in the original sentence. The two words extracted in this way are not necessarily adjacent to each other, however, they are semantically related. Because each single word may have different combinations with other words, the total number of the word pairs will be much larger than the number of word in the training dataset. If we use all word pairs which are extracted from the training dataset, it will significantly increase the size of our dictionary and reduce the performance. So, we only keep the first 10000 most frequent word pairs to be our word pair dictionary. TF-IDF stands for the term frequency -inverse document frequency, and the TF-IDF weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. BID23 BID22 BID21 BID29 The equation of the TF-IDF is as following: DISPLAYFORM0 N umber of times term t appears in a docuemnt T otal number of terms in the document DISPLAYFORM1 DISPLAYFORM0 Term Frequency (TF), Equation 1, measures how frequently a term occurs in a document. Inverse document frequency (IDF), Equation 2, measures how important a term is. There are many words in the training dataset. If we use all words to build the training dictionary, it will contain a lot of high frequency but useless words, like "first" and "name". Beside removing stop word and processing word tense in the dataset, we also used the TF-IDF algorithm to filter the dataset. In addition, based on the original TF-IDF algorithm, we proposed a two-step TF-IDF processing method. This method is applied to keep the number of word pairs to a manageable size as shown in First, word pairs are generated and word-level TF-IDF is performed. The of word level TF-IDF is used as a filter and a word pair is kept only if the TF-IDF scores of both words are higher than the threshold (0.01). After that, we treat each word pair as a single unit, and the TF-IDF algorithm is applied to the word pairs and further filter out word pairs that are either too common or too rare. Even with the TF-IDF processing, the size of the word pair dictionary is still prohibitively large and selecting only the most frequent ones is a brute force approach. We further cluster semantically close word pairs to reduce the dictionary size. The semantic distance between two words is measured as the distance of their embedding vectors calculated using Googles word2vec model. The only issue is how to determine the cluster centrum. Our first approach is to use upper level words in WordNet BID17 BID16 as cluster centrum. Our hypothesis is that, since those upper-level words have broader meaning, a relatively small number of these words can provide good coverage of the semantical space. We first build a word level tree based on the relations specified in the WordNet. We then pick those words that are closest to the root as the cluster centrum. Words usually have multiple paths to the root. We first add those synsets that have only one path to the root into the graph, then iteratively examine the concepts that we have left and add the paths that grow the tree by as little as possible. Based on which path is selected, this approach can be further divided into min-path clustering and max-path clustering. The min-path clustering will add the shortest path and the max-path clustering will add the longest path to the graph The second approach is K-means clustering. By applying the K-mean algorithm, we group the embedding vector of words into K clusters. Then we use the index of each cluster to represent a group of words. Because the word embedding is a very dense space, the k-mean clustering does not give good Silhouette score BID20. Our original hypothesis is that the K-mean clustering based approach will not be as effective as the WordNet-based approach. However, our experimental show that this is not right. So, we pick the K-means clustering algorithm to be our final feature dictionary organization method. In our experiment, we generate the topic distribution for each document by using RBM model. Then we retrieve the top N documents by calculating Euclidean distance. Our proposed method is evaluated on 3 datasets: OMDb, Reuters, and 20NewsGroup. For the Reuters and 20NewsGroup dataset, we download them from BID2. The OMDb dataset is collected manually by using OMDb APIs (Fritz). All the datasets are divided into three sub-dataset: training, validation, and testing. The split ratio is 70:10:20. For each dataset, a 5-fold cross-validation is applied.• OMDb stands for The Open Movie Database. The training dataset contains 6043 movie descriptions, the validation dataset contains 863 movie descriptions and the testing dataset contains 1727 movie descriptions. We define the class for each movie by applying K-means clustering on category information. The value of K is set to 20.• Reuters, these documents appeared on the Reuters newswire in 1987 and were manually classified by personnel from Reuters Ltd. There are 7674 documents in total which belongs to 8 classes. The training dataset contains 5485 news, the validation dataset contains 768 news and the testing dataset contains 1535 news.• 20NewsGroup, this dataset is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. The training dataset contains 13174 news, the validation dataset contains 1882 news and the testing dataset contains 3765 news. We use mean Average P recision (mAP) score to evaluate our proposed method. It is a score to evaluate the information retrieval quality. This evaluation method considers the effect of order in the information retrieval . If the relational is shown in the front position, the score will tend to 1; if the relational is shown in the back position, the score will tend to 0. mAP 1, mAP 3, mAP 5, and mAP 10 are used to evaluate the retrieval performance. For each document, we retrieve 1, 3, 5, and 10 documents whose topic vectors have the smallest Euclidean distance with that of the query document. The documents are considered as relevant if they share the same class label. Before we calculate the mAP, we need to calculate the Average P recision (AveP) for each document first. The equation of AveP is described below. DISPLAYFORM0 where rel(k) is an indicator function equaling 1 if the item at rank k is a relevant document, 0 otherwise. BID26 Note that the average is over all relevant documents and the relevant documents not retrieved get a precision score of zero. The equation of the mean Average P recision (mAP) score is as following: DISPLAYFORM1 where Q indicates the total number of queries. In this experiment, the total feature size is a fixed number. Then, we compare the performance between two RBM models. One of them only considers words as the input feature, while the other has combined words and word pairs as the input feature. The total feature size varies from 10500, 11000, 11500, 12000, 12500, 15000. For the word/word pair combined RBM model, the number of word is 10000, and the number of word pairs is varied to meet the total feature requirement. Both models are applied to the OMDb dataset, and the are shown in FIG2 and TAB0, the word/word pair combined model almost always performs better than the word-only model. For the mAP 1, the mAP 5 and the mAP 10, the most significant improvement shown in Feature Number = 11000, about 10.48%, 7.97% and 9.83%. For the mAP 3, the most significant improvement shown in Feature Number = 12000, about 9.35%. The two models are further applied on the Reuters dataset, and the are shown in FIG3 and TAB1. Again the word/word pair combined model outperforms the word-only model almost all the time. For the mAP 1, the most significant improvement happens when Feature Number = 12500. Under this feature size, the combined model improves the mAP score by approximately 1.05%. For the mAP 3, the mAP 5 and the mAP 10, the most significant improvement happens when Feature Number = 15000. Under this feature size, the combined model gives about 1.11%, 1.02% and 0.89% improvement. The for 20NewsGroup dataset are shown in Figure 5 and Table 3. Similar to previous two datasets, all the from word/word pair combined model are better than the word-only model. For the mAP 1 the mAP 3, the mAP 5 and the mAP 10, the most significant improvement happens when Feature Number = 11500. And the improvements are about 10.40%, 11.91%, 12.46% and 12.99%.Observing the for all three datasets, we found that they all reflect the same pattern. The word/word pair combined model consistently outperforms the word only model, given that both of them are constructed based on the same number of input features. In the second experiment, we focus on how the different K values affect the effectiveness of the generated word pairs in therms of their ability of topic modeling. First, we average the performance word/word pair combination all K value. The potential K values are 100, 300, 500, 800 and 1000. Then we compare the mAP between our model and the baseline model, which consists of word only input features. The OMDb dataset are shown in Table 4. As we can observe, all the K values give us better performance than the baseline. The most significant improvement shown in K = 100, which are about 2.41%, 2.15%, 1.46% and 4.46% for mAP 1, 3, 5, and 10 respectively. The of Reuters dataset are shown in FIG5 and Table 5. When the K value is greater than 500, all mAP for word/word pair combination model are better than the baseline. Because the mAP score for Reuters dataset in original model is already very high almost all of them higher than 0.9, it is hard to get the improvement as large as OMDb dataset. For the mAP 1, the most significant improvement happens when K = 500, which is 0.31%. For the mAP 5, the mAP 5 and the mAP 10, the most significant improvement shown in K = 800, about 0.50%, 0.38% and 0.42%.The for 20NewsGroup dataset are shown in FIG6 and Table 6. Similar to the Reuters dataset, when the K value is greater than 800, all mAP score for word/word pair combination model are better than the baseline. For the mAP 1, 3, 5, and 10, the most significant im- provements about 2.82%, 2.90%, 3.2% and 3.33% respectively, and they all happened when K = 1000.In summary, a larger K value can give us a better . As we can see from the Reuters dataset and the 20NewsGroup dataset, when K is greater than 800, the combined model outperforms the baseline model in all four mAP evaluation scores. However, the best appear when K = 800 or K = 1000. For the OMDb dataset, because the mAP score for the baseline model is not very high, the combined model gives better than the baseline model with any K value that we tested. In this experiment, we compare different word pair generation algorithms with the baseline. Similar to previous experiments, the baseline is the word-only RBM model whose input consists of the 10000 most frequent words. The "semantic" word pair generation is the method we proposed in this paper. By applying the idea from the skip-gram BID15 algorithm, we generate the word pairs from each word's adjacent neighbor, and we call it "N-gram" word pair generation. And the window size we used in here is N = 2. For the Non-K word pair generation, we use the same algorithm as the semantic except that no K-means clustering is applied on the generated word pairs. The first thing we observe from From the TAB2 is that both "semantic" word pair generation and "Non-K" word pair generation give us better mAP score than the baseline; however, the mAP score of the semantic generation is slightly higher than the mAP score of the Non-K generation. This is because, although both Non-K and semantic techniques extract word pairs using natural language processing, without the K-means clustering, semantically similar pairs will be considered separately. Hence there will be lots of redundancies in the input space. This will either increase the size of the input space, or, in order to control the input size, reduce the amount of information captured by the input set. The K-mean clustering performs the function of compress and feature extraction. The second thing that we observe is that, for the N-gram word pair gener-ation, its mAP score is even lower than the baseline. Beside the OMDb dataset, other two datasets show the same pattern. This is because the semantic model extracts word pairs from natural language processing, therefore those word pairs have the semantic meanings and grammatical dependencies. However, the N-gram word pair generation simply extracts words that are adjacent to each other. When introducing some meaningful word pairs, it also introduces more meaningless word pairs at the same time. These meaningless word pairs act as noises in the input. Hence, including word pairs without semantic importance does not help to improve the model accuracy. In this paper, we proposed a few techniques to processes the dataset and optimized the original RBM model. During the dataset processing part, first, we used a semantic dependency parser to extract the word pairs from each sentence of the text document. Then, by applying a two way parallel TF-IDF processing, we filtered the data in word level and word pair level. Finally, Kmeans clustering algorithm helped us merge the similar word pairs and remove the noise from the feature dictionary. We replaced the original word only RBM model by introducing word pairs. At the end, we showed that proper selection of K value and word pair generation techniques can significantly improve the topic prediction accuracy and the document retrieval performance. With our improvement, experimental have verified that, compared to original word only RBM model, our proposed word/word pair combined model can improve the mAP score up to 10.48% in OMDb dataset, up to 1.11% in Reuters dataset and up to 12.99% in the 20NewsGroup dataset."

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the article for finding a single-sentence tldr

Question: In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. Information concerning absolute position is inherently useful, and it is reasonable to assume that deep CNNs may implicitly learn to encode this information if there is a means to do so. In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks. A comprehensive set of experiments show the validity of this hypothesis and shed light on how and where this information is represented while offering clues to where positional information is derived from in deep CNNs. Convolutional Neural Networks (CNNs) have achieved state-of-the-art in many computer vision tasks, e.g. object classification (; and detection (;, face recognition , semantic segmentation (; ;) and saliency detection . However, CNNs have faced some criticism in the context of deep learning for the lack of interpretability . The classic CNN model is considered to be spatially-agnostic and therefore capsule or recurrent networks have been utilized to model relative spatial relationships within learned feature layers. It is unclear if CNNs capture any absolute spatial information which is important in position-dependent tasks (e.g. semantic segmentation and salient object detection). As shown in Fig. 1, the regions determined to be most salient tend to be near the center of an image. While detecting saliency on a cropped version of the images, the most salient region shifts even though the visual features have not been changed. This is somewhat surprising, given the limited spatial extent of CNN filters through which the image is interpreted. In this paper, we examine the role of absolute position information by performing a series of randomization tests with the hypothesis that CNNs might indeed learn to encode position information as a cue for decision making. Our experiments reveal that position information is implicitly learned from the commonly used padding operation (zero-padding). Zero-padding is widely used for keeping the same dimensionality when applying convolution. However, its hidden effect in representation learning has been long omitted. This work helps to better understand the nature of the learned features in CNNs and highlights an important observation and fruitful direction for future investigation. Previous works try to visualize learned feature maps to demystify how CNNs work. A simple idea is to compute losses and pass these backwards to the input space to generate a pattern image that can maximize the activation of a given unit . However, it is very difficult to model such relationships when the number of layers grows. Recent work presents a non-parametric method for visualization. A deconvolutional network is leveraged to map learned features back to the input space and their reveal what types of patterns a feature map actually learns. Another work proposes to combine pixel-level gradients with weighted class activation mapping to locate the region which maximizes class-specific activation. As an alternative to visualization strategies, an empirical study has shown that a simple network can achieve zero training Cropping in a shift in position rightward of features relative to the centre. It is notable that this has a significant impact on output and decision of regions deemed salient despite no explicit position encoding and a modest change to position in the input. loss on noisy labels. We share the similar idea of applying a randomization test to study the CNN learned features. However, our work differs from existing approaches in that these techniques only present interesting visualizations or understanding, but fail to shed any light on spatial relationships encoded by a CNN model. In summary, CNNs have emerged as a way of dealing with the prohibitive number of weights that would come with a fully connected end-to-end network. A trade-off ing from this is that kernels and their learned weights only have visibility of a small subset of the image. This would seem to imply solutions where networks rely more on cues such as texture and color rather than shape . Nevertheless, position information provides a powerful cue for where objects might appear in an image (e.g. birds in the sky). It is conceivable that networks might rely sufficiently on such cues that they implicitly encode spatial position along with the features they represent. It is our hypothesis that deep neural networks succeed in part by learning both what and where things are. This paper tests this hypothesis, and provides convincing evidence that CNNs do indeed rely on and learn information about spatial positioning in the image to a much greater extent than one might expect. CNNs naturally try to extract fine-level high spatial-frequency details (e.g. edges, texture, lines) in the early convolutional stages while at the deepest layers of encoding the network produces the richest possible category specific features representation;;. In this paper, we propose a hypothesis that position information is implicitly encoded within the extracted feature maps and plays an important role in classifying, detecting or segmenting objects from a visual scene. We therefore aim to prove this hypothesis by predicting position information from different CNN archetypes in an end-to-end manner. In the following sections, we first introduce the problem definition followed by a brief discussion of our proposed position encoding network. Problem Formulation: Given an input image I m ∈ R h×w×3, our goal is to predict a gradient-like position information maskf p ∈ R h×w where each pixel value defines the absolute coordinates of an pixel from left→right or top→bottom. We generate gradient-like masks G pos ∈ R h×w (Sec. 2.2) for supervision in our experiments, with weights of the base CNN archetypes being fixed. Our Position Encoding Network (PosENet) (See Fig. 2) consists of two key components: a feedforward convolutional encoder network f enc and a simple position encoding module, denoted as f pem. The encoder network extracts features at different levels of abstraction, from shallower to deeper layers. The position encoding module takes multi-scale features from the encoder network as input and predicts the absolute position information at the end. Encoder: We use ResNet and VGG based architectures to build encoder networks (f enc) by removing the average pooling layer and the layer that assigns categories. As shown in Fig. 2, the encoder module consists of five feature extractor blocks denoted by (f where W a denotes weights that are frozen. * denotes the convolution operation. Note that in probing the encoding network, only the position encoding module f pem is trained to focus on extracting position information while the encoder network is forced to maintain their existing weights. Once we have the same spatial dimension for multi-scale features, we concatenate them together followed by a sequence of k × k convolution operations. In our experiments, we vary the value of k between {1, 3, 5, 7} and most experiments are carried out with a single convolutional layer in the position encoding module f pem. The key operations can be summarized as follows: where W c pos is the trainable weights attached with the transformation function T pos. The main objective of the encoding module is to validate whether position information is implicitly learned when trained on categorical labels. Additionally, the position encoding module models the relationship between hidden position information and the gradient like ground-truth mask. The output is expected to be random if there is no position information encoded in the features maps and vice versa (ignoring any guidance from image content). To validate the existence of position information in a network, we implement a randomization test by assigning a normalized gradient-like 1 position map as ground-truth shown in Fig. 3. We first generate gradient-like masks in Horizontal (H) and vertical (V) directions. Similarly, we apply a Gaussian filter to design another type of ground-truth map, Gaussian distribution (G). The key motivation of generating these three patterns is to validate if the model can learn absolute position on one or two axes. Additionally, We also create two types of repeated patterns, horizontal and vertical stripes, (HS, VS). Regardless of the direction, the position information in the multi-level features is likely to be modelled through a transformation by the encoding module f pem. Our design of gradient ground-truth can be considered as a type of random label because there is no correlation between the input image and the ground-truth with respect to position. Since the extraction of position information is independent of the content of images, we can choose any image datasets. Meanwhile, we also build synthetic images, e.g. black, white and Gaussian noise to validate our hypothesis. As we implicitly aim to encode the position information from a pretrained network, we freeze the encoder network f enc in all of our experiments. Our position encoding module f pem generates the position mapf p of interest. During training, for a given input image I m ∈ R h×w×3 and associated ground-truth position map G h pos, we apply the supervisory signal onf p by upsampling it to the size of G h pos. Then, we define a pixel-wise mean squared error loss to measure the difference between predicted and ground-truth position maps. The overall objective function of our network can be written as: where x ∈ IR n and y ∈ IR n (n denotes the spatial resolution) are the vectorized predicted position and ground-truth map respectively. x i and y i refer to a pixel off p and G dataset. The synthetic images (white, black and Gaussian noise) are also used as described in Section 2.2. Note that we follow the common setting used in saliency detection just to make sure that there is no overlap between the training and test sets. However, any images can be used in our experiments given that the position information is relatively content independent. Evaluation Metrics: As position encoding measurement is a new direction, there is no universal metric. We use two different natural choices for metrics (Spearmen Correlation (SPC) and Mean Absoute Error (MAE)) to measure the position encoding performance. The SPC is defined as the Spearman's correlation between the ground-truth and the predicted position map. For ease of interpretation, we keep the SPC score within range [-1 1]. MAE is the average pixel-wise difference between the predicted position map and the ground-truth gradient position map. We initialize the architecture with a network pretrained for the ImageNet classification task. The new layers in the position encoding branch are initialized with xavier initialization. We train the networks using stochastic gradient descent for 15 epochs with momentum of 0.9, and weight decay of 1e−4. We resize each image to a fixed size of 224×224 during training and inference. Since the spatial extent of multi-level features are different, we align all the feature maps to a size of 28 × 28. We report experimental for the following baselines that are described as follows: VGG indicates PosENet is based on the features extracted from the VGG16 model. Similarly, ResNet represents the combination of ResNet-152 and PosENet. PosENet alone denotes only the PosENet model is applied to learn position information directly from the input image. H, V, G, HS and VS represent the five different ground-truth patterns, horizontal and vertical gradients, 2D Gaussian distribution, horizontal and vertical stripes respectively. Position Information in Pretrained Models: We first conduct experiments to validate the existence of position information encoded in a pretrained model. Following the same protocol, we train the VGG and ResNet based networks on each type of the ground-truth and report the experimental in Table 1. We also report when we only train PosENet without using any pretrained model to justify that the position information is not driven from prior knowledge of objects. Our experiments do not focus on achieving higher performance on the metrics but instead validate how much position information a CNN model encodes or how easily PosENet can extract this information. Note that, we only use one convolutional layer with a kernel size of 3 × 3 without any padding in the PosENet for this experiment. As shown in PosENet can extract position information consistent with the ground-truth position map only when coupled with a deep encoder network. As mentioned prior, the generated ground-truth map can be considered as a type of randomization test given that the correlation with input has been ignored. Nevertheless, the high performance on the test sets across different groundtruth patterns reveals that the model is not blindly overfitting to the noise and instead is extracting true position information. However, we observe low performance on the repeated patterns (HS and VS) compared to other patterns due to the model complexity and specifically the lack of correlation between ground-truth and absolute position (last two rows of Table 1). The H pattern can be seen as one quarter of a sine wave whereas the striped patterns (HS and VS) can be considered as repeated periods of a sine wave which requires a deeper comprehension. The qualitative for several architectures across different patterns are shown in Fig. 4. We can see the correlation between the predicted and the ground-truth position maps corresponding to H, G and HS patterns, which further reveals the existence of position information in these networks. The quantitative and qualitative strongly validate our hypothesis that position information is implicitly encoded in every architecture without any explicit supervision towards this objective. Moreover, PosENet alone shows no capacity to output a gradient map based on the synthetic data. We further explore the effect of image semantics in Sec. 4.1. It is interesting to note the performance gap among different architectures specifically the ResNet based models achieve higher performance than the VGG16 based models. The reason behind this could be the use of different convolutional kernels in the architecture or the degree of prior knowledge of the semantic content. We show an ablation study in the next experiment for further investigation. For the rest of this paper, we only focus on the natural images, PASCAL-S dataset, and three representative patterns, H, G and HS. In this section, we conduct ablation studies to examine the role of the proposed position encoding network by highlighting two key design choices. the role of varying kernel size in the position encoding module and stack length of convolutional layers we add to extract position information from the multi-level features. Table 1 show the existence of position information learned from an object classification task. In this experiment, we change the design of PosENet to examine if it is possible to extract hidden position information more accurately. The PosENet used in the prior experiment (Table 1) has only one convolutional layer with a kernel size of 3 × 3. Here, we apply a stack of convolutional layers of varying length to the PosENet and report the experimental in Table 2 (a). Even though the stack size is varied, we aim to retain a relatively simple PosENet to only allow efficient readout of positional information. As shown in Table 2, we keep the kernel size fixed at 3 × 3 while stacking multiple layers. Applying more layers in the PosENet can improve the readout of position information for all the networks. One reason could be that stacking multiple convolutional filters allows the network to have a larger effective receptive field, for example two 3 × 3 convolution layers are spatially equal to one 5 × 5 convolution layer. An alternative possibility is that positional information may be represented in a manner that requires more than first order inference (e.g. a linear readout). Our previous experiments reveal that the position information is encoded in a pretrained CNN model. It is also interesting to see whether position information is equally distributed across the layers. In this experiment, we train PosENet on each of the extracted features, f separately using VGG16 to examine which layer encodes more position information. Similar to Sec. 3.3, we only apply one 3 × 3 kernel in F pem to obtain the position map. As shown in Table 3, the VGG based PosENet with top f 5 pos features achieves higher performance compared to the bottom f 1 pos features. This may partially a of more feature maps being extracted from deeper as opposed to shallower layers, 512 vs 64 respectively. However, it is likely indicative of stronger encoding of the positional information in the deepest layers of the network where this information is shared by high-level semantics. We further investigate this effect for VGG16 where the top two layers (f We believe that the padding near the border delivers position information to learn. Zero-padding is widely used in convolutional layers to maintain the same spatial dimensions for the input and output, with a number of zeros added at the beginning and at the end of both axes, horizontal and vertical. To validate this, we remove all the padding mechanisms implemented within VGG16 but still initialize the model with the ImageNet pretrained weights. Note that we perform this experiment only using VGG based PosENet since removing padding on ResNet models will lead to inconsistent sizes of Table 4 : Quantitative comparison subject to padding in the convolution layers used in PosENet and VGG (w/o and with zero padding) on natural images. skip connections. We first test the effect of zero-padding used in VGG, no padding used in PosENet. As we can see from Table 4, the VGG16 model without zero-padding achieves much lower performance than the default setting(padding=1) on the natural images. Similarly, we introduce position information to the PosENet by applying zero-padding. PosENet with padding=1 (concatenating one zero around the frame) achieves higher performance than the original (padding=0). When we set padding=2, the role of position information is more obvious. This also validates our experiment in Section 3.3, that shows PosENet is unable to extract noticeable position information because no padding was applied, and the information is encoded from a pretrained CNN model. This is why we did not apply zero-padding in PosENet in our previous experiments. Moreover, we aim to explore how much position information is encoded in the pretrained model instead of directly combining with the PosENet. Fig. 6 illustrates the impact of zero-padding on encoding position information subject to padding using a Gaussian pattern. Recall that the position information is considered to be content independent but our in Table 1 show that semantics within an image may affect the position map. To visualize the impact of semantics, we compute the content loss heat map using the following equation: As shown in Figure 7, the heatmaps of PosENet have larger content loss around the corners. While the loss maps of VGG and ResNet correlate more with the semantic content. Especially for ResNet, the deeper understanding of semantic content leads to a stronger interference in generating a smooth gradient. The highest losses are from the face, person, cat, airplane and vase respectively (from left to right). This visualization can be an alternative method to show which regions a model focuses on, especially in the case of ResNet. Saliency Detection: We further validate our findings in the position-dependent tasks (semantic segmentation and salient object detection (SOD)). First, we train the VGG network with and without zero-padding from scratch to validate if the position information delivered by zero-padding is critical for detecting salient regions. For these experiments, we use the publicly available MSRA dataset as our SOD training set and evaluate on three other datasets (ECSSD, PASCAL-S, and DUT-OMRON). From Table 5 (a), we can see that VGG without padding achieves much worse on both of the metrics (F-measure and MAE) which further validates our findings that zero-padding is the key source of position information. Semantic Segmentation: We also validate the impact of zero-padding on the semantic segmentation task. We train the VGG16 network with and without zero padding on the training set of PASCAL VOC 2012 dataset and evaluate on the validation set. Similar to SOD, the model with zero padding significantly outperforms the model with no padding. We believe that CNN models pretrained on these two tasks can learn more position information than classification task. To validate this hypothesis, we take the VGG model pretrained on ImageNet as our baseline. Meanwhile, we train two VGG models on the tasks of semantic segmentation and saliency detection from scratch, denoted as VGG-SS and VGG-SOD respectively. Then we finetune these three VGG models following the protocol used in Section 3.3. From Table 6, we can see that the VGG-SS and VGG-SOD models outperform VGG by a large margin. These experiments further reveal that the zero-padding strategy plays an important role in a position-dependent task, an observation that has been long-ignored in neural network solutions to vision problems. Table 6: Comparison of VGG models pretrained for classification, SOD, and semantic segmentation. In this paper we explore the hypothesis that absolute position information is implicitly encoded in convolutional neural networks. Experiments reveal that positional information is available to a strong degree. More detailed experiments show that larger receptive fields or non-linear readout of positional information further augments the readout of absolute position, which is already very strong from a trivial single layer 3 × 3 PosENet. Experiments also reveal that this recovery is possible when no semantic cues are present and interference from semantic information suggests joint encoding of what (semantic features) and where (absolute position). Results point to zero padding and borders as an anchor from which spatial information is derived and eventually propagated over the whole image as spatial abstraction occurs. These demonstrate a fundamental property of CNNs that was unknown to date, and for which much further exploration is warranted.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Our work shows positional information has been implicitly encoded in a network. This information is important for detecting position-dependent features, e.g. semantic and saliency.

Represent the short summary for finding the corresponding scientific article (long)

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.