Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Natural Language Processing models lack a unified approach to robustness testing. In this paper we introduce WildNLP - a framework for testing model stability in a natural setting where text corruptions such as keyboard errors or misspelling occur. We compare robustness of models from 4 popular NLP tasks: Q&A, NLI, NER and Sentiment Analysis by testing their performance on aspects introduced in the framework. In particular, we focus on a comparison between recent state-of-the- art text representations and non-contextualized word embeddings. In order to improve robust- ness, we perform adversarial training on se- lected aspects and check its transferability to the improvement of models with various cor- ruption types. We find that the high perfor- mance of models does not ensure sufficient robustness, although modern embedding tech- niques help to improve it. We release cor- rupted datasets and code for WildNLP frame- work for the community. Adversarial examples have been shown to severely degrade performance of deep learning models BID10 BID14. Natural Language Processing systems are no different in this respect. Multiple areas of NLP, such as machine translation BID1, question answering BID12, or text classification have been studied to assess the impact of adversaries generated with various methods. However, these works tend to focus on one area only, often with attacks designed just for the selected problem. It makes comparisons between models, datasets, and NLP areas impossible. In particular, the robustness of modern word embedding systems -such as ELMo BID17, Flair BID0 and language model based BERT BID5 remains unstudied. In this article, we evaluate the behavior of natural language models in the wild. We propose WildNLP -a systematic and comprehensive robustness testing framework which can be used for any NLP model. Instead of focusing on elaborate attacks, which are unlikely to originate by accident, we measure the quality of models in a natural setting, where input data is poisoned with errors involuntarily generated by actual users. We put these notions into a set of tests called aspects. Moreover, we introduce the concept of corruption severity and prove that it is critical to model improvement via adversarial training. The framework is aimed at any NLP problem irrespective of its form of input and output. In summary, our contributions are the following:1. We offer a systematic framework for testing corruption robustness -the WildNLP.In total, we introduce 11 aspects of robustness testing, with multiple severity levels. We release the code and a collection of popular datasets that are corrupted with WildNLP for the community 1. The framework is easy to extend. New aspects can be defined by the community.2. We test corruption robustness of a number of NLP tasks: question answering (Q&A), natural language inference (NLI), named entity recognition (NER), and sentiment analysis (SA). We verify stability of models trained on contextualized embeddings like ELMo and Flair in contrast to noncontextualized FastText BID2 and GloVe BID16.We also analyze BERT in the task of Q&A. We find that new forms of text representation, despite greater contextual awareness, do not offer a sufficient increase in robustness.3. We find that model training on one aspect does improve performance on another aspect, contrary to previous studies BID1. For this to be true, two corruption types must be similar to some extent. In section 2 we present related literature in the domain of NLP robustness. In section 3 we present WildNLP framework, describing in detail each introduced aspect. In section 4 we compare robustness of NER, Q&A, NLI and Sentiment Analysis. In section 5 we perform adversarial training on Qwerty aspect with different severities and test these models on other aspects. We conclude in section 6. The problem of natural noise in textual data has been studied by BID1, however exclusively in the context of character-based machine translation models. They find that errors such as typos and misspelling cause significant drops in BLEU scores. Other recent approaches to generating textual adversaries include the work of, who exploit important word manipulations for text classification models from 2014 and 2015. BID7 identify important words and apply 4 kinds of character perturbations: swap, substitution, deletion and insertion. They test on vanilla LSTM and CNN model, applying them to 8 datasets. Among others, they aim for the character swaps to map a word vector to an'unknown' vector in traditional word embeddings. BID19 create rules of substitutions between texts which produce correct and semantically identical samples in Q&A domain. BID9 design adversaries for NLI systems, swapping words which share a relation such as antonymy or co-hyponymy. 3. Targeted robustness. These are the attacks designed for a specific problem and/or dataset, or demanding access to model internals. An example is the whole class of white box attacks BID6 as well as highly specialized attacks BID12. The WildNLP aspects define classes of common disturbances found in natural text. These corruptions are produced naturally due to haste, lacking space, individual writing habits or imperfect command of English. Articles. Randomly removes or swaps articles into wrong ones. Swap. Randomly shuffles two characters within a word. Qwerty. Simulates errors made while writing on a QWERTY-type keyboard. Characters are swapped for their neighbors on the keyboard. Remove char. Randomly removes characters from words. Remove space Removes a space from text, merging two words. Misspelling. Misspells words appearing in the Wikipedia list of commonly misspelled English words 2.Digits2words. Rewrites digit numbers into words. Homophones. Changes words into their homophones from the Wikipedia list of common misspellings/homophones 3. The list contains around 500 pairs or triples of homophonic words. Negatives. This aspect reflects attempts made by some Internet users to mask profanity Warsaw was believed to be one of the most beautiful cities in the world. Warsaw was believed to be one of a most beautiful cities in world. Warsaw aws believed to be one fo teh most beautiful cities in the world. Wadsaw was bdlieved to be one of the most beautiful citiee in the world.. Warsaw was believed to be one o th most eautiful cities in the world. Warsaw was believed tobe one of the most beautiful cities in the world. You cannot accidentally commit vandalism. Vandalism used to be a rare occurrence. You can not accidentaly commit vandalism. Vandalism used to be a rare occurrance. Bus Stops for Route 6, 6.1 Digits2words Bus Stops for Route six, six point one OriginalChoosing between affect and effect can be scary. Choosing between effect and effect can bee scary. Original Laughably foolish or false: an absurd explanation. Negatives Laughab*y fo*lish or fal*e: an a*surd explanation. OriginalSometimes it is good to be first, and sometimes it is good to be last. Sometimes it is go*d to be first, and sometimes it is goo* to be last. Sometimes, it is good to be first and sometimes, it, is good to be last.or hate speech in online forums to evade moderation. We perform masking of negative words from Opinion Lexicon 4. The lexicon contains a list of English positive and negative opinion words or sentiment words, in total around 6800 words. Positives. Masks positive words from Opinion Lexicon, similarly as in the case of Negatives (described above).Marks. Randomly removes and insert punctuation marks. Marks are inserted between last letter of a word and space. The severity of perturbations can be varied. In the case of Swap, Qwerty and Remove char we control it by defining how many words will be affected. In the case of Article, it is defined by a probability of corruption of each article. We test corruption robustness on various NLP tasks and models. Each of the models is run on the specific dataset it has been trained on in the original setting. The datasets are preprocessed by the WildNLP framework to obtain corrupted data with multiple aspects. An important point in the experimental setting is the application of various word embeddings. We focus on testing the robustness of models trained with newly introduced context-aware embeddings: ELMo, Flair and language model based BERT. We compare their performance on corrupted data to older embedding systems -GloVe, FastText (within InferSent) and in the case of one of sentiment analysis models, even one-hot encoded words. We do so to verify the assumption that greater context awareness and lack of problems with out-of-vocabulary (OOV) words in ELMo, Flair and BERT would increase robustness of models. We use our framework on the selection of well known models that are widely used in NLP community. For training ELMo-based models we use open-source implementations available in AllenNLP BID8, for BERT we follow implementation of HuggingFace 5 and for the rest of the models we use original author research code. In particular, following models and datasets are used in experiments:• Q&A task Models. We test BiDAF and BERT trained on the SQuAD dataset BID18. We analyze two versions of BiDAFwith ELMo (BiDAF-E) and GloVe (BiDAF-G) embeddings. BiDAF uses character and word embeddings with a bidirectional attention flow to obtain a query-aware context representation. BiDAF is one of the state-of-theart models on the SQuAD leaderboard. On the other hand, BERT applies a bidirectional Transformer to language modeling task and is currently used with great success in various NLP tasks, achieving the new state-ofthe-art. We evaluate the models with the common performance scores in Q&A task, which are Exact Match (EM) and F1 score. Dataset. SQuAD dataset comprises around 100,000 question-answer pairs prepared by crowdworkers. The dataset is based on Wikipedia articles. TAB3 displays examples of the question-answer pairs.• Table 3 contains an example of the three possible entailment relations.• NER task Models. We use two sequence tagging models with ELMo implementation (CRF-E) and Flair BID0. Flair comprises new word embeddings an a BiLSTM-CRF sequence labeling system. It models words as sequences of characters, which allows to effectively eliminate the notion of separate tokens. Flair is currently the state-of-the-art model in NER task. Dataset. The CoNLL 2003 dataset is a standard training dataset used in NER sequence tagging. It is a collection of news articles from Reuters corpus annotated as Person, Organization, Location, Miscellaneous, or Other for non-named entities. Due to licensing agreement this is the only corrupted dataset that we cannot release.• SA task Models. We use the current state-of-the-art ULMFiT model BID11 that consists of language model pretrained on Wikipedia and fine-tuned on the specific text corpus that is used in classification task. In adversarial training scenario, we pretrain this language model on corrupted data. We TAB1 and TAB3. For Q&A models, EM measure is displayed.compare ULMFiT with CNN based classification model, which uses one-hot encoding of words. Dataset. We train and test described models on IMDB dataset that consists of 25000 positive and 25000 negative reviews of movies. Figure 1 (Q&A models) and Figure 2 (other models) present aggregate of testing on all models and all corruption aspects. Variability and scale of performance drops are depicted in FIG2. Tables with full can be found in Appendix A. Robustness measure. To comprehensively measure model robustness to corruptions, we calculate an overall mean of drops across all aspects (Av-Drop). We use this aggregated metric to compare robustness between models. Q&A. The robustness of Q&A models was the lowest of all tested tasks. The corruptions which proved most damaging to the performance and in to Av-Drop were the following: Swap 5 (32 -37 EM drop), Remove char 5 (29 -37 EM drop), Qwerty 5 (25 -30 EM drop).BERT and ELMo-based systems were found to mitigate performance loss to some degree compared to GloVe. However, their performance loss pattern across corruptions was similar to GloVe, and the difference of Av-Drop between BERT (most robust model) and BiDAF GloVe (least robust model) was 2.8 pp, despite huge performance differences reflected in F1 and EM.We observe that severity of aspects plays an important role in drop of performance metrics across all Q&A models. For aspects that corrupt individual words like Qwerty, Remove char or Swap, drop in performance of GloVe-based models is intuitive -we substitute words from out of vocabulary (OOV) with unknown token. However, in the case of ELMo and BERT the problem of OOV tokens is not that severe -they are character or subword-based. Still, we observe an average drop of F1 metric on these three aspects (severity 5) at the level of 23.04 (BiDAF-E) and 24.46 (BERT) in comparison to drop of BiDAF-G at 32.9. Lower severities of word corruptions induce much lower drops -in case of severity 1 it is still a noticeable difference of 4.48 (BiDAF-E), 3.44 (BERT) and 5.63 (BiDAF-G).WildNLP also tests on aspects that do not alter words but sentences. As previously, we state that context-aware models should be indifferent to such changes as they do not alter sentence meaning. However, we observe that aspects such as Remove space and Marks decrease F1 values among all Q&A even by 8.89 in case of Remove space tested with BiDAF-E, whereas BERT proves to be more robust to this sentencelevel corruption with drop of F1 at 2.47.NLI. Natural Language Inference task tested by WildNLP framework is more robust when trained with decomposable attention model with ELMo embeddings (Dec-E) rather than simple MLP classifier that uses sentence embeddings created by InferSent method (InferSent). The Av-Drop for Dec-E is half the value of Av-Drop for InferSent, being at the level of 4.19. On all sets of aspects, Dec-E model has lower drops of performance metric. However, it still has relatively high drops when it comes to word corruption aspects like Qwerty, Remove char or Swap, with average drop of 10.92 at severity 5 and 2.09 at severity 1. InferSent performs worse by around 3 pp (5.56 and 12.82 respectively).However, when we consider sentence level aspects like adding extra commas to the sentence, Dec-E model is very robust, having only 0.85 of drop in accuracy on highest possible severity. NER. Both NER models seems to be robust, having the Av-Drop measure at the level of 2.37 (CRF-E) and 2.14 (Flair). However, in the case of state-of-the-art NER models, differences in performance are so small, that such relatively small values of Av-Drop can be seen as high. As we processed only non-NE words with WildNLP framework, we assume that model predictions of named entities are dependent on surrounding context words. SA. ULMFiT model was found to be slightly less robust than CNN using one-hot encodings (2.36 vs 2.28 of Av-Drop). Drop in performance of the CNN model was mainly caused by Positives and Negatives corruptions (7.22 and 9.7 Av-Drop) which can be observed as the two outliers in FIG2. Presumably this behavior is caused by the model's focus on detecting sentiment-carrying words, which were on average rarely affected by other corruptions. On the other hand, ULMFiT was less affected by Positives and Negatives corruptions (3.6 and 4.2 AvDrop) probably because of its reliance on context and more subtle expressions of sentiment. In spite of the fact that the CNN model suffered from out-of-vocabulary words problem (corrupted words were simply unrecognized) while ULMFiT did not, the CNN proved more robust to most deformations in WildNLP framework. We use adversarial training to research the potential of overcoming corruption errors. We validate two hypotheses:1. Adversarial training on data corrupted with aspects of greater severity helps to resolve problems with data corrupted with lesser severity. For example, training on Qwerty 5-corrupted data should increase performance of data corrupted with Qwerty 1 up to Qwerty 5 severities.2. Adversarial training on one corruption type should increase model robustness to other corruptions. However, BID1 suggest that this is not the case. They find that models trained on one type of noise do not perform well on others in character-based translation models. Based on this, we hope to prove that robustness can be improved between aspects which are related. Corruption severity. In agreement with our hypothesis we find that increased severity of corruption during training does increase performance on data corrupted with the same aspect type but lesser severity. TAB6 presents numeric scores for the training setting in Q&A BiDAF ELMo models, while FIG3 shows plots for multiple models. In all scenarios, we test on Qwerty 1 and Qwerty 5 corruptions. Interestingly, in the case of NER models, obtained on models trained on both corruption types are even better than for the original model (for Qwerty 5 model, this behavior is consistent across levels of severity of test data perturbations).Empirically, the severity of Qwerty perturbation (and others) does make the text unintelligible for humans at some point. For example, this boundary was found to be level 5 for Q&A questions. However, the Q&A BiDAF ELMo model trained on Qwerty 5 performs reasonably well even at severity level 8. This suggests that the model learned to decode this corruption even beyond human ability. Relation between corruption types. To verify relations between performance of models trained and tested on various corruption types, we test models trained on Qwerty corruption with severity 1 and 5. Qwerty exhibits similarities to Swap and Remove char types, since all of them imply manipulations of word characters. We observe that Bidaf ELMo and NER ELMo models trained on Qwerty and tested on similar aspects perform better than original models not trained in adversarial setting. Results are depicted in FIG4. In this work, we have presented the WildNLP framework for corruption robustness testing. We have introduced 11 text corruption types (at various severity levels) which can occur naturally in model deployment setting: misspellings, keyboard errors, attempts at masking emotional language, and others. We test on four NLP areas and 12 models in total, verifying corruption robustness of state-of-the-art BERT system and new LM-based embeddings: ELMo and Flair, contrasted with GloVe and Fasttext. We find that the problem of lacking corruption robustness is not solved by these recent systems. However, we find that the issue can be partially alleviated by adversarial training, even across aspects. We believe that problem of adversarial examples in NLP is still vague and hard to quantify. Without doubt, more work is needed to make models robust to natural noise, whether by robust word embeddings, model architectures, or better datasets.

represent the tldr for finding the corresponding scientific article:

For instance, <<An efficient multi-view video summarization scheme advanced to activity recognition in IoT environments.>>  to <<Multi-view video summarization (MVS) lacks researchers’ attention due to their major challenges of inter-view correlations and overlapping of cameras. Most of the prior MVS works are offline, relying on only summary, needing extra communication bandwidth and transmission time with no focus on uncertain environments. Different from the existing methods, we propose edge intelligence based MVS and spatio-temporal features based activity recognition for IoT environments. We segment the multi-view videos on each slave device over edge into shots using light-weight CNN object detection model and compute mutual information among them to generate summary. Our system does not rely on summary only but encode and transmit it to a master device with neural computing stick (NCS) for intelligently computing inter-view correlations and efficiently recognizing activities, thereby saving computation resources, communication bandwidth, and transmission time. Experiments report an increase of 0.4 in F-measure score on MVS Office dataset as well as 0.2% and 2% increase in activity recognition accuracy over UCF-50 and YouTube 11 datasets, respectively, with lower storage and transmission time compared to state-of-the-art. The time complexity is decreased from 1.23 to 0.45 secs for a single frame processing, thereby generating 0.75 secs faster MVS. Furthermore, we made a new dataset by synthetically adding fog to an MVS dataset to show the adaptability of our system for both certain and uncertain surveillance environments. Surveillance cameras installed indoor and outdoor at offices, public places, and roads generate huge amount of video data on daily basis. This gigantic volume of data has two big issues: first one is storage consumption and second is huge computational complexity for its purposeful usage. Video summarization aims at these problems by condensing the data size via extracting key information from lengthy videos and suppressing the redundant frames. A video summary generated from a single camera is called single-view video summarization (SVS) . On the other hand, a summary generated from a camera network is known as MVS (a). SVS is intensively researched with applications to various domains including surveillance , sports , and news videos . In contrast, MVS is not studied deeply because of several challenges such as computing inter-and intra-view correlations, overlapping regions among connected cameras, and variation in light conditions among different views. The basic flow of MVS includes input acquisition, preprocessing, feature extraction, post-processing, and summary generation. The mainstream MVS methods follow traditional machine learning approaches such as clustering along with low-level features extracted from entire frame with no focus on specific targets in surveillance. The most important part of MVS is considering different objects in surveillance that can be useful for summary generation. However, the existing techniques do not focus on objects such as persons and vehicles while generating summary. Thus, the final summary may miss some important frames having persons or vehicles that need to be considered for MVS. Furthermore, all the existing techniques rely only on MVS with no further steps for analysis of the generated summary. For instance, the generated summary can be used for indexing, browsing, and activity recognition. The existing methods are functional only in certain environments with no focus on uncertain scenarios , making them inadequate in real-world environments. Finally, all the existing methods process data on local/online servers or personal computers with huge computation power. It requires extra processing time, power of transmission, and does not guarantee quick responsive action for any abnormal situations, if not handled on the edge. To ensure proper and quick responsive arrangements, activity recognition at edge is a necessary requirement of the current technological era. Activity recognition literature is mature, but with no focus on processing over the edge. Almost all the existing techniques classify activities over high computational local or cloud servers. Classifying activity on edge is an important task of surveillance in smart cities. Therefore, to tackle these challenges effectively, we present a novel framework applicable in both certain and uncertain environments for MVS and activity recognition over the edge. Figure 1: Input and output flow of our proposed framework. (a) Video frames (both certain and uncertain environment) from resource constrained devices. (b) Annotate frames by detecting objects of interest, apply keyframes selection mechanism, generate summary, encode and transmit it to master device. (c) Decode generated summary, perform features extraction, and forward it to activity prediction model at master device to get the output class with probability score. The problems aimed in this paper are different from the schemes presented in existing literature. We integrated two different domains including MVS and activity recognition under the umbrella of a unified framework in an IoT environment. We presented interconnected resource constrained IoT devices working together to achieve several targets i.e., object detection, summary generation, and activity recognition as shown in Figure 1. The overall framework consists of numerous slaves and a master resource constrained device connected through a common wireless sensor network (WSN). The slave devices are equipped with a camera to capture multi-view video data, segment it into shots, generate summary, encode a sequence of keyframes, and transmit it to the master device. The master device is equipped with an INTEL Movidius NCS to classify the ongoing activity in the acquired sequence. INTEL Movidius is a modular and deep learning accelerator in a standard USB 3.0 stick. It has a Vision Processing Unit (VPU) that is functional with ultra-low power and better performance. It enables activity recognition with significantly lower power, storage, and computational cost. Further, a widely used concept of temporal point processing is utilized for activity classification, ensuring an effective recognition model. While addressing the problems in MVS and activity recognition over resource constrained devices, we made the following contributions. • Employing an algorithm for MVS on resource constrained devices, reducing the time complexity compared to existing approaches with higher accuracy. The generated summary is further utilized to recognize the underlying activity of all the views through an auto-encoder and learned spatio-temporal features followed by different variants of SVM classifiers to demonstrate the efficiency and effectiveness of our proposed framework. • Adding uncertainties such as fog to an outdoor MVS benchmark dataset to demonstrate the working of proposed framework in any type of scenario and introduce a new trend in MVS literature for researchers. • The presented framework has high-level adaptability with special care for the capacity and traffic of WSN. It has many flavors with tradeoff among transmission time, quality of keyframes, and accuracy of activity recognition model with computationally different classifiers. In the subsequent sections, Section 2 provides a literature review and Section 3 explains the presented framework in detail. In Section 4, experimental for MVS and activity recognition are given, and Section 5 concludes the overall paper with future directions. This section is mainly divided into two sub-sections, covering the representative studies of MVS and activity recognition related to our work. The literature of MVS was initiated almost a decade ago in 2010 by with the first MVS indoor dataset. The employed MVS methods are computationally complex but they are not suitable for resource constrained devices. In contrast to MVS, activity recognition is a richer research field with a variety of techniques focusing on different applications. Majority of the MVS methods are based on handcrafted-features integrated with traditional machine learning approaches for final summary generation. The initial MVS schemes utilized features such as SIFT descriptors, Gaussian and Laplacian difference, and motion features followed by statistical learning approaches such as K-means or other clustering techniques for summary generation. The next trend of MVS (; b) used the same features by adding pre-or post-processing like subtraction along with supervised learning such as SVM or unsupervised learning. The final summary in this trend is either uniform length or of length provided by user. The next MVS trend (; a) utilized mid-level features, motion-based shot boundary detection, spatiotemporal graphs, and low-level features such as color and edge histograms. The final summary is generated through the same statistical learning-based techniques. A breakthrough in MVS field is noticed after the usage of learned features in prerequisite steps for generating summaries. This trend is followed in 2016 (a), where BVLC CaffeNet and Spatiotemporal C3D features are used for sparse coding and video representation, respectively. Besides clustering, template matching and sparse representative selection over learned embedding are used for generating final summaries. Panda et al. also used C3D features for video representation by computing inter-and intra-view similarities through sparse coefficients and summary generation using clustering. The recent success of deep learning based methods in activity recognition achieved high-level accuracies. These methods extract features from final layers of various deep learning models and apply sequential learning for activity recognition . For instance, used features extracted from a pre-trained AlexNet, followed by bi-directional LSTM for human action recognition with better . Following this, explored optical flow convolutional features with multi-layer LSTM for activity recognition. Their approach outperformed and other state-of-the-art, however, its huge running time restricts its adoptability in real-world surveillance networks. Similar to the previous methods, used different types of CNN architectures and configurations for human action representation, considering both stationary and non-stationary environments. Till date, the activity recognition methods do not perform computation over resource constrained devices with significant accuracy. Thus, to tackle this problem effectively, we optimized VGG-19 model to make it functional over resource constrained devices with details given in Section 3. The overall framework is divided into two major modules based on the functionalities of the resource constrained devices. The first module is related to multi-view video acquisition, shot segmentation, and summary generation. The second module performs computationally expensive processes involving deep features extraction, features comparison and encoding, and activity prediction. The whole framework contains a number of resource constrained devices (master and slave) connected to a WSN in IoT. This section provides details of the proposed system with major steps are given in appendix section A and visualized in Figure 2. Figure 2: The conceptual diagram of our proposed framework, where slave devices with camera capture multi-view video data, apply shot segmentation, extract keyframes, encode and transmit them to master device for computing inter-view correlations and activity recognition. Surveillance analysts are usually interested in activities of humans and vehicles; thus we consider them as objects of our interest. For this purpose, a tiny version of YOLO CNN model is employed, which is much faster and has higher accuracy compared to other detectors. This tiny object detector works well in normal scenarios but it has poor performance when videos contain fog. To overcome this limitation, we used the pre-trained weights of YOLO tiny model and retrained them over foggy data as show in experimental section 3. Fog is applied on the entire image globally without targeting our objects of interest only. Adding fog globally has an advantage of better learning, because in real-life surveillance scenarios, the fog is always observed throughout the image. The foggy data is generated from the COCO dataset by adding synthetic fog to all the images using MATLAB script with different levels of fog, ranging from 0 to 255. We conducted experiments with different values of fog parameter (f p) as shown in experimental section and selected f p = 220 as optimal. The process of foggy data creation is given in Algorithm A in appendix section. Training data for object detection specifically in foggy/normal environment has an additional advantage of reliability. Most of the available multi-view datasets are related to indoor such as Office and Bl-7f. Thus, we conducted experiments on the available multi-view outdoor datasets only, such as Road . The trained model is used over each slave device for object detection. The attached camera of device transmits frames to our customized object detection model. The frames having persons or vehicles with confidence score greater than 0.9 are stored and considered for further steps while the remaining frames are discarded. A summary is generated from the segmented frames with objects, because events occur due to their interaction. Therefore, we considered those shots that can possibly contain events such as sitting on a chair and entering a room etc. in MVS datasets. These events are possible only due to human presence. Due to this reason, we segmented those shots in which humans are detected. The shots with desired objects are larger in number, containing redundant frames and thus cannot be considered in the final summary. To avoid the redundancy and provide users a very compact representation, we compute mutual information between features of consecutive frames. This process is executed over slave devices, having limited execution resources, thus, we used low-level features intelligently. Further, instead of using only Euclidean or any other distance measuring method, we compared mutual information between two consecutive feature vectors to avoid redundancy and obtain better . The feature descriptor is acquired from each frame by computing the ORB points . The frames with persons, having exactly the same information, are discarded and only a single frame is selected from them. The selected frame is considered as a keyframe on the concerned resource constrained device. The formula used for mutual information is given in Eq. 1. Here f p is the previous frame while f n is the current frame and N is the size of feature vector. The mutual information is computed using scikit-learn library in Python. Human activity recognition is an important task in surveillance video analysis. An activity is a sequence of movements in continuous video frames, therefore, it requires extraction of spatiotemporal features. In our system, the activity needs to be recognized on master device after getting summary from the slave device. For activity recognition, we used a pre-trained VGG-19 CNN model for spatial feature representations followed by auto-encoder which captures the temporal features in the sequence. In contrast to VGG-19, the well-known AlexNet and GoogleNet are not effective to capture tiny patterns in visual data because these models have 1111 and 77 filter size with 4 and 2-pixel stride, respectively. We also made experiments on MobileNetV2 , where we extracted features from final convolution layer, but the were not convincing compared to state-of-the-art (see Table 2). VGG-16 model is recently studied for activity recognition problem in and we outperformed their method over various datasets by investigating VGG-19. It can extract discriminative visual features because it uses 33 filter in all convolutional layers with fixed 1-pixel stride. This helps in processing the smallest receptive field to grab the notions of tiny patterns. It has sixteen convolutional and three fully connected layers where we extract deep features from the final fully connected layer (FC8) having 11000 dimension. FC7 layer of VGG-19 outputs 4096 dimension representations which a large sized feature vector for a single sequence and yield huge processing for encoding of temporal representations. Thus, it cannot be considered for activity recognition over resource constrained devices in real-time scenarios. We are processing 15 fps, therefore, we extract 15000 spatial frame-level features from one second summarized frames. The features extracted in a single interval of time are compared to compute inter-view correlations. The features in the same interval from different views, are considered as overlapped and hence a single feature vector is selected from them for further processing. Our feature extraction is performed on Movidius VPU stick, therefore, the original VGG-19 model is compressed using neural computing stick software development kit (NCSDK). This optimized the original model of size 574.7 MB to 287.3 MB to Movidiuss compatible graph format. The temporal changes and sequential patterns in 15000 features are learned using our proposed auto-encoder. Motivated from the wide usage of auto-encoder network for many applications, we utilized it in our framework. Auto-encoder reduces the dimensionality by analyzing patterns in high dimensional data, therefore, we claim that it can learn the spatio-temporal patterns in deep features of 15 consecutive frames. We performed several experiments for squeezing 15000 features to low dimensions to effectively learn the temporal changes. We first encoded 15000 features to 8000, followed by its encoding to 1000 features (first setting). This kind of setting is very effective because its mean square error (MSE) for encoded features is very low, providing precise accuracy. However, its time complexity is not meeting the needs of embedded devices because the total squeezing has one extra step of encoding to 8000 features. In another setting, we encoded the 15000 high dimensional features directly to 1000 (second setting). In this setting, the accuracy is slightly compromised when using linear SVM, however, we met the computational requirements of embedded devices. We trained our auto-encoders for 40 epochs in which we utilized sparsity regularization with proportion value of 0.1 and L2 weights regularization to reduce the chance of over-fitting and avoid being stuck in local minima. To this end, MSE with support of sparsity regularization and L2 regularization is trained as a cost function of our auto-encoders. In first setting, the MSE decreased to 0.012 after 40 epochs, while in the second setting, it reduced to 0.044. The encoded features are spatiotemporal representation of an activity, which are passed to SVM classifier for activity recognition. We trained one versus all multi-class SVM because it trains N-1 (N is the number of total classes) classifiers, which is efficient for resource restricted devices. All SVMs are examined for activity recognition problem including linear, quadratic, and cubic SVM. We designed this framework considering the major limitations of WSNs and IoT devices such as computational complexity and storage capacities. We also aimed to provide an independent activity analysis system, which is malleable in both certain and uncertain environments. For this purpose, we first investigated different MVS and activity recognition methods, and studied the capabilities of WSNs. Concluding our literature study, the need of an independent system is realized, which could capture multi-view video data, analyze it, and provide a compact summary with recognized activities in lengthy surveillance videos. To this end, we made extensive experiments with different configurations and parameters, which can be interchangeably used according to users requirements and available resources. For instance, a slave node with extensively limited resources in WSN can encode its summary with PNG compression before its transmission to master node for activity recognition. With this mechanism, a real-time summary generation and transmission to master node is ensured despite the slower connectivity and limited communication bandwidth. Similarly, at master node, different options can be considered to balance execution time and accuracy of activity recognition. Various classifiers were used for experiments with different computational complexities. In addition, different experiments were performed for direct feature extraction from a sample summary as well as encoded features with different configurations for activity recognition. The complete evaluation details of these configurations are discussed in Section 4. We divide the experimental evaluation of our proposed framework into three subsections: MVS, activity recognition, and statistical analysis of data transmission. First, we explain the performance evaluation of MVS employed on resource constrained devices and compared with state-of-the-art techniques. Next, we evaluated the activity recognition module that is carried out on master device with Movidius setup. In the MVS module, we used Raspberry Pi (RPi) for our experiments, but our work is not limited to only RPi and can be applied to any other device with embedded vision. For activity recognition, we used MATLAB as a simulation tool and then transformed it to Keras deep learning framework in Python. The model trained using Keras with Tensorflow in backend can be converted into Movidius compatible graph format using NCSDK. Currently, we tested our model on RPi that can be adjustably used over other Movidius-supported devices. Sample of integrated framework for an input video from UCF-50 dataset with different views are visualized in Figure 4. Figure 4: Sample generated on UCF-50 video captured from two different views. Keyframes from both input videos are generated and then VGG-19 features are extracted. These features are compared and the selected single feature vector is encoded into 11000. Final output of activity is generated by passing this feature vector to trained model that gives probability score along with the predicted class. We used five different datasets for experiments where three of them are utilized for object detection in foggy scenarios and MVS and rest of the them are used to evaluate our activity recognition method. Office is widely used for performance evaluation of MVS methods because of its public availability along with the ground truth. It is created by utilizing four stably-held cameras fitted in an office environment. The cameras have motion with no synchronization between the captured videos with light variations. Bl-7f (a) is the most challenging dataset in MVS literature that has 19 different cameras installed at 7th floor of Taiwan University. The recorded videos contain different events closely related to each other with maximum overlapping. The installed cameras are synchronized, still, and fixed but with different light conditions in different views. The ground truth as well as the dataset is publicly available for research purposes. Road multi-view dataset is captured using handheld cameras with extreme level of shuddering effects and variable light conditions. Different from the above two datasets, it contains persons and vehicles, recorded in an outdoor day light. This dataset is used for foggy outdoor videos to test our object detection model in uncertain environment. Results of object detection over Road dataset are given in Figure 3 in uncertain environment. The activity recognition part of our framework is evaluated using UCF50 and YouTube 11 action datasets. UCF50 contains assorted human action videos with high-level shuddering due to camera motion, difference in viewpoint, and scalability of objects. The total number of classes are 50 with some actions closely related to each other, effecting the overall accuracy of classifiers. YouTube dataset, in contrast to UCF50 is less challenging and includes motion of camera, cluttered , light variation, and changes in viewpoint. A total of 11 different action videos are available in this dataset. 1.00 0. Table 1 shows the MVS of our implemented framework compared to other state-of-the-art techniques. We provide objective evaluation for two benchmark datasets in MVS literature i.e., Office and Bl-7f. The time complexity of our framework is given in the last column, indicating the execution time for a RPi device. The better performance of our method compared to existing methods can be seen from Table 1. On Office dataset, our method achieved the highest F-measure score compared to all methods under consideration. The most recent methods for MVS on Office dataset scores 0.90 and 0.91 with higher computational complexity. The method presented in uses heavy-weight CNN model for spatio-temporal point processing to select video skims. A gap in computational complexity from a recent MVS method over resource constrained devices is indicated in Table I where our method generates MVS 0.78 secs faster than (a). In case of Bl-7f dataset, we outperformed (a) and have similar F1-score with , but our framework is functional over resource constrained devices. The activity recognition module of our system is evaluated using two benchmark action datasets including UCF-50 and YouTube 11. The comparison using an overall accuracy metric with state-ofthe-art techniques is given in Table 2. We conducted different experiments for activity recognition using original VGG-19 features, encoded features with S1, and encoded features with S2 along with variants of SVM including linear, quadratic, and cubic SVM. S1 in Table 2 represents the first setting, where the sequential features are encoded into 8000, followed by 1000 from 15000 features of a single sequence. S2 presents the second setting of directly encoding 15000 features to a 1000 feature vector. These different settings can be alternatively used for different scenarios, considering both accuracy and complexity. On UCF50 dataset, trajectories analysis , hierarchical clustering , ML-LSTM , and deep auto-encoder with CNN achieved 92.3%, 93.2%, 94.9%, and 96.4% accuracy, repetitively. On the other hand, our S1 and S2 with cubic SVM achieved 96.2% and 93.8% overall accuracy, respectively. On YouTube 11 action dataset, the single stream CNN , hierarchical clustering , DB-LSTM , ML-LSTM , and.6% on quadratic and cubic SVM, respectively. From the above comparisons, it can be observed that the accuracy of our different settings are either higher or similar to the state-of-the-art. However, in terms of computational complexity, the existing methods are designed for high processing GPUs but our system is functional over resource constrained devices. The performance of our settings with linear SVM is under 80% but with quadric SVM, it touches the milestone of 90%. Therefore, it can be considered as state-of-the-art for processing over embedded devices with low cost VPU stick. Finally, the experiments performed over MobileNetV2 give lower accuracy on both the datasets. Thus, light-weight MobileNetV2 is not a suitable option for consideration in real-time surveillance for activity recognition. Wireless networks have constrained resources of bandwidth and communication cost. It is therefore not a feasible solution to transmit huge-sized surveillance videos to cloud or local servers due to limited transmission and storage of data. Also, searching for an event in long videos manually is a tiresome task. This section provides statistical details about saving storage capacity, bandwidth, and transmission time as illustrated in Figure 5. To the best of our knowledge, there is no specific dataset available to confirm the efficiency and effectiveness of our framework in terms of bandwidth, transmission time, and storage capacity. Thus, we considered an example video to generate keyframes, encode and transmit them to the master device for recognizing activities. We then compared all the mentioned parameters as given in Figure 5. The frames in the example video are considered as non-compressed with ideal situation of transmission. First, we calculate the total number of pixels by multiplying height (h) of the frame with width (w), no of pixels = hw. In office video, h = 480 and w = 640, ing in 307,200, which is multiplied by the depth of the frame i.e., 8. Finally, we divided the obtained number by 8 to get the size of the frame in bytes and is further converted to MB. The size of a single frame from this video is 0.29 MB. Similarly, to calculate the size of the office video-0, multiply the size of each frame by the number of frames inside it. This video has 26,940 frames, and multiplying it with size of each frame, we get 7892.57. It can be observed that there is a huge difference of storage, indicating the effectiveness of our system in terms of saving storage capacity. Saving storage directly refers to preservation of communication bandwidth because the bandwidth consumed by data is the same as their size. The transmission time in our framework is saved in two ways: 1) in low-bandwidth networks: encode the keyframes with PNG compression, and 2) transmit only important frames over IoT network saving huge time. Suppose an ideal situation in the network to transmit a frame in the WSN from slave to master device. Assume the distance as 0.3 km and speed as 200,000 km/s with data rate of 32 Mbps. To receive the data from slave (source), the data will take 3 types of times: 1) getting frame on WSN, 2) transmitting the data through the network, 3) loading data from router/hub to the master (destination) device. The time denoted as t1 in Eq. 2 is transmission time from source to network router. Transmission time t 2 in Eq. 3 is considered as the time taken by network to transmit data from source router to destination. The transmission time from destination router to the master device is the same as t1. The overall transmission time is the sum of t 1, t 2, and t 3, where t 1 and t 3 are same. The calculated transmission time for overall frames, keyframes, and encoded keyframes is 4137.98, 4.45, 1.13 seconds, respectively, as given in Figure 5. t 2 = distance f rom slave to master device transmission speed Figure 5: Transmission time and storage size comparison of transmitting and storing overall video (Office-0 video) versus keyframes and encoded keyframes. A huge difference of saving transmission time and storage capacity can be observed among all these possible settings. In this paper, we integrated MVS and activity recognition under an umbrella of a unified framework. A complete setup including slaves and a master resource constrained device working independently in an IoT is presented. The hardware requirements include slave devices equipped with camera and wireless sensors, a master device with Intel Movidius NCS for running optimized deep learning models on the edge. The slave devices capture multi-view video data, detect objects, extract features, compute mutual information, and finally generate summary. The generated summary is received at master device with optimized trained model for activity recognition. The MVS algorithm as well activity recognition models presented in this paper outperform state-of-the-art. In future, we have intention to extend this work by deeply investigating multi-view action recognition algorithms with different parameters and configurations in resource constrained environments. Further, we want to explore spiking neural networks used for various tasks in our framework for spatio-temporal features extraction advanced to activity recognition.>>

We compare robustness of models from 4 popular NLP tasks: Q&A, NLI, NER and Sentiment Analysis by testing their performance on perturbed inputs.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: Represent the article for finding a single-sentence TLDR

Question: Training generative models like Generative Adversarial Network (GAN) is challenging for noisy data. A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper. The curriculum construction is based on the centrality of underlying clusters in data points. The data points of high centrality takes priority of being fed into generative models during training. To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality. Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models. The experiments on cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect to specified quality metrics for noisy data. An interesting finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper. Deep generative models have piqued researchers' interest in the past decade. The fruitful progress has been achieved on this topic, such as auto-encoder and variational auto-encoder (VAE) , generative adversarial network (GAN) (;, normalizing flow (; ;), and autoregressive models (van den b; a; . However, it is non-trivial to train a deep generative model that can converge to a proper minimum of the associated optimization. For example, GAN suffers non-stability, mode collapse, and generative distortion during training. Many insightful algorithms have been proposed to circumvent those issues, including feature engineering , various discrimination metrics , distinctive gradient penalties , spectral normalization to discriminator , and orthogonal regularization to generator . What is particularly of interest is that the breakthrough for GANs has been made with a simple technique of progressively growing neural networks of generators and discriminators from low-resolution images to high-resolution counterparts (a). This kind of progressive growing also helps push the state of the arts to a new level by enabling StyleGAN to produce photo-realistic and detail-sharp (b), shedding new light on wide applications of GANs in solving real problems. This idea of progressive learning is actually a general manner of cognition process , which has been formally named curriculum learning in machine learning . The central topic of this paper is to explore a new curriculum for training deep generative models. To facilitate robust training of deep generative models with noisy data, we propose curriculum learning with clustering. The key contributions are listed as follows: • We first summarize four representative curricula for generative models, i.e. architecture (generation capacity), semantics (data content), dimension (data space), and cluster (data structure). Among these curricula, cluster curriculum is newly proposed in this paper. • Cluster curriculum is to treat data according to the centrality of each data point, which is pictorially illustrated and explained in detail. To foster large-scale learning, we devise the active set algorithm that only needs an active data subset of small fixed size for training. • The geometric principle is formulated to analyze hardness of noisy data and advantage of cluster curriculum. The geometry pertains to counting a small sphere packed in an ellipsoid, on which is based the percolation theory we use. The research on curriculum learning is diverse. Our work focuses on curricula that are closely related to data attributes, beyond which is not the scope we concern in this paper. Curriculum learning has been a basic learning approach to promoting performance of algorithms in machine learning. We quote the original words from the seminal paper as its definition: Curriculum learning. " The basic idea is to start small, learn easier aspects of the task or easier sub-tasks, and then gradually increase the difficulty level" according to pre-defined or self-learned curricula. From cognitive perspective, curriculum learning is common for human and animal learning when they interact with environments , which is the reason why it is natural as a learning rule for machine intelligence. The learning process of cognitive development is gradual and progressive . In practice, the design of curricula is task-dependent and data-dependent. Here we summarize the representative curricula that are developed for generative models. Architecture curriculum. The deep neural architecture itself can be viewed as a curriculum from the viewpoint of learning concepts or disentangling representations . For example, the different layers decompose distinctive features of objects for recognition (; ;) and generation . Besides, Progressive growing of neural architectures is successfully exploited in GANs (a; ; ; b). Semantics curriculum. The most intuitive content for each datum is the semantic information that the datum conveys. The hardness of semantics determines the difficulty of learning knowledge from data. Therefore, the semantics can be a common curriculum. For instance, the environment for a game in deep reinforcement learning and the number sense of learning cognitive concepts with neural networks can be such curricula. Dimension curriculum. The high dimension usually poses the difficulty of machine learning due to the curse of dimensionality , in the sense that the amount of data points for learning grows exponentially with dimension of variables . Therefore, the algorithms are expected to be beneficial from growing dimensions. The effectiveness of dimension curriculum is evident from recent progress on deep generative models, such as ProGANs (a; b) by gradually enlarging image resolution and language generation from short sequences to long sequences of more complexity . For fitting distributions, dense data points are generally easier to handle than sparse data or outliers. To train generative models robustly, therefore, it is plausible to raise cluster curriculum, meaning that generative algorithms first learn from data points close to cluster centers and then with more data progressively approaching cluster boundaries. Thus the stream of feeding data points to models for curriculum learning is the process of clustering data points according to cluster centrality that will be explained in section 3.2. The toy example in Figure 1 illustrates how to form cluster curriculum. The importance of clusters for data points is actually obvious from geometric point of view. The data sparsity in high-dimensional spaces causes the difficulty of fitting the underlying distribution of n = 100 n = 200 n = 300 n = 400 Figure 1: Cluster Curriculum. From magenta color to black color, the centrality of data points reduces. The value n is the number of data points taken with centrality order. data points . So generative algorithms may be beneficial when proceeding from the local spaces where data points are relatively dense. Such data points form clusters that are generally informative subsets with respect to the entire dataset. In addition, clusters contain common regular patterns of data points, where generative models are easier to converge. What is most important is that noisy data points deteriorate performance of algorithms. For classification, the effectiveness of curriculum learning is theoretically proven to circumvent the negative influence of noisy data . We will analyze this aspect for generative models with geometric facts. With cluster curriculum, we are allowed to gradually learn generative models from dense clusters to cluster boundaries and finally to all data points. In this way, generative algorithms are capable of avoiding the direct harm of noise or outliers. To this end, we first need a measure called centrality that is the terminology in graph-based clustering. It quantifies the compactness of a cluster in data points or a community in complex networks . A large centrality implies that the associated data point is close to one of cluster centers. For easy reference, we provide the algorithm of the centrality we use in Appendix. For experiments in this paper, all the cluster curricula are constructed by the centrality of stationary probability distribution, i.e. the eigenvector corresponding to the largest eigenvalue of the transition probability matrix drawn from the data. To be specific, let c ∈ R m denote the centrality vector of m data points. Namely, the i-th entry c i of c is the centrality of data point x i. Sorting c in descending order and adjusting the order of original data points accordingly give data points arranged by cluster centrality. Let.., − → X l } signify the set of centrality-sorted data points, where − → X 0 is the base set that contains sufficient data to attain convergent generative models, and the rest of − → X is evenly divided into l subsets according to centrality order. In general, the number of data points in − → X 0 is much less than m and determined according to X. Such division of − → X 0 serves to efficiency of training, because we do not need to train models from a very small dataset. The cluster curriculum learning is carried out by incrementally feeding subsets in − → X into generative algorithms. In other words, algorithms are successively trained on, meaning that the curriculum for each round of training is accumulated with − → X i. In order to determine the optimal curriculum we need the aid of quality metric of generative models, such as Fréchet inception distance (FID) or sliced Wasserstein distance (SWD) . For generative models trained with each curriculum, we calculate the associated score s i via the specified quality metric. The optimal curriculum for effective training can be identified by the minimal value for all s i, where i = 1,..., l + 1. The interesting phenomenon of this score curve will be illustrated in the experiment. The minimum of score s is apparently metric-dependent. One can refer to for the review of evaluation metrics. In practice, we can opt one of reliable metrics to use or multiple metrics for decision-making of the optimal model. There are two ways of using the incremental subset − → X i during training. One is that the parameters of models are re-randomized when the new data are used, the procedure of which is given in Algorithm 1 in Appendix. The other is that the parameters are fine-tuned based on pre-training of the previous model, which will be presented with a fast learning algorithm in the following section. To obtain the precise minimum of s, the cardinality of − → X i needs to be set much smaller than m, meaning that l will be large even for a dataset of moderate scale. The training of many loops will be time-consuming. Here we propose the active set to address the issue, in the sense that for each loop of cluster curriculum, the generative models are always trained with a subset of a small fixed size instead of − → X 0 ← − → X 0 ∪ − → X i whose size becomes incrementally large. (a) Figure 2: Schematic illustration of active set for cluster curriculum. The cardinality |A| of the active set A is 200. When − → X 2 is taken for training, we need to randomly sample another 100 (i.e. − → X 2|) data points from the history data Then the complete active set is composed by We can see that data points in become less dense after sampling. To form the active set A, the subset − → A 0 of data points are randomly sampled from − → X 0 to combine with − → X i for the next loop, where For easy understanding, we illustrate the active set with toy example in Figure 2. In this scenario, progressive pre-training must be applied, meaning that the update of model parameters for the current training is based on parameters of previous loop. The procedure of cluster curriculum with active set is detailed in Algorithm 2 in Appendix. The active set allows us to train generative models with a small dataset that is actively adapted, thereby significantly reducing the training time for large-scale data. Cluster curriculum bears the interesting relation to high-dimensional geometry, which can provide geometric understanding of our algorithm. Without loss of generality, we work on a cluster obeying the normal distribution. The characteristic of the cluster can be extended into other clusters of the same distribution. For easy analysis, let us begin with a toy example. As Figure 3(a) shows, the confidence ellipse E 2 fitted from the subset of centrality-ranked data points is nearly conformal to E 1 of all data points, which allows us to put the relation of these two ellipses by virtue of the confidence-level equation. Let N (0, Σ) signify the center and covariance matrix of the cluster C of interest, where C = {x i |x i ∈ R d, i = 1, . . ., n}. To make it formal, we can write the equation by The annulus formed by of removing the inner ellipse from the outer one. where χ To analyze the hardness of training generative models, a fundamental aspect is to examine the number n(E) of given data points falling in a geometric entity E 1 and the number N (E) of lattice points in it. The less n(E) is compared to N (E), the harder the problem will be. However, the enumeration of lattice points is computationally prohibitive for high dimensions. Inspired by the information theory of encoding data of normal distributions , we count the number of small spheres S ε of radius ε packed in the ellipsoid E instead. Thus we can use this number to replace the role of N (E) as long as the radius of the sphere S ε is set properly. With a little abuse of normal d=2 d=20 d=50 d=100 d=500 d=1000 lattice d=2 d=20 d=50 d=100 d=500 d=1000 Figure 4: Comparison between the number n(A) of data points sampled from the isotropic normal distributions and N (A) of spheres (lattice) packed in the annulus A with respect to the Chi quantile χ α2. d is the dimension of data points. For each dimension, we sample 70,000 data points from N (0, I). The scales of y-axis and x-axis are normalized by 10,000 and χ α1, respectively. notation, we still use N (E) to denote the packing number in the following context. Theorem 1 gives the exact form of N (E). Theorem 1. For a set C = {x i |x i ∈ R d} of n data points drawn from normal distribution N (0, Σ), the ellipsoid E α of confidence 1 − α is defined as x Σ −1 x ≤ χ 2 α, where Σ has no zero eigenvalues and α ∈. Let N (E α) be the number of spheres of radius ε packed in the ellipsoid E α. Then we can establish We can see that N (E α) admits a tidy form with Mahalanobis distance χ α, dimension d, and sphere radius ε as variables. The proof is provided in Appendix. The geometric region of interest for cluster curriculum is the annulus A formed by removing the ellipsoid 2 E α2 from the ellipsoid E α1, as Figure 3 (b) displays. We investigate the varying law between n(A) and N (A) in the annulus A when the inner ellipse E α2 grows with cluster curriculum. For this purpose, we need the following two corollaries that immediately follows from Theorem 1. Corollary 1. Let N (A) be the number of spheres of radius ε packed in the annulus A that is formed by removing the ellipsoid E α1 from the ellipsoid E α1, where α 1 ≤ α 2. Then we have It is obvious that N (A) goes infinite when d → ∞ under the conditions that χ α1 > χ α2 and ε is bounded. Besides, when E α2 (cluster) grows, N (A) reduces with exponent d if E α1 is fixed. In light of Corollary 1, we can now demonstrate the functional law between n(A) and N (A). First, we determine χ α1 as follows which means that E α1 is the ellipsoid of minimal Mahalanobis distance to the center that contains all the data points in the cluster. In addition, we need to estimate a suitable sphere radius ε, such that n(E α1) and N (E α1) have comparable scales in order to make n(A) and N (A) comparable in scale. To achieve this, we define an oracle ellipse E where n(E) = N (E). For simplicity, we let E α1 be the oracle ellipse. Thus we can determine ε with Corollary 3. Corollary 3. If we let E α1 be the oracle ellipse such that n(E α1) = N (E α1), then the free parameter ε can be computed with ε = χ α1 det(Σ)/n(E α1) To make the demonstration amenable to handle, data points we use for simulation are assumed to obey the isotropic normal distribution, meaning that data points are generated with nearly equal Figure 5: Examples of LSUN cat dataset and CelebA face dataset. The samples in the first row are of high centrality and the samples of low centrality in the second row are noisy data or outliers that we call in the context. variance along each dimension. Figure 4 shows that n(A) gradually exhibits the critical phenomena of percolation processes 3 when the dimension d goes large, implying that the data points in the annulus A are significantly reduced when E α2 grows a little bigger near the critical point. In contrast, the number N (A) of lattice points is still large and varies negligibly until E α2 approaches the boundary. This discrepancy indicates clearly that fitting data points in the annulus is pretty hard and guaranteeing the precision is nearly impossible when crossing the critical point of n(A) even for a moderate dimension (e.g. d = 500). Therefore, the plausibility of cluster curriculum can be drawn naturally from this geometric fact. The generative model that we use for experiments are Progressive growing of GAN (ProGAN) (a). This algorithm is chosen because ProGAN is the state-of-the-arts algorithm of GANs with official open sources available. According to convention, we opt the Fréchet inception distance (FID) for ProGAN as the quality metric. We randomly sample the 200,000 cat images from the LSUN dataset . These cat images are captured in the wild. So their styles vary significantly. Figure 5 shows the cat examples of high and low centralities. We can see that the noisy cat images differ much from the clean ones. There actually contain the images of very few informative cat features, which are the outliers we refer to. The curriculum parameters are set as | − → X 0 | = 20, 000 and | − → X i | = 10, 000, which means that the algorithms are trained with 20,000 images first and after the initial training, another 10,000 images according to centrality order are merged into the current training data for further re-training. For active set, its size is fixed to be 30, 000. The CelebA dataset is a large-scale face attribute dataset . We use the cropped and well-aligned faces with a bit of image s preserved for generation task. For clustercurriculum learning, we randomly sample 70,000 faces as the training set. The face examples of different centralities are shown in Figure 5. The curriculum parameters are set as | − → X 0 | = 10, 000 and | − → X i | = 5, 000. We bypass the experiment of the active set on faces because it is used for the large-scale data. Each image in two databases is resized to be 64 × 64. To form cluster curricula, we exploit ResNet34 pre-trained on ImageNet to extract 512-dimensional features for each face and cat images. The directed graphs are built with these feature vectors. We determine the parameter σ of edge weights by enforcing the geometric mean of weights to be 0.8. The robustness of varying the value was validated in for clustering. The number of nearest neighbors is set to be K = 4 log m. The centrality is the stationary probability distribution. All codes are written with TensorFlow.: FID curves of cluster-curriculum learning for ProGAN on the cat dataset and CelebA face dataset. The centrality and the FID share the x-axis due to that they have the same order of data points. The same colors of the y-axis labels and the curves denote the figurative correspondence. The network parameters for "normal training" are randomly re-initialized for each re-training. The active set is based on progressive pre-training of the fixed small dataset. The scale of the x-axis is normalized by 10,000. From Figure 6a, we can see that the FID curves are all nearly V-shaped, indicating that the global minima exist amid the training process. This is the clear evidence that the noisy data and outliers deteriorate the quality of generative models during training. From the optimal curricula found by two algorithms (i.e. curricula at 110,000 and 100,000), we can see that the curriculum of the active set differs from that of normal training only by one-step data increment, implying that the active set is reliable for fast cluster-curriculum learning. The performance of the active set measured by FID is much worse than that of normal training, especially when more noisy data are fed into generative models. However, this does not change the whole V-shape of the accuracy curve. Namely, it is applicable as long as the active set admits the metric minimum corresponding to the appropriate curriculum. The V-shape of the centrality-FID curve on the cat data is due to that the noisy data of low centrality contains little effective information to characterize the cats, as already displayed in Figure 5. However, it is different for the CelebA face dataset where the face images of low centrality also convey the part of face features. As evident by Figure 6b, ProGAN keeps being optimized by the majority of the data until the curriculum of size 55, 000. To highlight the meaning of this nearly negligible minimum, we also conduct the exactly same experiment on the FFHQ face dataset containing 70, 000 face images of high-quality (b). For FFHQ data, the noisy face data can be ignored. The gray curve of normal training in Figure 6b indicates that the FID of ProGAN is monotonically decreased for all curricula. This gentle difference of the FID curves at the ends between CelebA and FFHQ clearly demonstrates the difficulty of noisy data to generative algorithms. To understand cluster curriculum deeply, we employ the geometric method formulated in section 4 to analyze the cat and face data. The percolation processes are both conducted with 512-dimensional features from ResNet34. Figure 7 displays the curve of n(A) that is the variable of interest in this scenario. As expected, the critical point in the percolation process occurs for both cases, as shown by blue curves. An obvious fact is that the optimal curricula (red strips) both fall into the (feasible) domains of percolation processes after the critical points, as indicated by gray color. This is a desirable property because data become rather sparse in the annuli when crossing the critical points. Then noisy data play the non-negligible role on tuning the parameters of generative models. Therefore, a fast learning strategy can be derived from the percolation process. Training may begin from the curriculum specified by the critical point, thus significantly accelerating cluster-curriculum learning. The pink strips are intervals of optimal curricula derived by generative models. For example, the value 9 of the pink interval in (a) is obtained by 9 = 20 − 11, where 11 is one of the minima (i.e. 110,000) in Figure 6a. The others are derived in the same way. The subtraction transforms the data number in the cluster to be the one in the annulus. The critical points are determined by searching the maxima of the absolute discrete difference of the associated curves. The scales of y-axes are normalized by 10,000. Another intriguing phenomenon is that the more noisy the data, the closer the optimal interval (red strip) is to the critical point. We can see that the optimal interval of the cat data is much closer to the critical point than that of the face data. What surprises us here is that the optimal interval of cluster curricula associated with the cat data nearly coincides with the critical point of the percolation process in the annulus! This means that the optimal curriculum may be found at the intervals close to the critical point of of n(A) percolation for heavily noisy data, thus affording great convenience to learning an appropriate generative model for such datasets. Cluster curriculum is proposed for robust training of generative models. The active set of cluster curriculum is devised to facilitate scalable learning. The geometric principle behind cluster curriculum is analyzed in detail as well. The experimental on the LSUN cat dataset and CelebA face dataset demonstrate that the generative models trained with cluster curriculum is capable of learning the optimal parameters with respect to the specified quality metric such as Fréchet inception distance and sliced Wasserstein distance. Geometric analysis indicates that the optimal curricula obtained from generative models are closely related to the critical points of the associated percolation processes established in this paper. This intriguing geometric phenomenon is worth being explored deeply in terms of the theoretical connection between generative models and high-dimensional geometry. It is worth emphasizing that the meaning of model optimality refers to the global minimum of the centrality-FID curve. As we already noted, the optimality is metric-dependent. We are able to obtain the optimal model with cluster curriculum, which does not mean that the algorithm only serves to this purpose. We know that more informative data can help learn a more powerful model covering the large data diversity. Here a trade-off arises, i.e. the robustness against noise and the capacity of fitting more data. The centrality-FID curve provides a visual tool to monitor the state of model training, thus aiding us in understanding the learning process and selecting suitable models according to noisy degree of given data. For instance, we can pick the trained model close to the optimal curriculum for heavily noisy data or the one near the end of the centrality-FID curve for datasets of little noise. In fact, this may be the most common way of using cluster curriculum. In this paper, we do not investigate the cluster-curriculum learning for the multi-class case, e.g. the ImageNet dataset with BigGAN . The cluster-curriculum learning of multiple classes is more complex than that we have already analyzed on the face and cat data. We leave this study for future work. The centrality or clustering coefficient pertaining to a cluster in data points or a community in a complex network is a well-studied traditional topic in machine learning and complex systems. Here we introduce the graph-theoretic centrality for the utilization of cluster curriculum. Firstly, we construct a directed graph (digraph) with K nearest neighbors by the method in . The weighted adjacency matrix W of the digraph can be formed in this way: 2 ) if x j is one of the nearest neighbors of x i and 0 otherwise, where d ij is the distance between x i and x j and σ is a free parameter. The density of data points can be quantified with the stationary probability distribution of a Markov chain. For a digraph built from data, the transition probability matrix can be derived by row normalization, say, P ij = W ij / j W ij. Then the stationary probability u can be obtained by solving an eigenvalue problem P u = u, where denotes the matrix transpose. It is straightforward to know that u is the eigen-vector of P corresponding to the largest eigenvalue (i.e. 1). u is also defined as a kind of PageRank in many scenarios. For density-based cluster curriculum, the centrality c coincides with the stationary probability u. Figure 1 in the main context shows the plausibility of using the stationary probability distribution to quantify the data density. Then we derive the length of semi-axis with respect to χ α, i.e. For a d-dimensional ellipsoid E, the volume of E is where r i the leng of semi-axis of E and Γ(·) is the Gamma function. Substituting into the above equation, we obtain the final formula of volume Using the volume formula in, it is straightforward to get the volume of the packing sphere S ε By the definition of N (E α), we can write We conclude the proof of the theorem.

Represent the TLDR for finding the corresponding scientific article

A novel cluster-based algorithm of curriculum learning is proposed to solve the robust training of generative models.

Represent the short summary for finding the corresponding scientific article (long)

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.