Question: Represent the text to find a very short summary (1-2 sentences max):

Question: In this paper, we address the challenge of limited labeled data and class imbalance problem for machine learning-based rumor detection on social media. We present an offline data augmentation method based on semantic relatedness for rumor detection. To this end, unlabeled social media data is exploited to augment limited labeled data. A context-aware neural language model and a large credibility-focused Twitter corpus are employed to learn effective representations of rumor tweets for semantic relatedness measurement. A language model fine-tuned with the a large domain-specific corpus shows a dramatic improvement on training data augmentation for rumor detection over pretrained language models. We conduct experiments on six different real-world events based on five publicly available data sets and one augmented data set. Our experiments show that the proposed method allows us to generate a larger training data with reasonable quality via weak supervision. We present preliminary achieved using a state-of-the-art neural network model with augmented data for rumor detection. Research areas that have recently been received much attention in using Machine Learning (ML) and Natural Language Processing for automated rumor and fake news detection BID5 BID11 BID19 BID25 BID22 and fact-checking BID2 BID21 BID10. One major bottleneck of state-of-the-art (SoA) ML methods is that they require a vast amount of labeled data to be trained and manual labeling of rumors source on social media requires special skills and time-consuming BID26. Due to limited labeled training data, existing neural networks (NNs) for rumor detection usually have shallow architecture BID3 BID13. The scarcity of labeled data is a major challenge of studying rumors on social media BID0. Another problem is that publicly available data sets for rumor-related tasks such as PHEME data BID10 suffer from imbalanced class distributions. Existing methods for handling the class imbalance problem (e.g., oversampling and the use of synthetic data BID24) may cause over-fitting and poor generalization performance. A methodology for rumor data augmentation with the minimum of human supervision is necessary. Previous studies presented that rumors can evolve into many variants which share similar propagation patterns in their early stage BID14 BID3 BID1 BID4. Based on this hypothesis, we argue that enriching existing labeled data with unlabeled source tweets conveying the same or similar meanings is a promising attempt for rumor detection methods that rely on the structure of rumor propagation in social media. In this work, we propose a novel data augmentation method for automatic rumor detection based on semantic relatedness. We exploit a publicly available paraphrase identification corpus as well as context-sensitive embeddings of labeled references and unlabeled candidate source tweets. Pairwise similarity is used to guide the assignment of pseudolabels to unlabeled tweets. ELMo BID18, a SoA context-sensitive neural language model (NLM), is fine-tuned on a large credibility-focused social media corpus and used to encode tweets. Our show that data augmentation can contribute to rumor detection with deep learning with increased training data size and a reasonable level of quality. This has potential for further performance improvements using deeper NNs. We present data augmentation for three events and the performance of a SoA DNN model for rumor detection with augmented data in Section 5. Four publicly available data sets covering a wide range of real-world events on social media as well as a Twitter paraphrase corpus are used for this project. SemEval-2015 task 1 data BID23 This data is built for paraphrase identification and semantic similarity measurement. This data set is employed in our semantic relatedness method in order to fine-tune optimum relatedness thresholds through pairwise comparisons between the tweet embeddings of labeled reference and unlabeled candidates (see details in Section 4). PHEME data set BID8 The latest PHEME data is used as reference set for data augmentation covering 9 manually labeled rumor events. CrisisLexT26 BID16 This data comprises tweets associated with 26 hazardous events happened between 2012 and 2013. "2013 Boston bombings" data from this data set is used as reference set in this experiment. Twitter event data BID25 This data consists of over 147 million tweets associated with 30 real-world events unfolded between February 2012 and May 2016, among which six events are selected as a pool of candidates source tweets. This covers'Ferguson unrest','Sydney siege','Ottawa shooting','Charliehebdo attacks','Germanwings plane crash', and'Boston marathon bombings'. We refer to the first five events with reference set generated from PHEME data as'PHEME5'. For the'Boston bombings' event, we generate references from CrisisLexT26 and a fact-checking website'Snopes.com' 1 (refer to Section 3). CREDBANK BID15 This large corpus comprises more than 60M tweets grouped into 1049 events, each of which were manually annotated with credibility ratings. This is leveraged to fine-tune ELMo in order to provide better representations for rumor-related tasks (see Section 3). An overview of our data augmentation method is presented in Figure 1. We exploit a limited amount of labeled data as weak supervision (i.e., references). References are generated separately for PHEME5 and "Boston bombings" data from different data with varying annotations schemes (as mentioned in Section 2). Due to space constraints, we omit the detailed process of reference generation here. Candidate tweets refer to any tweets that report an event of interest. The leftmost box shows how semantic similarity is computed between a given pair of reference and candidate tweets. Firstly, the contextual embedding model (ELMo) is fine-tuned with a domain-specific corpus to learn representations of rumors. Given corpora that contain pairs of tweets, we apply language-based filtering and perform linguistic preprocessing. The preprocessing includes lowercasing, removing'rt @', URLs 2, and non-alphabetic characters, and tokenization. Tweets with at least 4 tokens are considered to reduce noise BID6. Then, we compute ELMo embeddings of tweets for subsequent semantic relatedness measurement. Cosine similarity between each embeding pair is used as a relatedness measure. We use SemEval-2015 task 1 data set as a benchmark for relatedness threshold fine-tuning (see Section 4). Having optimum thresholds, semantic similarity computation is performed for reference-candidate pairs. Rumor and non-rumor source tweets are selected from the candidate pool using the fine-tuned thresholds. In the final step, data collection is performed to retrieve social-temporal context data (typically retweets and replies) for the selected source tweets. Source tweets without contexts are filtered out. We download source tweets for six selected events in the Twitter event and CREDBANK using Twitter API 3. For the CREDBANK, we downloaded 77,954,446 tweets (i.e., 97.1% of the original data). After deduplication, the train corpus contains 6,157,180 tweets with 146,340,647 tokens and 2,235,075 vocabularies. Rumor-specific Embedding (ELMo) Previous research shows that fine-tuning NLMs with in-domain data allows them to learn more meaningful word representations and provides a performance gain BID7 BID18. To fine-tune a pretrained ELMo, we generate a data set using the CREDBANK. Sentences are shuffled and spit into training and hold-out sets (with a ratio of around 0.02%). We also generate a test set that consists of 6,162 tweets in total using the PHEME data. TAB1 shows the number of tweets, tokens and vocabularies in the CREDBANK after deduplication. Following the practice in , a linear combination of the states of each LSTM layer and the token embeddings is adopted to encode tweets. The training corpus is split into small batches with a maximum of 5000 tweets for each batch. Training time took more than 800 hours on a NVIDIA Kepler K40M GPU with less than 10 GiB GPU memory. Since the CREDBANK training set is still a relatively small for NLMs, we only fine-tune a pretrained ELMo with 1 epoch to avoid over-fitting. The shows a large improvement in perplexity on both hold-out and test sets (See TAB2). Semantic Relatedness Fine-Tuning TAB3 compares different models for word representation on the SemEval-2015 data. We show the based on the maximum F-score each model achieves. It shows the effectiveness of our CREDBANK fine-tuned ELMo over a pretrained ELMo ("Original (5.5B)") and other SoA word embedding models. To ensure higher quality, we argue that a higher precision is required. Therefore, relatedness thresholds are fine-tuned based on precision achieved by the best-performing model. We highlight some statistics in Tabel 4.Data Augmentation We follow our data augmentation procedure described in Section 3. After performing pairwise similarity computation, a relatedness threshold (0.8) is adopted to select rumor source tweets from a pool of candidates. We randomly sample 3*(# of rumor source tweets) non-rumor source tweets if the score of a candidate tweet is less than 0.3. Sampling more negative examples is an attempt to balance class distributions after source tweets without contexts are removed. This is based on a hypothesis that non-rumors are less likely to have reactions than rumors Table 4: Results of fine-tuning thresholds based on precision. as they generally draw less attention. In terms of computational performance, ELMo embedding computation and semantic relatedness measurement are performed with CPU. Tweet encoding takes around 10 per second and pairwise comparison takes around 869 pairs per second on average. Data Augmentation Before filtering out source tweets without replies, 1,238 rumors and 3,714 non-rumors are collected for "bostonbombings". After filtering, 165 rumors and 228 non-rumors remain. Although the augmented data size is very limited for "bostonbombings", experiments on "sydneysiege" and "ottawashooting" show encouraging . A total of 25,486 rumors and 76,106 non-rumors are additionally obtained for "sydneysiege", and 21,519 rumors and 62,590 non-rumors are additionally obtained for "ottawashooting". We make our augmented data publicly available 4. Rumor Detection We conduct rumor detection experiments using two different data sets: PHEME5, PHEME5 with the "bostonbombings" data ("PHEME5+Boston"). We employ BID10's method as a SoA baseline model for rumor detection with slight modifications. For the sake of simplicity, we modify the implementation of "MTL2 Veracity+Detection" for rumor detection only. We construct input by using a source tweet and the top (i.e., most recent) 24 replies in this task. We perform leave-one-out cross-validation (LOOCV) on the PHEME5 and augmented data sets. The overall experimental for rumor detection are presented in TAB4. TAB5 shows LOOCV . We observe that overall performance decreases with the augmented data (i.e., PHEME5+Boston). The "fergusonunrest" is the most difficult event for a rumor detection model as it has a unique class distribution distinguished from all other events BID10. It is worth noting that our data augmentation improves the performance of rumor detection on the "fergusonunrest". The completion of data augmentation for events other than "'bostonbombings" has potential to boost overall and per event performance of rumor detection. We present a methodology of data augmentation for rumor detection that exploits semantic relatedness between limited labeled data and unlabeled data. This study is part of further research that aims to use a massive amount of publicly available unlabeled Twitter data and the potential of DNNs in a wide range of tasks related to rumors on social media. Our current research has demonstrated the potential efficiency and effectiveness of semantically augmented data in combating the labeled data scarcity and class imbalance problems of publicly available rumor data sets. In future work, we plan to augment data for more events to build comprehensive data sets for rumor detection, and conduct experiments on rumor detection via deep learning. We will evaluate the effectiveness of augmented data in alleviating over-fitting and its usefulness in facilitating deeper NNs for rumor detection. Further experiments will be conducted to examine the generalization of rumor detection models on unseen rumors.

represent the tldr for finding the corresponding scientific article:

We propose a methodology of augmenting publicly available data for rumor studies based on samantic relatedness between limited labeled and unlabeled data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce a lightweight architecture for named entity recognition and carry out incremental active learning, which is able to match state-of-the-art performance with just 25% of the original training data.


Question: Represent the scientific article to find a short TLDR

Question: Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention . Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU. There has been much recent progress in sequence modeling through recurrent neural networks (RNN; BID54, convolutional networks (CNN; BID28 BID14 BID7) and self-attention models BID40 BID58. RNNs integrate context information by updating a hidden state at every time-step, CNNs summarize a fixed size context through multiple layers, while as self-attention directly summarizes all context. Attention assigns context elements attention weights which define a weighted sum over context representations BID52 BID8 BID36. Source-target attention summarizes information from another sequence such as in machine translation while as self-attention operates over the current sequence. Self-attention has been formulated as content-based where attention weights are computed by comparing the current time-step to all elements in the context FIG0 ). The ability to compute comparisons over such unrestricted context sizes are seen as a key characteristic of self-attention BID58. However, the ability of self-attention to model long-range dependencies has recently come into question BID57 and the unlimited context size is computationally very challenging due to the quadratic complexity in the input length. Furthermore, in practice long sequences require the introduction of hierarchies.In this paper, we introduce lightweight convolutions which are depth-wise separable BID51 BID7, softmax-normalized and share weights over the channel dimension. The is a convolution with several orders of magnitude fewer weights than a standard nonseparable convolution. Different to self-attention, lightweight convolutions reuse the same weights for context elements, regardless of the current time-step. Dynamic convolutions build on lightweight convolutions by predicting a different convolution kernel at every time-step. The kernel is a function of the current time-step only as opposed to the entire context as in self-attention (FIG0 . Dynamic convolutions are similar to locally connected layers in the sense that the weights change at every position, however, the difference is that weights are dynamically generated by the model rather than fixed after training BID30 BID56 BID6 . Our approach also bears similarity to location-based attention which does not access the context to determine attention weights, however, we do not directly take the attention weights from the previous time-step into account BID8 BID36 . BID49 reduce complexity by performing attention within blocks of the input sequence and BID48 BID50 perform more fine-grained attention over each feature. BID47 and BID17 use input-dependent filters for text classification tasks. Our experiments show that lightweight convolutions perform competitively to strong self-attention and that dynamic convolutions can perform even better. On WMT English-German translation dynamic convolutions achieve a new state of the art of 29.7 BLEU, on WMT English-French they match the best reported in the literature, and on IWSLT German-English dynamic convolutions outperform self-attention by 0.8 BLEU. Dynamic convolutions achieve 20% faster runtime than a highly-optimized self-attention baseline. For language modeling on the Billion word benchmark dynamic convolutions perform as well as or better than self-attention and on CNN-DailyMail abstractive document summarization we outperform a strong self-attention model. We first outline sequence to sequence learning and self-attention. Our work builds on non-separable convolutions as well as depthwise separable convolutions. Sequence to sequence learning maps a source sequence to a target sequence via two separate networks such as in machine translation BID54 . The encoder network computes representations for the source sequence such as an English sentence and the decoder network autoregressively generates a target sequence based on the encoder output. The self-attention module of BID58 applies three projections to the input X ∈ R n×d to obtain key (K), query (Q), and value (V) representations, where n is the number of time steps, d the input/output dimension (FIG1 . It also defines a number of heads H where each head can learn separate attention weights over d k features and attend to different positions. The module computes dot-products between key/query pairs, scales to stabilize training, and then softmax normalizes the . Finally, it computes a weighted sum using the output of the value projection (V): DISPLAYFORM0 Depthwise convolutions perform a convolution independently over every channel. The number of parameters can be reduced from d 2 k to dk where k is the kernel width. The output O ∈ R n×d of a depthwise convolution with weight W ∈ R d×k for element i and output dimension c is defined as: DISPLAYFORM1 In this section, we introduce LightConv, a depthwise convolution which shares certain output channels and whose weights are normalized across the temporal dimension using a softmax. Compared to self-attention, LightConv has a fixed context window and it determines the importance of context elements with a set of weights that do not change over time steps. We will show that models equipped with lightweight convolutions show better generalization compared to regular convolutions and that they can be competitive to state-of-the-art self-attention models (§6). This is surprising because the common belief is that content-based self-attention mechanisms are necessary to obtaining stateof-the-art in natural language processing applications. Furthermore, the low computational profile of LightConv enables us to formulate efficient dynamic convolutions (§4).LightConv computes the following for the i-th element in the sequence and output channel c: DISPLAYFORM0 Weight sharing. We tie the parameters of every subsequent number of d H channels, which reduces the number of parameters by a factor of d H. As illustration, a regular convolution requires 7,340,032 (d 2 × k) weights for d = 1024 and k = 7, a depthwise separable convolution has 7,168 weights (d × k), and with weight sharing, H = 16, we have only 112 (H × k) weights. We will see that this vast reduction in the number of parameters is crucial to make dynamic convolutions possible on current hardware. BID60 ties the weights of all channels (H = 1). H×k across the temporal dimension k using a softmax operation: DISPLAYFORM0 Module. FIG1 shows the architecture of the module where we integrate LightConv. We first apply an input projection mapping from dimension d to 2d, followed by a gated linear unit (GLU;, and the actual lightweight convolution. The GLU uses half of the inputs as gates by applying sigmoid units and then computes a pointwise product with the other inputs. We also apply an output projection of size W O ∈ R d×d to the output of LightConv. We found DropConnect to be a good regularizer for the LightConv module BID59 . Specifically, we drop every entry of the normalized weights sof tmax(W) with probability p and divide it by 1 − p during training. This amounts to removing some of the temporal information within a channel. Implementation. Existing CUDA primitives for convolutions did not perform very well to implement LightConv and we found the following solution faster on short sequences: We copy and expand the normalized weights W ∈ R H×k to a band matrix of size BH × n × n, where B is the batch size. We then reshape and transpose the inputs to size BH × n × d H, and perform a batch matrix multiplication to get the outputs. We expect a dedicated CUDA kernel to be much more efficient. A dynamic convolution has kernels that vary over time as a learned function of the individual time steps. A dynamic version of standard convolutions would be impractical for current GPUs due to their large memory requirements. We address this problem by building on LightConv which drastically reduces the number of parameters (§3).DynamicConv takes the same form as LightConv but uses a time-step dependent kernel that is computed using a function f: DISPLAYFORM0 we model f with a simple linear module with learned weights DISPLAYFORM1. Similar to self-attention, DynamicConv changes the weights assigned to context elements over time. However, the weights of DynamicConv do not depend on the entire context, they are a function of the current time-step only. Self-attention requires a quadratic number of operations in the sentence length to compute attention weights, while the computation of dynamic kernels for DynamicConv scales linearly in the sequence length. Our experiments (§6) show that models using DynamicConv match or exceed the performance of state-of-the-art models that use context-based self-attention. This challenges the typical intuitions about the importance of content-based self-attention in natural language processing applications. We use an encoder-decoder architecture for sequence to sequence learning BID54 and we closely follow the architectural choices presented in BID58. Our self-attention baseline is the fairseq re-implementation of the Transformer Big architecture. The encoder and decoder networks have N blocks each. Encoder blocks contain two sub-blocks: The first is a self-attention module (§2), a LightConv module, or a DynamicConv module (§4). The second sub-block is a feed-forward module: DISPLAYFORM0 unless otherwise stated. Sub-blocks are surrounded by residual connections BID22 and layer normalization BID1.Decoder blocks are identical except that they have an additional source-target attention sub-block between the self-attention and feed-forward module. The source-target attention is equivalent to the self-attention module, except that the values and keys are projections over the encoder output for each source word. Words are fed to the encoder and decoder networks in d dimensional embeddings. We add sinusoidal position embeddings to encode the absolute position of each word in the sequence BID58. The model computes a distribution over vocabulary V by transforming the decoder output via a linear layer with weights W V ∈ R d×V followed by softmax normalization. LightConv and DynamicConv are identical to Transformer Big, except that self-attention modules are swapped with either fixed or dynamic convolutions. These models also use fewer parameters per block (cf. FIG1 and FIG1) and we therefore increase the number of blocks to N = 7 for the encoder to roughly match the parameter count of Transformer Big. We generally set H = 16. Both LightConv and DynamicConv set the the encoder and decoder kernel sizes to 3, 7, 15, 31x4 for each block respectively; except for the decoder where we have only three top layers with kernel size 31. To get a thorough understanding of the limitations of LightConv and DynamicConv we evaluate on three different tasks: machine translation, language modeling and abstractive summarization. Machine Translation. We report on four benchmarks: For WMT English to German (EnDe) we replicate the setup of BID58, based on WMT'16 training data with 4.5M sentence pairs, we validate on newstest2013 and test on newstest2014. 3 The vocabulary is a 32K joint source and target byte pair encoding (BPE; BID44 . For WMT English to French (EnFr), we borrow the setup of BID15 with 36M training sentence pairs from WMT'14, validate on newstest2012+2013 and test on newstest2014. The 40K vocabulary is based on a joint source and target BPE factorization. For WMT English to Chinese (Zh-En), we pre-process the WMT'17 training data following BID21 ing in 20M sentence pairs. We develop on devtest2017 and test on newstest2017. For IWSLT'14 German-English (De-En) we replicate the setup of for 160K training sentence pairs and 10K joint BPE vocabulary. For this benchmark only, data is lowercased. For WMT En-De, WMT En-Fr, we measure case-sensitive tokenized BLEU. 4 For WMT En-De only we apply compound splitting similar to BID58. For WMT Zh-En we measure detokenized BLEU to be comparable to BID21. 5 We train three random initializations of a each configuration and report test accuracy of the seed which ed in the highest validation BLEU. Ablations are conducted on the validation set and we report the mean BLEU and standard deviation on this set. WMT En-De, WMT En-Fr are based on beam search with a beam width of 5, IWSLT uses beam 4, and WMT Zh-En beam 8 following BID21. For all datasets, we tune a length penalty as well as the number of checkpoints to average on the validation set. Language Modeling. We evaluate on the large-scale Billion word dataset BID4 which contains 768M tokens and has a vocabulary of nearly 800K types. Sentences in this dataset are shuffled and we batch sentences independently of each other. Models are evaluated in terms of perplexity on the valid and test portions. Summarization. We test the model's ability to process long documents on the CNN-DailyMail summarization task BID23 BID37 comprising over 280K news articles paired with multi-sentence summaries. Articles are truncated to 400 tokens BID43 and we use a BPE vocabulary of 30K types. We evaluate in terms of F1-Rouge, that is Rouge-1, Rouge-2 and Rouge-L BID33. 6 When generating summaries, we follow standard practice in tuning the maximum output length, disallowing repeating the same trigram, and we apply a stepwise length penalty BID40. Translation. We use a dropout rate of 0.3 for WMT En-De and IWSLT De-En, 0.1 for WMT EnFr, and 0.25 for WMT Zh-En. WMT models are optimized with Adam and a cosine learning rate schedule BID29 BID35 where the learning rate is first linearly warmed up for 10K steps from 10 −7 to 10 −3 and then annealed following a cosine rate with a single cycle. For IWSLT'14 De-En, we use a schedule based on the inverse square root of the current step BID58. We train the WMT models on 8 NVIDIA V100 GPUs for a total of 30K steps on WMT En-De, 40K steps for WMT Zh-En and 80K steps for WMT En-Fr. For IWSLT De-En we train for 50K steps on a single GPU.We use floating point 16 precision and accumulate the gradients for 16 batches before applying an update, except for IWSLT where we do not accumulate gradients. Batches contain up to 459K source tokens and the same number of target tokens for both WMT En-De and WMT Zh-En, 655K for En-Fr, and 4K for IWSLT De-En. We use label smoothing with 0.1 weight for the uniform prior distribution over the vocabulary BID55 BID41.Language Modeling. We follow the same setup as for translation but remove the encoder module. For the Billion word benchmark we use an adaptive softmax output layer to reduce the computational burden of the large vocabulary BID18 BID42 We train on 32 GPUs with batches of 65K tokens for 975K updates. As optimizer we use Nesterov's accelerated gradient method BID53 ) with a momentum value of 0.99 and we renormalize gradients if their norm exceeds 0.1 BID39. The learning rate is linearly warmed up from 10 −7 to 1 for 16K steps and then annealed using a cosine learning rate schedule BID35 with one cycle. Summarization. We train with Adam using the cosine learning rate schedule with a warmup of 10K steps and a period of 20K updates. We use weight decay 1e-3 and dropout 0.3. We first report on WMT En-De and WMT En-Fr where we compare to the best in the literature, most of which are based on self-attention. Table 1 shows that LightConv performs very competitively and only trails the state of the art by 0.1 BLEU on WMT En-Fr; the state of the art is based on self-attention. This is despite the simplicity of LightConv which operates with a very small number of fixed weights over all time steps whereas self-attention computes dot-products with all context elements at every time-step. DynamicConv outperforms the best known on WMT En-De by 0.4 BLEU and achieves a new state of the art, whereas on WMT En-Fr it matches the state of the art. This shows that content-based self-attention is not necessary to achieve good accuracy on large translation benchmarks. IWSLT is a much smaller benchmark and we therefore switch to a smaller architecture: d f f = 1024, d = 512, and H = 4. The self-attention baseline on this dataset is the best reported in the literature TAB1. Table 3: Ablation on WMT English-German newstest2013. (+) indicates that a includes all preceding features. Speed based on beam size 4, batch size 256 on an NVIDIA P100 GPU. In this section we evaluate the impact of the various choices we made for LightConv (§3) and DynamicConv (§4). We first show that limiting the maximum context size of self-attention has no impact on validation accuracy (Table 3). Note that our baseline is stronger than the original of BID58. Next, we replace self-attention blocks with non-separable convolutions (CNN) with kernel size 3 and input/output dimension d = 1024. The CNN block has no input and output projections compared to the baseline and we add one more encoder layer to assimilate the parameter count. This CNN with a narrow kernel trails self-attention by 1 BLEU.We improve this by switching to a depthwise separable convolution (CNN Depthwise) with input and output projections of size d = 1024. When we progressively increase the kernel width from lower to higher layers then this further improves accuracy. This narrows the gap to self-attention to only 0.5 BLEU. DropConnect gives a slight performance improvement and weight sharing does not decrease performance. Adding softmax normalization to the weights is only 0.3 BLEU below the accuracy of the baseline. This corresponds to LightConv. In Appendix A we compare softmaxnormalization to various alternatives. Finally, dynamic convolutions (DynamicConv) achieve the same validation accuracy as self-attention with slightly fewer parameters and at 20% higher inference speed. Softmax-normalization is important for DynamicConv since training diverged in our experiments when removing it. To make the models more comparable, we do not introduce GLU after the input projection. For comparison, we re-implemented averaged attention networks (AAN;) which compute a uniform average over past model states instead of a weighted average as in self-attention. Our re-implementation is efficient: we measure 129 sentences/sec for a base transformer-AAN on newstest2014 compared to 20 sentences/sec for. Table 3 shows that our models outperform this approach. Note that AANs still use self-attention in the encoder network while as our approach does away with self-attention both in the encoder and decoder. As second task we consider language modeling on the Billion word benchmark. The self-attention baseline has N = 16 blocks, each with a self-attention module and a feed-forward module using d f f = 4096 and d = 1024. DynamicConv uses N = 17 blocks to assimilate the parameter count and we use kernel sizes 15x2, 31x4 and 63x11. TAB3 shows that DynamicConv achieves slightly better perplexity than our self-attention baseline which is very competitive. BID58. TAB4 shows that LightConv outperforms the self-attention baseline as well as comparable previous work and DynamicConv performs even better. We also show for a reinforcement learning approach BID3 and note that RL is equally applicable to our architecture. We presented lightweight convolutions which perform competitively to the best reported in the literature despite their simplicity. They have a very small parameter footprint and the kernel does not change over time-steps. This demonstrates that self-attention is not critical to achieve good accuracy on the language tasks we considered. Dynamic convolutions build on lightweight convolutions by predicting a different kernel at every time-step, similar to the attention weights computed by self-attention. The dynamic weights are a function of the current time-step only rather than the entire context. Our experiments show that lightweight convolutions can outperform a strong self-attention baseline on WMT'17 Chinese-English translation, IWSLT'14 German-English translation and CNNDailyMail summarization. Dynamic convolutions improve further and achieve a new state of the art on the test set of WMT'14 English-German. Both lightweight convolution and dynamic convolution are 20% faster at runtime than self-attention. On Billion word language modeling we achieve comparable to self-attention. We are excited about the future of dynamic convolutions and plan to apply them to other tasks such as question answering and computer vision where inputs are even larger than the tasks we considered in this paper. We compare our proposed softmax-normalization of weights to other alternatives in Table 6. For each setting, we use three seeds and report the mean and the standard deviation of the BLEU score on WMT English-German newstest2013. The softmax and norms are computed over the kernel dimension. Simply using the absolute value of the weights or squaring them does not make the training more stable, which shows that having all non-negative weights is not critical. Dividing the weights by the 2 -norm or bounding the weights with sigmoid or the hyperbolic tangent function also stablizes the training procedure; however, the softmax-normalization performs best. 26.7 ± 0.2 Table 6: Alternatives to softmax-normalization in DynamicConv on WMT English-German newstest2013 (= 10 −6). In this section we compare DynamicConv to current non-autoregressive models in the literature. We measured generation speed for DynamicConv on a P100 GPU using batch size one to be comparable with other . Results in the literature are based on either NVIDIA GTX-1080 GPUs or P100 GPUs. The effects of different GPU types is likely negligible because GPUs are vastly underutilized with batch size one. Table 7 shows that DynamicConv with a single decoder layer outperforms all previously reported non-autoregressive both in terms of speed as well as accuracy. Only two non-autoregressive concurrent efforts BID20 BID32 achieve a speedup over DynamicConv with a small drop in BLEU. Notably, both BID20 and BID32 distill autoregressive models into non-autoregressive models BID24, in order to improve their . Model (batch size = 1, beam size = 1) Param BLEU Sent/sec Tok/sec NAT (+ FT) -17.7 25.6 -NAT (+ FT + NPD=10) BID19 -18.7 12.7 -NAT (+ FT + NPD=100) BID19 (6-decoder layers (k=3,7,15,31,31,31) ) 200M 28.5 3.9 110.9 Table 7: Inference speed of non-autoregressive models and small decoder versions of DynamicConv on WMT English-German newstest2014. For some models, the decoding speed (sent/sec) is derived by taking the inverse of the sentence generation latency in the literature.

Dynamic lightweight convolutions are competitive to self-attention on language tasks.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the article for finding a single-sentence TLDR

Question: Owing to their connection with generative adversarial networks (GANs), saddle-point problems have recently attracted considerable interest in machine learning and beyond. By necessity, most theoretical guarantees revolve around convex-concave (or even linear) problems; however, making theoretical inroads towards efficient GAN training depends crucially on moving beyond this classic framework. To make piecemeal progress along these lines, we analyze the behavior of mirror descent (MD) in a class of non-monotone problems whose solutions coincide with those of a naturally associated variational inequality – a property which we call coherence. We first show that ordinary, “vanilla” MD converges under a strict version of this condition, but not otherwise; in particular, it may fail to converge even in bilinear models with a unique solution. We then show that this deficiency is mitigated by optimism: by taking an “extra-gradient” step, optimistic mirror descent (OMD) converges in all coherent problems. Our analysis generalizes and extends the of Daskalakis et al. for optimistic gradient descent (OGD) in bilinear problems, and makes concrete headway for provable convergence beyond convex-concave games. We also provide stochastic analogues of these , and we validate our analysis by numerical experiments in a wide array of GAN models (including Gaussian mixture models, and the CelebA and CIFAR-10 datasets). The surge of recent breakthroughs in deep learning has sparked significant interest in solving optimization problems that are universally considered hard. Accordingly, the need for an effective theory has two different sides: first, a deeper understanding would help demystify the reasons behind the success and/or failures of different training algorithms; second, theoretical advances can inspire effective algorithmic tweaks leading to concrete performance gains. For instance, using tools from the theory of dynamical systems, BID28 BID29 and Panageas & Piliouras showed that a wide variety of first-order methods (including gradient descent and mirror descent) almost always avoid saddle points. More generally, the optimization and machine learning communities alike have dedicated significant effort in understanding non-convex landscapes by searching for properties which could be leveraged for efficient training. As an example, the "strict saddle" property was shown to hold in a wide range of salient objective functions ranging from low-rank matrix factorization BID8 BID20 and dictionary learning [a,b], to principal component analysis BID19, and many other models. On the other hand, adversarial deep learning is nowhere near as well understood, especially in the case of generative adversarial networks (GANs) BID22. Despite an immense amount of recent scrutiny, our theoretical understanding cannot boast similar breakthroughs as in "single-agent" deep learning. Because of this, a considerable corpus of work has been devoted to exploring and enhancing the stability of GANs, including techniques as diverse as the use of Wasserstein metrics, critic gradient penalties BID23, feature matching, minibatch discrimination, etc. [;].Even before the advent of GANs, work on adaptive dynamics in general bilinear zero-sum games (e.g. Rock-Paper-Scissors) established that they lead to persistent, chaotic, recurrent (i.e. cycle-like) behavior [; ;]. Recently, simple specific instances of cycle-like behavior in bilinear games have been revisited mainly through the lens of GANs BID15; ]. Two important recent have established unified pictures about the behavior of continuous and discrete-time first order methods in bilinear games: First, established that continuous-time descent methods in zero-sum games (e.g., gradient descent, follow-the-regularized-leader and the like) are Poincaré recurrent, returning arbitrarily closely to their initial conditions infinitely many times. Second, BID4 examined the discrete-time analogues (gradient descent, multiplicative weights and follow-the-regularized-leader) showing that orbits spiral slowly outwards. These recurrent systems have formal connections to Hamiltonian dynamics and do not behave in a gradient-like fashion BID6; BID5. This is a critical failure of descent methods, but one which BID15 showed can be overcome through "optimism", interpreted in this context as an "extra-gradient" step that pushes the training process further along the incumbent gradient -as a , optimistic gradient descent (OGD) succeeds in cases where vanilla gradient descent (GD) fails (specifically, unconstrained bilinear saddle-point problems).A common theme in the above is that, to obtain a principled methodology for training GANs, it is beneficial to first establish improvements in a more restricted setting, and then test whether these gains carry over to more demanding learning environments. Following these theoretical breadcrumbs, we focus on a class of non-monotone problems whose solutions are related to those of a naturally associated variational inequality, a property which we call coherence. Then, hoping to overcome the shortcomings of ordinary descent methods by exploiting the problem's geometry, we examine the convergence of MD in coherent problems. On the positive side, we show that if a problem is strictly coherent (a condition satisfied by all strictly convex-concave problems), MD converges almost surely, even in stochastic problems (Theorem 3.1). However, under null coherence (the "saturated" opposite to strict coherence), MD spirals outwards from the problem's solutions and may cycle in perpetuity. The null coherence property covers all bilinear models, so this encompasses fully the analysis of BID4 for GD and follow-the-regularized-leader (FTRL) in general bilinear zero-sum games within our coherence framework. Thus, in and by themselves, gradient/mirror descent methods do not suffice for training convoluted, adversarial deep learning models. To mitigate this deficiency, we consider the addition of an extra-gradient step which looks ahead and takes an additional step along a "future" gradient. This technique was first introduced by BID27 and subsequently gained great popularity as the basis of the mirror-prox algorithm of Nemirovski which achieves an optimal O(1/n) convergence rate in Lipschitz monotone variational inequalities (see also , for a primal-dual variant of the method and BID25, for an extension to stochastic variational inequalities and saddle-point problems).In the learning literature, the extra-gradient technique (or, sometimes, a variant thereof) is often referred to as optimistic mirror descent (OMD) [] and its effectiveness in GAN training was recently examined by BID15 and Yadav et al. (the latter involving a damping mechanism for only one of the players). More recently, BID21 considered a variant method which incorporates a mechanism that "extrapolates from the past" in order to circumvent the need for a second oracle call in the extra-gradient step. Specifically, BID21 showed that the extra-gradient algorithm with gradient reuse converges a) geometrically in strongly monotone, deterministic variational inequalities; and b) ergodically in general stochastic variational inequalities, achieving in that case an oracle complexity bound that is √ 13/7/2 ≈ 68% of a bound previously established by BID25 for the mirror-prox algorithm. However, beyond convex-concave problems, averaging offers no tangible benefits because there is no way to relate the value of the ergodic average to the value of the iterates. As a , moving closer to GAN training requires changing both the algorithm's output as well as the accompanying analysis. With this as our guiding principle, we first show that the last iterate of OMD converges in all coherent problems, including null-coherent ones. As a special case, this generalizes and extends the of Noor et al. for OGD in pseudo-monotone problems, and also settles in the affirmative an open question of BID15 concerning the convergence of the last iterate of OGD in nonlinear problems. Going beyond deterministic problems, we also show that OMD converges with probability 1 even in stochastic saddle-point problems that are strictly coherent. These complement the existing literature on the topic by showing that a cheap extra-gradient add-on can lead to significant performance gains when applied to state-of-the-art methods (such as Adam). We validate this prediction for a wide array of standard GAN models in Section 5. Saddle-point problems. Consider a saddle-point problem of the general form min DISPLAYFORM0 where each feasible region X i, i = 1, 2, is a compact convex subset of a finite-dimensional normed space V i ≡ d i, and f: X ≡ X 1 × X 2 → denotes the problem's value function. From a gametheoretic standpoint, (SP) can be seen as a zero-sum game between two optimizing agents (or players): Player 1 (the minimizer) seeks to incur the least possible loss, while Player 2 (the maximizer) seeks to obtain the highest possible reward -both determined by f (x 1, x 2).To solve (SP), we will focus on incremental processes that exploit the individual loss/reward gradients of f (assumed throughout to be at least C 1 -smooth). Since the individual gradients of f will play a key role in our analysis, we will encode them in a single vector as DISPLAYFORM1 and, following standard conventions, we will treat g(x) as an element of Y ≡ V *, the dual of the ambient space V ≡ V 1 × V 2, assumed to be endowed with the product norm x 2 = x 1 2 + x 2 2.Variational inequalities and coherence. Most of the literature on saddle-point problems has focused on the monotone case, i.e., when f is convex-concave. In such problems, it is well known that solutions of (SP) can be characterized equivalently as solutions of the Stampacchia variational inequality g(x *), x − x * ≥ 0 for all x ∈ X (SVI) or, in Minty form: DISPLAYFORM2 The equivalence between solutions of (SP), (SVI) and (MVI) extends well beyond the realm of monotone problems: it trivially includes all bilinear problems (f (x 1, x 2) = x 1 Mx 2 ), pseudo-monotone objectives as in Noor et al., etc. For a concrete example which is not even pseudo-monotone, consider the problem min DISPLAYFORM3 The only saddle-point of f is x * =: it is easy to check that x * is also the unique solution of the corresponding variational inequality (VI) problems, despite the fact that f is not even pseudomonotone.1 This shows that this equivalence encompasses a wide range of phenomena that are innately incompatible with convexity/monotonicity, even in the lowest possible dimension; for an in-depth discussion we refer the reader to BID16.Motivated by all this, we introduce below the following notion of coherence:Definition 2.1. We say that (SP) is coherent if:1. Every solution of (SVI) also solves (SP).2. There exists a solution p of (SP) that satisfies (MVI). * of (SP) satisfies (MVI) locally, i.e., for all x sufficiently close to x *.In the above, if (MVI) holds as a strict inequality whenever x is not a solution thereof, (SP) will be called strictly coherent; by contrast, if (MVI) holds as an equality for all x ∈ X, we will say that (SP) is null-coherent. The notion of coherence will play a central part in our considerations, so a few remarks are in order. To the best of our knowledge, its first antecedent is a gradient condition examined by BID9 in the context of nonlinear programming; we borrow the term "coherence" from the more recent paper of Zhou et al. [2017b] who used the term "variational coherence" for a stronger variant of the above definition. We should also note here that the set of solutions of a coherent problem does not need to be convex: for instance, if Player 1 controls (x, y), and the objective function is f (x, y) = x 2 y 2 (i.e., Player 2 has no impact in the game), the set of solutions is the non-convex set X * = {(x, y): x = 0 or y = 0}. Moreover, regarding the distinction between coherence and strict coherence, we show in Appendix A that (SP) is strictly coherent when f is strictly convex-concave. At the other end of the spectrum, typical examples of problems that are null-coherent are bilinear objectives with an interior solution: for instance, f (x 1, x 2) = x 1 x 2 with x 1, x 2 ∈ [−1, 1] has g(x), x = x 1 x 2 − x 2 x 1 = 0 for all x 1, x 2 ∈ [−1, 1], so it is null-coherent. Finally, neither strict, nor null coherence imply a unique solution to (SP), a property which is particularly relevant for GANs (the first example above is strictly coherent, but does not admit a unique solution). The method. Motivated by its prolific success in convex programming, our starting point will be the well-known mirror descent (MD) method of Nemirovski & Yudin, suitably adapted to our saddle-point context. Several variants of the method exist, ranging from dual averaging [] to follow-the-regularized-leader; for a survey, we refer the reader to BID12.The basic idea of mirror descent is to generate a new state variable x + from some starting state x by taking a "mirror step" along a gradient-like vector y. To do this, let h: X → be a continuous and K-strongly convex distance-generating function (DGF) on X, i.e., DISPLAYFORM0 for all x, x ∈ X and all t ∈. In terms of smoothness (and in a slight abuse of notation), we also assume that the subdifferential of h admits a continuous selection, i.e., a continuous function ∇h: dom ∂h → Y such that ∇h(x) ∈ ∂h(x) for all x ∈ dom ∂h. 2 Then, following BID11, h generates a pseudo-distance on X via the relation DISPLAYFORM1 This pseudo-distance is known as the Bregman divergence. As we show in Appendix B, we have DISPLAYFORM2, so the convergence of a sequence X n to some target point p can be verified by showing that D(p, X n) → 0. On the other hand, D(p, x) typically fais to be symmetric and/or satisfy Require: K-strongly convex regularizer h: X →, step-size sequence γ n > 0 1: choose X ∈ dom ∂h # initialization 2: for n = 1, 2,... do 3:oracle query at X returns g # gradient feedback 4: set X ← P X (−γ n g) # new state 5: end for 6: return X the triangle inequality, so it is not a true distance function per se. Moreover, the level sets of D(p, x) may fail to form a neighborhood basis of p, so the convergence of X n to p does not necessarily imply that D(p, X n) → 0; we provide an example of this behavior in Appendix B. For technical reasons, it will be convenient to assume that such phenomena do not occur, i.e., that D(p, X n) → 0 whenever X n → p. This mild regularity condition is known in the literature as "Bregman reciprocity" BID13 BID0 BID30 BID10, and it will be our standing assumption in what follows (note also that it holds trivially for both Examples 3.1 and 3.2 below). Now, as with standard Euclidean distances, the Bregman divergence generates an associated proxmapping defined as P x (y) = arg min DISPLAYFORM3 In analogy with the Euclidean case (discussed below), the prox-mapping (3.3) produces a feasible point x + = P x (y) by starting from x ∈ dom ∂h and taking a step along a dual (gradient-like) vector y ∈ Y. In this way, we obtain the mirror descent (MD) algorithm DISPLAYFORM4 where γ n is a variable step-size sequence andĝ n is the calculated value of the gradient vector g(X n) at the n-th stage of the algorithm (for a pseudocode implementation, see Section 3).For concreteness, two widely used examples of prox-mappings are as follows:Example 3.1 (Euclidean projections). When X is endowed with the L 2 norm · 2, the archetypal prox-function is the (square of the) norm itself, i.e., h(x) = and the induced prox-mapping is P x (y) = Π(x + y), (3.4) with Π(x) = arg min x ∈X x − x 2 denoting the ordinary Euclidean projection onto X.Example 3.2 (Entropic regularization). When X is a d-dimensional simplex, a widely used DGF is the (negative) Gibbs-Shannon entropy h(x) = d j=1 x j log x j. This function is 1-strongly convex with respect to the L 1 norm [] and the associated pseudo-distance is the Kullback-Leibler divergence D KL (p, x) = d j=1 p j log(p j /x j); in turn, this yields the prox-mapping DISPLAYFORM5 The update rule x ← P x (y) is known in the literature as the multiplicative weights (MW) algorithm BID2, and is one of the centerpieces for learning in games BID18 BID17 BID14, adversarial bandits BID3, etc. Regarding the gradient input sequenceĝ n of (MD), we assume that it is obtained by querying a first-order oracle which outputs an estimate of g(X n) when called at X n. This oracle could be either perfect, returningĝ n = g(X n) for all n, or imperfect, providing noisy gradient estimations.3 By that token, we will make the following standard assumptions for the gradient feedback sequenceĝ n [; ; BID25 : a) Unbiasedness: DISPLAYFORM6 In the above, y * ≡ sup{y, x : x ∈ V, x ≤ 1} denotes the dual norm on Y while F n represents the history (natural filtration) of the generating sequence X n up to stage n (inclusive). Sinceĝ n is generated randomly from X n at stage n, it is obviously not F n -measurable, i.e.,ĝ n = g(X n) + U n+1, where U n is an adapted martingale difference sequence with ¾[U n+1 2 * | F n] ≤ σ 2 for some finite σ ≥ 0. Clearly, when σ = 0, we recover the exact gradient feedback frameworkĝ n = g(X n).Convergence analysis. When (SP) is convex-concave, it is customary to take as the output of (MD) the so-called ergodic averageX DISPLAYFORM7 or some other average of the sequence X n where the objective is sampled. The reason for this is that convexity guarantees -via Jensen's inequality and gradient monotonicity -that a regret-based analysis of (MD) can lead to explicit rates for the convergence ofX n to the solution set of (SP) [;]. However, when the problem is not convex-concave, the standard proof techniques for establishing convergence of the method's ergodic average no longer apply; instead, we need to examine the convergence properties of the generating sequence X n of (MD) directly. With all this in mind, our main for (MD) may be stated is as follows: Theorem 3.1. Suppose that (MD) is run with a gradient oracle satisfying (3.6) and a variable step-size sequence γ n such that DISPLAYFORM8 This establishes an important dichotomy between strict and null coherence: in strictly coherent problems, X n is attracted to the solution set of (SP); in null-coherent problems, X n drifts away and cycles without converging. In particular, this dichotomy leads to the following immediate corollaries: Corollary 3.2. Suppose that f is strictly convex-concave. Then, with assumptions as above, X n converges (a.s.) to the (necessarily unique) solution of (SP). Corollary 3.3. Suppose that f is bilinear and admits an interior saddle-point x * ∈ X •. If X 1 x * and (MD) is run with exact gradient input (σ = 0), we have lim n→∞ D(x *, X n) > 0.Since bilinear models include all finite two-player, zero-sum games, Corollary 3.3 also encapsulates the non-convergence of BID15 and Bailey & Piliouras for gradient descent and FTRL respectively (for a more comprehensive formulation, see Proposition C.3 in Appendix C). The failure of (MD) to converge in this case is due to the fact that, witout a mitigating mechanism in place, a "blind" first-order step could overshoot and spiral outwards, even with a vanishing step-size. This becomes even more pronounced in GANs where it can lead to mode collapse and/or cycles between different modes; the next two sections address precisely these issues.4 Extra-gradient analysisThe method. In convex-concave problems, taking an average of the algorithm's generated samples as in (3.7) may resolve cycling phenomena by inducing an auxiliary sequence that gravitates towards the "center of mass" of the driving sequence X n (which orbits interior solutions). However, this technique cannot be employed in problems that are not convex-concave because the structure of f cannot be leveraged to establish convergence of the ergodic average of the process. In view of this, we replace averaging with an optimistic "extra-gradient" step which uses the obtained information to amortize the next prox step (possibly by exiting the convex hull of generated states).The seed of this "extra-gradient" idea dates back to BID27 and Nemirovski, and has since found wide applications in optimization theory and beyond -for a survey, see BID12 and references therein. In a nutshell, given a state x, the extra-gradient method first generates an intermediate, "waiting" statê x = P x (−γg(x)) by taking a prox step as usual. However, instead of continuing fromx, the method samples g(x) and goes back to the original state x in order to generate a new state x + = P x (−γg(x)). Based on this heuristic, we obtain the optimistic mirror descent (OMD) algorithm DISPLAYFORM9 Algorithm 2: optimistic mirror descent (OMD) for saddle-point problemsRequire: K-strongly convex regularizer h: X →, step-size sequence γ n > 0 1: choose X ∈ dom ∂h # initialization 2: for n = 1, 2,... do 3:oracle query at X returns g # gradient feedback 4:set X + ← P X (−γ n g) # waiting state 5:oracle query at X + returns g + # gradient feedback 6: set X ← P X (−γ n g +) # new state 7: end for 8: return X where, in obvious notation,ĝ n andĝ n+1/2 represent gradient oracle queries at the incumbent and intermediate states X n and X n+1/2 respectively. For a pseudocode implementation, see Algorithm 2; see also Rakhlin & Sridharan and BID15 for a variant of the method with a "momentum" step, and BID21 for a gradient reuse mechanism that replaces a second oracle call with a past gradient. Convergence analysis. In his original analysis, Nemirovski considered the ergodic average (3.7) of the algorithm's iterates and established an O(1/n) convergence rate in monotone problems. However, as we explained above, even though this kind of averaging is helpful in convex-concave problems, it does not provide any tangible benefits beyond this class: in more general problems, X n appears to be the most natural solution candidate. Our first below justifies this choice in the class of coherent problems: Theorem 4.1. Suppose that (SP) is coherent and g is L-Lipschitz continuous. If (OMD) is run with exact gradient input (σ = 0) and γ n such that 0 < inf n γ n ≤ sup n γ n < K/L, the sequence X n converges monotonically to a solution x * of (SP), i.e., D(x *, X n) decreases monotonically to 0.Corollary 4.2. Suppose that f is bilinear. If (OMD) is run with assumptions as above, the sequence X n converges monotonically to a solution of (SP). for pseudo-monotone problems. Importantly, Theorem 4.1 shows that the extra-gradient step plays a crucial role in stabilizing (MD): not only does (OMD) converge in problems where (MD) provably fails, but this convergence is, in fact, monotonic. In other words, at each iteration, (OMD) comes closer to a solution of (SP), whereas (MD) may spiral outwards, ultimately converging to a limit cycle. This phenomenon is seen clearly in Fig. 1, and also in the detailed analysis of Appendix C.Of course, except for very special cases, the monotonic convergence of X n cannot hold when the gradient input to (OMD) is imperfect: a single "bad" sample ofĝ n would suffice to throw X n off-track. In this case, we have: Theorem 4.3. Suppose that (SP) is strictly coherent and (OMD) is run with a gradient oracle satisfying (3.6) and a variable step-size sequence γ n such that DISPLAYFORM10 Then, with probability 1, X n converges to a solution of (SP).It is worth noting here that the step-size policy in Theorem 4.3 is different than that of Theorem 4.1. This is due to a) the lack of randomness (which obviates the summability requirement ∞ n=1 γ 2 n < ∞ in Theorem 4.1); and b) the lack of Lipschitz continuity assumption (which, in the case of Theorem 4.1 guarantees monotonic decrease at each step, provided the step-size is not too big). Importantly, the maximum allowable step-size is also controlled by the strong convexity modulus of h, suggesting that the choice of distance-generating function can be fine-tuned further to allow for more aggressive step-size policies -a key benefit of mirror descent methods. Gaussian mixture models. For the experimental validation of our theoretical , we began by evaluating the extra-gradient add-on in a highly multi-modal mixture of 16 Gaussians arranged in a 4 × 4 grid as in Metz et al.. The generator and discriminator have 6 fully connected layers with 384 neurons and Relu activations (plus an additional layer for data space projection), and the generator generates 2-dimensional vectors. The output after {4000, 8000, 12000, 16000, 20000} iterations is shown in FIG3. The networks were trained with RMSprop [] and Adam BID26, and the are compared to the corresponding extra-gradient variant (for an explicit pseudocode representation in the case of Adam, see BID15 and Appendix E). Learning rates and hyperparameters were chosen by an inspection of grid search so as to enable a fair comparison between each method and its look-ahead version. Overall, the different optimization strategies without look-ahead exhibit mode collapse or oscillations throughout the training period (we ran all models for at least 20000 iterations in order to evaluate the hopping behavior of the generator). In all cases, the extra-gradient add-on performs consistently better in learning the multi-modal distribution and greatly reduces occurrences of oscillatory behavior. Experiments with standard datasets. In our experiments with Gaussian mixture models (GMMs), the most promising training method was Adam with an extra-gradient step (a concrete pseudocode implementation is provided in Appendix E). Motivated by this, we trained a Wasserstein-GAN on the CelebA and CIFAR-10 datasets using Adam, both with and without an extra-gradient step. The architecture employed was a standard DCGAN; hyperparameters and network architecture details may be found in Appendix E. Subsequently, to quantify the gains of the extra-gradient step, we employed the widely used inception score and Fréchet distance metrics, for which we report the in FIG4. Under both metrics, the extra-gradient add-on provides consistently higher scores after an initial warm-up period (and is considerably more stable). For visualization purposes, we also present in FIG5 an ensemble of samples generated at the end of the training period. Overall, the generated samples provide accurate feature representation and low distortion (especially in CelebA). Our suggest that the implementation of an optimistic, extra-gradient step is a flexible add-on that can be easily attached to a wide variety of GAN training methods (RMSProp, Adam, SGA, etc.), and provides noticeable gains in performance and stability. From a theoretical standpoint, the dichotomy between strict and null coherence provides a justification of why this is so: optimism eliminates cycles and, in so doing, stabilizes the method. We find this property particularly appealing because it paves the way to a local analysis with provable convergence guarantees in multi-modal settings, and beyond zero-sum games; we intend to examine this question in future work. We begin our discussion with some basic on coherence:Proposition A.1. If f is convex-concave, (SP) is coherent. In addition, if f is strictly convex-concave, (SP) is strictly coherent. Proof. Let x * be a solution point of (SP). Since f is convex-concave, first-order optimality gives DISPLAYFORM0 and DISPLAYFORM1 Combining the two, we readily obtain the (Stampacchia) variational inequality DISPLAYFORM2 In addition to the above, the fact that f is convex-concave also implies that g(x) is monotone in the sense that DISPLAYFORM3 for all x, x ∈ X []. Thus, setting x ← x * in (A.3) and invoking (SVI), we get DISPLAYFORM4 i.e., (MVI) is satisfied. To establish the converse implication, focus for concreteness on the minimizer, and note that (MVI) implies that DISPLAYFORM5 Now, if we fix some x 1 ∈ X 1 and consider the function φ(t) = f (x * 1 + t(x 1 − x * 1), x * 2 ), the inequality (A.5) yields DISPLAYFORM6 2 ). The maximizing component follows similarly, showing that x * is a solution of (SP) and, in turn, establishing that (SP) is coherent. For the strict part of the claim, the same line of reasoning shows that if g(x), x − x * = 0 for some x that is not a saddle-point of f, the function φ(t) defined above must be constant on, indicating in turn that f cannot be strictly convex-concave, a contradiction. We proceed to show that the solution set of a coherent saddle-point problem is closed (we will need this regularity in the convergence analysis of Appendix C):Lemma A.2. Let X * denote the solution set of (SP). If (SP) is coherent, X * is closed. Proof. Let x * n, n = 1, 2,..., be a sequence of solutions of (SP) converging to some limit point x * ∈ X. To show that X * is closed, it suffices to show that x * ∈ X.Indeed, given that (SP) is coherent, every solution thereof satisfies (MVI), so we have g(x), x−x * n ≥ 0 for all x ∈ X. With x * n → x * as n → ∞, it follows that DISPLAYFORM7 i.e., x * satisfies (MVI). By coherence, this implies that x * is a solution of (SP), as claimed. In this appendix, we provide some auxiliary and estimates that are used throughout the convergence analysis of Appendix C. Some of the we present here (or close variants thereof) are not new [see e.g., ; BID25 . However, the hypotheses used to obtain them vary wildly in the literature, so we provide all the necessary details for completeness. To begin, recall that the Bregman divergence associated to a K-strongly convex distance-generating function h : X → is defined as DISPLAYFORM0 with ∇h(x) denoting a continuous selection of ∂h(x). The induced prox-mapping is then given by DISPLAYFORM1 and is defined for all x ∈ dom ∂h, y ∈ Y (recall here that Y ≡ V * denotes the dual of the ambient vector space V). In what follows, we will also make frequent use of the convex conjugate h DISPLAYFORM2 By standard in convex analysis [, Chap. 26], h * is differentiable on Y and its gradient satisfies the identity ∇h * (y) = arg max DISPLAYFORM3 For notational convenience, we will also write Q(y) = ∇h * (y) (B.5) and we will refer to Q: Y → X as the mirror map generated by h. All these notions are related as follows: Lemma B.1. Let h be a distance-generating function on X. Then, for all x ∈ dom ∂h, y ∈ Y, we have: DISPLAYFORM4 Finally, if x = Q(y) and p ∈ X, we have DISPLAYFORM5 Remark. By (B.6b), we have ∂h(x +) ∅, i.e., x + ∈ dom ∂h. As a , the update rule x ← P x (y) is well-posed, i.e., it can be iterated in perpetuity. For (B.7), by a simple continuity argument, it suffices to show that the inequality holds for interior p ∈ X •. To establish this, let DISPLAYFORM6 Since h is strongly convex and y ∈ ∂h(x) by (B.6a), it follows that φ(t) ≥ 0 with equality if and only if t = 0. Since ψ(t) = ∇h(x + t(p − x)) − y, p − x is a continuous selection of subgradients of φ and both φ and ψ are continuous on, it follows that φ is continuously differentiable with φ = ψ on. Hence, with φ convex and φ(t) ≥ 0 = φ for all t ∈, we conclude that φ = ∇h(x) − y, p − x ≥ 0, which proves our assertion. We continue with some basic bounds on the Bregman divergence before and after a prox step. The basic ingredient for these bounds is a generalization of the (Euclidean) law of cosines which is known in the literature as the "three-point identity" BID13:Lemma B.2. Let h be a distance-generating function on X. Then, for all p ∈ X and all x, x ∈ dom ∂h, we have DISPLAYFORM7 Proof. By definition, we have: DISPLAYFORM8 (B.10)Our claim then follows by adding the last two lines and subtracting the first. With this identity at hand, we have the following series of upper and lower bounds: Proposition B.3. Let h be a K-strongly convex distance-generating function on X, fix some p ∈ X, and let x + = P x (y) for x ∈ dom ∂h, y ∈ Y. We then have: DISPLAYFORM9 Proof of (B.11a). By the strong convexity of h, we get DISPLAYFORM10 Proof of (B.11b) and (B.11c). By the three-point identity (B.9), we readily obtain DISPLAYFORM0 In turn, this gives DISPLAYFORM1 where, in the last step, we used (B.7) and the fact that x + = P x (y), so ∇h(x) + y ∈ ∂h(x +). The above is just (B.11b), so the first part of our proof is complete. For (B.11c), the bound (B.14) gives DISPLAYFORM2 Therefore, by Young's inequality [], we get DISPLAYFORM3 and hence (B.17) with the last step following from Lemma B.1 applied to x in place of p. DISPLAYFORM4 The first part of Proposition B.3 shows that X n converges to p if D(p, X n) → 0. However, as we mentioned in the main body of the paper, the converse may fail: in particular, we could have lim inf n→∞ D(p, X n) > 0 even if X n → p. To see this, let X be the L 2 ball of d and take h(x) = − 1 − x 2 2. Then, a straightforward calculation gives DISPLAYFORM5 (B.19) which admits p as a solution for all c ≥ 0 (so p belongs to the closure of L c (p) even though D(p, p) = 0 by definition). As a , under this distance-generating function, it is possible to have X n → p even when lim inf n→∞ D(p, X n) > 0 (simply take a sequence X n that converges to p while remaining on the same level set of D). As we discussed in the main body of the paper, such pathologies are discarded by the Bregman reciprocity condition D(p, X n) → 0 whenever X n → p. (B.20) This condition comes into play at the very last part of the proofs of Theorems 3.1 and 4.1; other than that, we will not need it in the rest of our analysis. Finally, for the analysis of the OMD algorithm, we will need to relate prox steps taken along different directions: Proposition B.4. Let h be a K-strongly convex distance-generating function on X and fix some p ∈ X, x ∈ dom ∂h. Then: a) For all y 1, y 2 ∈ Y, we have: DISPLAYFORM6 i.e., P x is (1/K)-Lipschitz.b) In addition, letting x + 1 = P x (y 1) and x + 2 = P x (y 2), we have: DISPLAYFORM7 Proof. We begin with the proof of the Lipschitz property of P x. Indeed, for all p ∈ X, (B.7) gives ∇h(x DISPLAYFORM8) and rearranging, we obtain ∇h(x DISPLAYFORM9 (B.24) By the strong convexity of h, we also have K x DISPLAYFORM10 (B.25) Hence, combining (B.24) and (B.25), we get K x DISPLAYFORM11 26) and our assertion follows. For the second part of our claim, the bound (B.11b) of Proposition B.3 applied to x + 2 = P x (y 2) readily gives DISPLAYFORM12 (B.27) thus proving (B.22a). To complete our proof, note that (B.11b) with DISPLAYFORM13 where we used Young's inequality and (B.11a) in the second inequality. The bound (B.22b) then follows by substituting (B.30) in (B.27). We begin by recalling the definition of the mirror descent algorithm. With notation as in the previous section, the algorithm is defined via the recursive scheme DISPLAYFORM0 where γ n is a variable step-size sequence andĝ n is the calculated value of the gradient vector g(X n) at the n-th stage of the algorithm. As we discussed in the main body of the paper, the gradient input sequenceĝ n of (MD) is assumed to satisfy the standard oracle assumptions a) Unbiasedness: DISPLAYFORM1 where F n represents the history (natural filtration) of the generating sequence X n up to stage n (inclusive).With this preliminaries at hand, our convergence proof for (MD) under strict coherence will hinge on the following : Proposition C.1. Suppose that (SP) is coherent and (MD) is run with a gradient oracle satisfying (3.6) and a variable step-size γ n such that DISPLAYFORM2 Proposition C.2. Suppose that (SP) is strictly coherent and (MD) is run with a gradient oracle satisfying (3.6) and a step-size γ n such that DISPLAYFORM3 n < ∞. Then, with probability 1, there exists a (possibly random) solution x * of (SP) such that lim inf n→∞ D(x *, X n) = 0.Proposition C.1 can be seen as a "dichotomy" : it shows that the Bregman divergence is an asymptotic constant of motion, so (MD) either converges to a saddle-point x * (if D(x *) = 0) or to some nonzero level set of the Bregman divergence (with respect to x *). In this way, Proposition C.1 rules out more complicated chaotic or aperiodic behaviors that may arise in general -for instance, as in the analysis of Palaiopanos et al. for the long-run behavior of the multiplicative weights algorithm in two-player games. However, unless this limit value can be somehow predicted (or estimated) in advance, this cannot be easily applied. This is the main role of Proposition C.2: it shows that (MD) admits a subsequence converging to a solution of (SP) so, by (B.20), the limit of D(x *, X n) must be zero. Our first step is to prove Proposition C.2. To do this, we first recall the following law of large numbers for L 2 martingales:Theorem (, Theorem 2.18). Let Y n = n k=1 ζ k be a martingale and T n a nondecreasing sequence such that lim n→∞ τ n = ∞. Then, DISPLAYFORM4 on the set DISPLAYFORM5 Proof of Proposition C.2. We begin with the technical observation that the solution set X * of (SP) is closed -and hence, compact (cf. Lemma A.2 in Appendix A). Clearly, if X * = X, there is nothing to show; hence, without loss of generality, we may assume in what follows that X * X.Assume now ad absurdum that, with positive probability, the sequence X n generated by (MD) admits no limit points in X *. Conditioning on this event, and given that X * is compact, there exists a (nonempty) compact set C ⊂ X such that C ∩ X * = ∅ and X n ∈ C for all sufficiently large n. Moreover, letting p be as in Definition 2.1, we have g(x), x − p > 0 whenever x ∈ C. Therefore, by the continuity of g and the compactness of X * and C, there exists some a > 0 such that DISPLAYFORM0 To proceed, let D n = D(p, X n). Then, by Proposition B.3, we have DISPLAYFORM1 where, in the last line, we set U n+1 =ĝ n − g(X n), ξ n+1 = − U n+1, X n − p, and we invoked the assumption that (SP) is coherent. Hence, telescoping (C.3) yields the estimate DISPLAYFORM2 Subsequently, letting τ n = n k=1 γ k and using (C.2), we obtain DISPLAYFORM3 By the unbiasedness hypothesis of (3.6) for U n, we have DISPLAYFORM4 Moreover, since U n is bounded in L 2 and γ n is 2 summable (by assumption), it follows that DISPLAYFORM5 Therefore, by the law of large numbers for L 2 martingales stated above [, Theorem 2 .18], we conclude that τ −1 n n k=1 γ k ξ k+1 converges to 0 with probability 1. Finally, for the last term of (C.4), let DISPLAYFORM6 i.e., S n is a submartingale with respect to F n. Furthermore, by the law of total expectation, we also have DISPLAYFORM7 Hence, by Doob's submartingale convergence theorem [, Theorem 2 .5], we conclude that S n converges to some (almost surely finite) random variable S ∞ with ¾[S ∞] < ∞, implying in turn that lim n→∞ S n+1 /τ n = 0 (a.s.).Applying all of the above, the estimate (C.4) gives D n+1 ≤ D 1 − aτ n /2 for sufficiently large n, so D(p, X n) → −∞, a contradiction. Going back to our original assumption, this shows that, at least one of the limit points of X n must lie in X * (a.s.), as claimed. We now turn to the proof of Proposition C.1:Proof of Proposition C.1. Let x * ∈ X * be a limit point of X n, as guaranteed by Proposition C.2, and let D n = D(x *, X n). Then, by Proposition B.3, we have DISPLAYFORM8 and hence, for large enough n: DISPLAYFORM9 where we used the ansatz that g(X n), X n − x * ≤ 0 for sufficiently large n (to be proved below), and, as in the proof of Proposition C.2, we set U n+1 =ĝ n − g(X n), ξ n+1 = − U n+1, X n − x *. Thus, conditioning on F n and taking expectations, we get DISPLAYFORM10 where we used the oracle assumptions (3.6) and the fact that X n is F n -measurable (by definition). DISPLAYFORM11 i.e., R n is an F n -adapted supermartingale. Since DISPLAYFORM12 i.e., R n is uniformly bounded in L 1. Thus, by Doob's convergence theorem for supermartingales [, Theorem 2.5], it follows that R n converges (a.s.) to some finite random variable R ∞ with ¾[R ∞] < ∞. In turn, by inverting the definition of R n, this shows that D n converges (a.s.) to some random variable D(x *) with ¾[D(x *)] < ∞, as claimed. It remains to be shown that g(X n), X n − x * ≥ 0 for sufficiently large n. By Definition 2.1, this amounts to showing that, for all large enough n, X n lies in a neighborhood U of x * such that (MVI) holds. Since x * has been chosen so that lim inf D(x *, X n) = 0, it follows that, for all ε > 0, there exists some n 0 such that ∞ n=n 0 γ 2 n < ε and X n 0 ∈ U. Hence, arguing in the same way as in the proof of Theorem 5.2 of Zhou et al.[2017a], we conclude that (X n ∈ U for all n ≥ n 0) = 1, implying in turn that g(X n), X n − x * ≥ 0 for all n ≥ n 0. This proves our last claim and concludes our proof. With all this at hand, we are finally in a position to prove our main for (MD):Proof of Theorem 3.1(a). Proposition C.2 shows that, with probability 1, there exists a (possibly random) solution x * of (SP) such that lim inf n→∞ X n − x * = 0 and, hence, lim inf n→∞ D(x *, X n) = 0 (by Bregman reciprocity). Since lim n→∞ D(x *, X n) exists with probability 1 (by Proposition C.1), it follows that lim n→∞ D(x *, X n) = lim inf n→∞ D(x *, X n) = 0, i.e., X n converges to x *.We proceed with the negative hinted at in the main body of the paper, namely the failure of (MD) to converge under null coherence:Proof of Theorem 3.1 (b). The evolution of the Bregman divergence under (MD) satisfies the identity DISPLAYFORM13 where, in the last line, we used the null coherence assumption g(x), x − x * = 0 for all x ∈ X. Since D(X n, X n+1) ≥ 0, taking expecations above shows that D(x *, X n) is nondecreasing, as claimed. With Theorem 3.1 at hand, the proof of Corollary 3.2 is an immediate consequence of the fact that strictly convex-concave problems satisfy strict coherence (Proposition A.1). As for Corollary 3.3, we provide below a more general for two-player, zero-sum finite games. To state it, let A i = {1, . . ., A i}, i = 1, 2, be two finite sets of pure strategies, and let X i = ∆(A i) denote the set of mixed strategies of player i. A finite, two-player zero-sum game is then defined by a matrix M ∈ A 1 ×A 2 so that the loss of Player 1 and the reward of Player 2 in the mixed strategy profile x = (x 1, x 2) ∈ X are concurrently given by DISPLAYFORM14 Then, writing Γ ≡ Γ(A 1, A 2, M) for the ing game, we have: Proposition C.3. Let Γ be a two-player zero-sum game with an interior Nash equilibrium x *. If X 1 x * and (MD) is run with exact gradient input (σ DISPLAYFORM15 DISPLAYFORM16 Remark. Note that non-convergence does not require any summability assumptions on γ n .In words, Proposition C.3 states that (MD) does not converge in finite zero-sum games with a unique interior equilibrium and exact gradient input: instead, X n cycles at positive Bregman distance from the game's Nash equilibrium. Heuristically, the reason for this behavior is that, for small γ → 0, the incremental step V γ (x) = P x (−γg(x)) − x of (MD) is essentially tangent to the level set of D(x *, ·) that passes through x.4 For finite γ > 0, things are even worse because V γ (x) points noticeably away from x, i.e., towards higher level sets of D. As a , the "best-case scenario" for (MD) is to orbit x * (when γ → 0); in practice, for finite γ, the algorithm takes small outward steps throughout its runtime, eventually converging to some limit cycle farther away from x *.We make this intuition precise below (for a schematic illustration, see also Fig. 1 Proof of Proposition C.3. Write v 1 (x) = −Mx 2 and v 2 (x) = x 1 M for the players' payoff vectors under the mixed strategy profile x = (x 1, x 2). By construction, we have g(x) = −(v 1 (x), v 2 (x)). Furthermore, since x * is an interior equilibrium of f, elementary game-theoretic considerations show that v 1 (x *) and v 2 (x *) are both proportional to the constant vector of ones. We thus get g(x), x − x * = v 1 (x), x 1 − x * 1 + v 2 (x), x 2 − x * 2 = −x 1 Mx 2 + (x * 1) Mx 2 + x 1 Mx 2 − x 1 Mx * 2 = 0, (C.16) where, in the last line, we used the fact that x * is interior. This shows that f satisfies null coherence, so our claim follows from Theorem 3.1(b).For our second claim, arguing as above and using (B.11c), we get D(x *, X n+1) ≤ D(x *, X n) + γ n g(X n), X n − x * + γ where we used the fact that g is L-Lipschitz and that p is a solution of (SP) such that (MVI) holds for all x ∈ X.We are now finally in a position to prove Theorem 4.1 (reproduced below for convenience): Theorem. Suppose that (SP) is coherent and g is L-Lipschitz continuous. If (OMD) is run with exact gradient input and a step-size sequence γ n such that 0 < lim n→∞ γ n ≤ sup n γ n < K/L, (D.3) the sequence X n converges monotonically to a solution x * of (SP), i.e., D(x *, X n) is non-increasing and converges to 0.Proof. Let p be a solution of (SP) such that (MVI) holds for all x ∈ X (that such a solution exists is a consequence of Definition 2.1). Then, by the stated assumptions for γ n, Lemma D.1 yields DISPLAYFORM0 where α ∈ is such that γ 2 n < αK/L for all n (that such an α exists is a consequence of the assumption that sup n γ n < K/L). Now, telescoping (D.1), we obtain 5) and hence: DISPLAYFORM1 DISPLAYFORM2 With sup n γ n < K/L, the above estimate readily yields ∞ n=1 X n+1/2 − X n 2 < ∞, which in turn implies that X n+1/2 − X n → 0 as n → ∞.By the compactness of X, we further infer that X n admits an accumulation point x In this section we present the of our image experiments using OMD training techniques. Inception and FID scores obtained by our model during training were reported in FIG4: as can be seen there, the extra-gradient add-on improves the performance of GAN training and efficiently stabilizes the model; without the extra-gradient step, performance tends to drop noticeably after approximately 100k steps. For ease of comparison, we provide below a collection of samples generated by Adam and optimistic Adam in the CelebA and CIFAR-10 datasets. Especially in the case of CelebA, the generated samples are consistently more representative and faithful to the target data distribution. For the reproducibility of our experiments, we provide TAB2 the network architectures and the hyperparameters of the GANs that we used. The architecture employed is a standard DCGAN architecture with a 5-layer generator with batchnorm, and an 8-layer discriminator. The generated samples were 32×32×3 RGB images.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We show how the inclusion of an extra-gradient step in first-order GAN training methods can improve stability and lead to improved convergence results.

Represent the TLDR for finding the corresponding scientific article

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.