Question: Represent the article for finding a single-sentence TLDR

Question: Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks. Multi-view learning methods provide the ability to extract information from different views of the data and enable self-supervised learning of useful features for future prediction when annotated data is not available BID16. Minimising the disagreement among multiple views helps the model to learn rich feature representations of the data and, also after training, the ensemble of the feature vectors from multiple views can provide an even stronger generalisation ability. Distributional hypothesis BID22 noted that words that occur in similar contexts tend to have similar meaning BID51, and distributional similarity BID19 consolidated this idea by stating that the meaning of a word can be determined by the company it has. The hypothesis has been widely used in machine learning community to learn vector representations of human languages. Models built upon distributional similarity don't explicitly require human-annotated training data; the supervision comes from the semantic continuity of the language data. Large quantities of annotated data are usually hard and costly to obtain, thus it is important to study unsupervised and self-supervised learning. Our goal is to propose learning algorithms built upon the ideas of multi-view learning and distributional hypothesis to learn from unlabelled data. We draw inspiration from the lateralisation and asymmetry in information processing of the two hemispheres of the human brain where, for most adults, sequential processing dominates the left hemisphere, and the right hemisphere has a focus on parallel processing BID9, but both hemispheres have been shown to have roles in literal and non-literal language comprehension BID14 BID15.Our proposed multi-view frameworks aim to leverage the functionality of both RNN-based models, which have been widely applied in sentiment analysis tasks BID57, and the linear/log-linear models, which have excelled at capturing attributional similarities of words and sentences BID5 BID6 BID24 BID51 for learning sentence representations. Previous work on unsupervised sentence representation learning based on distributional hypothesis can be roughly categorised into two types:Generative objective: These models generally follow the encoder-decoder structure. The encoder learns to produce a vector representation for the current input, and the decoder learns to generate sentences in the adjacent context given the produced vector BID28 BID24 BID20 BID50. The idea is straightforward, yet its scalability for very large corpora is hindered by the slow decoding process that dominates training time, and also the decoder in each model is discarded after learning as the quality of generated sequences is not the main concern, which is a waste of parameters and learning effort. Our first multi-view framework has a generative objective and uses an RNN as the encoder and an invertible linear projection as the decoder. The training time is drastically reduced as the decoder is simple, and the decoder is also utilised after learning. A regularisation is applied on the linear decoder to enforce invertibility, so that after learning, the inverse of the decoder can be applied as a linear encoder in addition to the RNN encoder. Discriminative Objective: In these models, a classifier is learnt on top of the encoders to distinguish adjacent sentences from those that are not BID31 BID26 BID40 BID33; these models make a prediction using a predefined differential similarity function on the representations of the input sentence pairs or triplets. Our second multi-view framework has a discriminative objective and uses an RNN encoder and a linear encoder; it learns to maximise agreement among adjacent sentences. Compared to earlier work on multi-view learning BID16 BID17 BID52 that takes data from various sources or splits data into disjoint populations, our framework processes the exact same data in two distinctive ways. The two distinctive information processing views tend to encode different aspects of an input sentence; forcing agreement/alignment between these views encourages each view to be a better representation, and is beneficial to the future use of the learnt representations. Our contribution is threefold:• Two multi-view frameworks for learning sentence representations are proposed, in which one framework uses a generative objective and the other one adopts a discriminative objective. Two encoding functions, an RNN and a linear model, are learnt in both frameworks.• The show that in both frameworks, aligning representations from two views gives improved performance of each individual view on all evaluation tasks compared to their single-view trained counterparts, and furthermore ensures that the ensemble of two views provides even better than each improved view alone.• Models trained under our proposed frameworks achieve good performance on the unsupervised tasks, and overall outperform existing unsupervised learning models, and armed with various pooling functions, they also show solid on supervised tasks, which are either comparable to or better than those of the best unsupervised transfer model. It is shown BID24 that the consistency between supervised and unsupervised evaluation tasks is much lower than that within either supervised or unsupervised tasks alone and that a model that performs well on supervised tasks may fail on unsupervised tasks. BID13 subsequently showed that, with a labelled training corpus, such as SNLI BID8 and MultiNLI BID56, the ing representations of the sentences from the trained model excel in both supervised and unsupervised tasks. Multi-task learning BID48 also gives impressive performance on downstream tasks while labelled data is costly. Our model is able to achieve good on both groups of tasks without labelled information. Our goal is to marry RNN-based sentence encoder and the avg-on-word-vectors sentence encoder into multi-view frameworks with simple training objectives. The motivation for the idea is that, as mentioned in the prior work, RNN-based encoders process the sentences sequentially, and are able to capture complex syntactic interactions, while the avg-on-wordvectors encoder has been shown to be good at capturing the coarse meaning of a sentence which could be useful for finding paradigmatic parallels BID51.We present two multi-view frameworks, each of which learns two different sentence encoders; after learning, the vectors produced from two encoders of the same input sentence are used to compose the sentence representation. The details of our learning frameworks are described as follows: In our multi-view frameworks, we first introduce two encoders that, after learning, can be used to build sentence representations. One encoder is a bi-directional Gated Recurrent Unit BID10 f (s; φ), where s is the input sentence and φ is the parameter vector in the GRU. During learning, only hidden state at the last time step is sent to the next stage in learning. The other encoder is a linear avg-on-word-vectors model g(s; W), which basically transforms word vectors in a sentence by a learnable weight matrix W and outputs an averaged vector. Given the finding BID50 that neither an autoregressive nor an RNN decoder is necessary for learning sentence representations that excel on downstream tasks, our learning framework only learns to predict words in the next sentence. The framework has an RNN encoder f, and a linear decoder h. Given an input sentence s i, the encoder produces a vector z f i = f (s i ; φ), and the decoder h projects the vector to DISPLAYFORM0, which has the same dimension as the word vectors v w. Negative sampling is applied to calculate the likelihood of generating the j-th word in the (i + 1)-th sentence, shown in Eq. 1. DISPLAYFORM1 where v w k are pretrained word vectors for w k, the empirical distribution P e (w) is the unigram distribution raised to power 0.75 BID38, and K is the number of negative samples. The learning objective is to maximise the likelihood for words in all sentences in the training corpus. Ideally, the inverse of h should be easy to compute so that during testing we can set g = h −1. As h is a linear projection, the simplest situation is when U is an orthogonal matrix and its inverse is equal to its transpose. Often, as the dimensionality of vector z f i doesn't necessarily need to match that of word vectors v w, U is not a square matrix BID0. To enforce invertibility on U, a row-wise orthonormal regularisation on U is applied during training, which leads to U U = I, where I is the identity matrix, thus the inverse function is simply h −1 (x) = U x, which is easily computed. The regularisation formula is ||U U − I|| F, where || · || F is the Frobenius norm. Specifically, the update rule BID11 for the regularisation is: DISPLAYFORM2 where β is set to 0.01. After learning, we set W = U, then the inverse of the decoder h becomes the encoder g. Compared to prior work with generative objective, our framework reuses the decoding function rather than ignoring it for building sentence representations after learning, thus information encoded in the decoder is also utilised. Our multi-view framework with discriminative objective learns to maximise the agreement between the representations of a sentence pair across two views if one sentence in the pair is in the neighbourhood of the other one. An RNN encoder f (s; φ) and a linear avg-on-word-vectors g(s; W) produce a vector representation z DISPLAYFORM0 where DISPLAYFORM1 where τ is the trainable temperature term, which is essential for exaggerating the difference between adjacent sentences and those that are not. The neighbourhood/context window c, and the batch size N are hyperparameters. The choice of cosine similarity based loss is based on the observations BID51 that, of word vectors derived from distributional similarity, vector length tends to correlate with frequency of words, thus angular distance captures more important meaning-related information. Also, since our model is unsupervised/self-supervised, whatever similarity there is between neighbouring sentences is what is learnt as important for meaning. The postprocessing step BID6, which removes the top principal component of a batch of representations, is applied on produced representations from f and g respectively after learning with a final l 2 normalisation. In addition, in our multi-view framework with discriminative objective, in order to reduce the discrepancy between training and testing, the top principal component is estimated by the power iteration method BID39 and removed during learning. Three unlabelled corpora from different genres are used in our experiments, including BookCorpus BID59, UMBC News BID21 and Amazon Book Review 2 BID35; six models are trained separately on each of three corpora with each of two objectives. The summary statistics of the three corpora can be found in TAB6. Adam optimiser BID27 and gradient clipping BID43 are applied for stable training. Pretrained word vectors, fastText BID7, are used in our frameworks and fixed during learning. TAB6: Summary statistics of the three corpora used in our experiments. For simplicity, the three corpora will be referred to as 1, 2 and 3 in the following tables respectively. Table 2: Representation pooling in testing phase. "max(·)", "mean(·)", and "min(·)" refer to global max-, mean-, and min-pooling over time, which in a single vector. The table also presents the diversity of the way that a single sentence representation can be calculated. X i refers to word vectors in i-th sentence, and H i refers to hidden states at all time steps produced by f. DISPLAYFORM0 All of our experiments including training and testing are done in PyTorch BID44. The modified SentEval BID12 package with the step that removes the first principal component is used to evaluate our models on the downstream tasks. Hyperparameters, including negative samples K in the framework with generative objective, context window c in the one with discriminative objective, are tuned only on the averaged performance on STS14 of the model trained on the BookCorpus; STS14/G1 and STS14/D1 are thus marked with a in TAB1 to indicate possible overfitting on that dataset/model only. Batch size N and dimension d in both frameworks are set to be the same for fair comparison. Hyperparameters are summarised in supplementary material. Representation: For a given sentence input s with M words, suggested by BID45 BID30, the representation is calculated as z = ẑ f +ẑ g /2, whereẑ refers to the post-processed and normalised vector, and is mentioned in Table 2. BID34. We compare our models with: • Unsupervised learning: We selected models with strong from related work, including fastText, fastText+WR.• Semisupervised learning: The word vectors are pretrained on each task BID54 without label information, and word vectors are averaged to serve as the vector representation for a given sentence BID6.• Supervised learning: ParaNMT BID55 is included as a supervised learning method as the data collection requires a neural machine translation system trained in supervised way. InferSent 3 BID13 trained on SNLI and MultiNLI is included as well. The are presented in TAB1. Since the performance of FastSent BID24 and QT BID33 were only evaluated on STS14, we compare to their in TAB2.All six models trained with our learning frameworks outperform other unsupervised and semi supervised learning methods, and the model trained on the UMBC News Corpus with discriminative objective gives the best performance likely because the STS tasks contain multiple news-and headlines-related datasets which is well matched by the domain of the UMBC News Corpus. The evaluation on these tasks involves learning a linear model on top of the learnt sentence representations produced by the model. Since a linear model is capable of selecting the most relevant dimensions in the feature vectors to make predictions, it is preferred to concatenate various types of representations to form a richer, and possibly more redundant feature vector, which allows the linear model to explore the combination of different aspects of encoder functions to provide better . Representation: Inspired by prior work BID36 BID46, the representation z f is calculated by concatenating the outputs from the global mean-, max-and min-pooling on top of the hidden states H, and the last hidden state, and z g is calculated with three pooling functions as well. The post-processing and the normalisation step is applied individually. These two representations are concatenated to form a final sentence representation. Table 2 presents the details. Tasks: Semantic relatedness (SICK) BID34, paraphrase detection (MRPC) BID18, question-type classification (TREC) BID32, movie review sentiment (MR) BID42, Stanford Sentiment Treebank (SST) BID47, customer product reviews (CR) BID25, subjectivity/objectivity classification (SUBJ) BID41, opinion polarity (MPQA) BID53. The are presented in TAB3.Comparison: Our as well as related of supervised task-dependent training models, supervised learning models, and unsupervised learning models are presented in TAB3. Note that, for fair comparison, we collect the of the best single model (MC-QT, BID33) trained on BookCorpus. Six models trained with our learning frameworks either outperform other existing methods, or achieve similar on some tasks. The model trained on the Amazon Book Review gives the best performance on sentiment analysis tasks, since the corpus conveys strong sentiment information. In both frameworks, RNN encoder and linear encoder perform well on all tasks, and generative objective and discriminative objective give similar performance. The orthonormal regularisation applied on the linear decoder to enforce invertibility in our multi-view framework encourages the vector representations produced by f and those by h −1, which is g in testing, to agree/align with each other. A direct comparison is to train our multi-view framework Table 6: Ablation study on our multi-view frameworks. Variants of our frameworks are tested to illustrate the advantage of our multi-view learning frameworks. In general, under the proposed frameworks, learning to align representations from both views helps each view to perform better and an ensemble of both views provides stronger than each of them. The arrow and value pair indicate how a differs from our multi-view learning framework. Better view in colour. DISPLAYFORM0 Multi-view with g 1 and g 2: a ij = cos(z without the invertible constraint, and still directly use U as an additional encoder in testing. The of our framework with and without the invertible constraint are presented in Table 6 . DISPLAYFORM1 The ensemble method of two views, f and g, on unsupervised evaluation tasks (STS12-16 and SICK14) is averaging, which benefits from aligning representations from f and g by applying invertible constraint, and the RNN encoder f gets improved on unsupervised tasks by learning to align with g. On supervised evaluation tasks, as the ensemble method is concatenation and a linear model is applied on top of the concatenated representations, as long as the encoders in two views process sentences distinctively, the linear classifier is capable of picking relevant feature dimensions from both views to make good predictions, thus there is no significant difference between our multi-view framework with and without invertible constraint. In order to determine if the multi-view framework with two different views/encoding functions is helping the learning, we compare our framework with discriminative objective to other reasonable variants, including the multi-view model with two functions of the same type but parametrised independently, either two f -s or two g-s, and the single-view model with only one f or g. Table 6 presents the of the models trained on UMBC News Corpus. As specifically emphasised in previous work BID24, linear/log-linear models, which include g in our model, produce better representations for unsupervised evaluation tasks than RNN-based models do. This can also be observed in Table 6 as well, where g consistently provides better on unsupervised tasks than f. In addition, as expected, multi-view learning with f and g, improves the ing performance of f on unsupervised tasks, also improves the ing g on supervised evaluation tasks. Provided the of models with generative and discriminative objective in Table 6, we confidently show that, in our multi-view frameworks with f and g, the two encoding functions improve each other's view. In general, aligning the representations generated from two distinct encoding functions ensures that the ensemble of them performs better. The two encoding functions f and g encode the input sentence with emphasis on different aspects, and the subsequently trained linear model for each of the supervised downstream tasks benefits from this diversity leading to better predictions. However, on unsupervised evaluation tasks, simply averaging representations from two views without aligning them during learning leads to poor performance and it is worse than g (linear) encoding function solely. Our multi-view frameworks ensure that the ensemble of two views provides better performance on both supervised and unsupervised evaluation tasks. Compared with the ensemble of two multi-view models, each with two encoding functions of the same type, our multi-view framework with f and g provides slightly better on unsupervised tasks, and similar on supervised evaluation tasks, while our model has much higher training efficiency. Compared with the ensemble of two single-view models, each with only one encoding function, the matching between f and g in our multi-view model produces better . We proposed multi-view sentence representation learning frameworks with generative and discriminative objectives; each framework combines an RNN-based encoder and an average-on-word-vectors linear encoder and can be efficiently trained within a few hours on a large unlabelled corpus. The experiments were conducted on three large unlabelled corpora, and meaningful comparisons were made to demonstrate the generalisation ability and transferability of our learning frameworks and consolidate our claim. The produced sentence representations outperform existing unsupervised transfer methods on unsupervised evaluation tasks, and match the performance of the best unsupervised model on supervised evaluation tasks. Our experimental support the finding BID24 that linear/log-linear models (g in our frameworks) tend to work better on the unsupervised tasks, while RNN-based models (f in our frameworks) generally perform better on the supervised tasks. As presented in our experiments, multi-view learning helps align f and g to produce better individual representations than when they are learned separately. In addition, the ensemble of both views leveraged the advantages of both, and provides rich semantic information of the input sentence. Future work should explore the impact of having various encoding architectures and learning under the multi-view framework. Our multi-view learning frameworks were inspired by the asymmetric information processing in the two hemispheres of the human brain, in which the left hemisphere is thought to emphasise sequential processing and the right one more parallel processing BID9. Our experimental raise an intriguing hypothesis about how these two types of information processing may complementarily help learning. The details, including size of each dataset and number of classes, about the evaluation tasks are presented below 1. The Power Iteration was proposed in BID39, and it is an efficient algorithm for estimating the top eigenvector of a given covariance matrix. Here, it is used to estimate the top principal component from the representations produced from f and g separately. We omit the superscription here, since the same step is applied to both f and g. Suppose there is a batch of representations Z = [z1, z2 ..., zN] ∈ R 2d×N from either f or g, the Power Iteration method is applied here to estimate the top eigenvector of the covariance matrix 2: C = ZZ, and it is described in Algorithm 1:Algorithm 1 Estimating the First Principal Component BID39 Input: Covariance matrix C ∈ R 2d×2d, number of iterations T Output: First principal component u ∈ R u ← Cu, 4: u ← u ||u||In our experiments, T is set to be 5. The hyperparameters we need to tune include the batch size N, the dimension of the GRU encoder d, and the context window c, and the number of negative samples K. The we presented in this paper is based on the model trained with N = 512, d = 1024. Specifically, in discriminative objective, the context window is set c = 3, and in generative objective, the number of negative samples is set K = 5. It takes up to 8GB on a GTX 1080Ti GPU.The initial learning rate is 5 × 10 −4, and we didn't anneal the learning rate through the training. All weights in the model are initialised using the method proposed in BID23, and all gates in the bi-GRU are initialised to 1, and all biases in the single-layer neural network are zeroed before training. The word vectors are fixed to be those in the FastText BID7, and we don't finetune them. Words that are not in the FastText's vocabulary are fixed to 0 vectors through training. The temperature term is initialised as 1, and is tuned by the gradient descent during training. The temperature term is used to convert the agreement aij to a probability distribution pij in Eq. 1 in the main paper. In our experiments, τ is a trainable parameter initialised to 1 that decreased consistently through training. Another model trained with fixed τ set to the final value performed similarly. Table 2: Effect of the Post-processing Step.'WR' refers to the post-processing step BID6 which removes the principal component of a set of learnt vectors. The postprocessing step overall improves the performance of our models on unsupervised evaluation tasks, and also improves the models with generative objective on supervised sentence similarity tasks. However, it doesn't have a significant impact on single sentence classification tasks. 5 Combining both generative and discriminative objective in our multi-view framework Models with both generative and discriminative objectives are trained to see if further improvement can be provided by combining an RNN encoder, an inverse of a linear decoder in the generative objective and a linear encoder in the discriminative objective. The of models trained on BookCorpus and UMBC News are presented in TAB1. As presented in the table, no further improvement against models with only one objective is shown. In our understanding, the inverse of the linear decoder in generative objective behaves similarly to the linear encoder in the discriminative objective, which is presented in Table 6 in the main paper. Therefore, combining two objectives doesn't perform better than only one of them. The number of parameters of each of the selected models is:1. Ours: ≈ 8.8M2. Quick-thought BID33: ≈ 19.8M3. Skip-thought BID28: ≈ 57.7M

represent the tldr for finding the corresponding scientific article.

Multi-view learning improves unsupervised sentence representation learning

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: There are myriad kinds of segmentation, and ultimately the `"right" segmentation of a given scene is in the eye of the annotator. Standard approaches require large amounts of labeled data to learn just one particular kind of segmentation. As a first step towards relieving this annotation burden, we propose the problem of guided segmentation: given varying amounts of pixel-wise labels, segment unannotated pixels by propagating supervision locally (within an image) and non-locally (across images). We propose guided networks, which extract a latent task representation---guidance---from variable amounts and classes (categories, instances, etc.) of pixel supervision and optimize our architecture end-to-end for fast, accurate, and data-efficient segmentation by meta-learning. To span the few-shot and many-shot learning regimes, we examine guidance from as little as one pixel per concept to as much as 1000+ images, and compare to full gradient optimization at both extremes. To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances. Our segmentor concentrates different amounts of supervision of different types of classes into an efficient latent representation, non-locally propagates this supervision across images, and can be updated quickly and cumulatively when given more supervision. Many tasks of scientific and practical interest require grouping pixels, such as cellular microscopy, medical imaging, and graphic design. Furthermore, a single image might need to be segmented in several ways, for instance to first segment all people, then focus on a single person, and finally pick out their face. Learning a particular type of segmentation, or even extending an existing model to a new task like a new semantic class, generally requires collecting and annotating a large amount of data and (re-)training a large model for many iterations. Interactive segmentation with a supervisor in-the-loop can cope with less supervision, but requires at least a little annotation for each image, entailing significant effort over image collections or videos. Faced with endless varieties of segmentation and countless images, yet only so much expertise and time, a segmentor should be able to learn from varying amounts of supervision and propagate that supervision to unlabeled pixels and images. We frame these needs as the problem of guided segmentation: given supervision from few or many images and pixels, collect and propagate this supervision to segment any given images, and do so quickly and with generality across tasks. The amount of supervision may vary widely, from a lone annotated pixel, millions of pixels in a fully annotated image, or even more across a collection of images as in conventional supervised learning for segmentation. The number of classes to be segmented may also vary depending on the task, such as when segmenting categories like cats vs. dogs, or when segmenting instances to group individual people. Guided segmentation extends fewshot learning to the structured output setting, and the non-episodic accumulation of supervision as data is progressively annotated. Guided segmentation broadens the scope of interactive segmentation by integrating supervision across images and segmenting unannotated images. As a first step towards solving this novel problem, we propose guided networks to extract guidance, a latent task representation, from variable amounts of supervision (see Figure 1). To do so we meta-learn how to extract and follow guidance by training episodically on tasks synthesized from a large, fully annotated dataset. Once trained, our model can quickly and cumulatively incorporate annotations to perform new tasks not seen during training. Guided networks reconcile static and interactive modes of inference: a guided model is both able to make predictions on its own, like a fully supervised model, and to incorporate expert supervision for defining new tasks or correcting errors, Figure 1: A guide g extracts a latent task representation z from an annotated image (red) for inference by f θ (x, z) on a different, unannotated image (blue). like an interactive model. Guidance, unlike static model parameters, does not require optimization to update: it can be quickly extended or corrected during inference. Unlike annotations, guidance is latent and low-dimensional: it can be collected and propagated across images and episodes for inference without the supervisor in-the-loop as needed by interactive models. We evaluate our method on a variety of challenging segmentation problems in Section 5: interactive image segmentation, semantic segmentation, video object segmentation, and real-time interactive video segmentation, as shown in 2. We further perform novel exploratory experiments aimed at understanding the characteristics and limits of guidance. We compare guidance with standard supervised learning across the few-shot and many-shot extremes of support size to identify the boundary between few-shot and many-shot learning for segmentation. We demonstrate that in some cases, our model can generalize to guide tasks at a different level of granularity, such as meta-learning from instance supervision and then guiding semantic segmentation of categories. Guided segmentation extends few-shot learning to structured output models, statistically dependent data, and variable supervision in amount of annotation (shot) and numbers of classes (way). Guided segmentation spans different kinds of segmentation as special cases determined by the supervision that constitutes a task, such as a collection of category masks for semantic segmentation, sparse positive and negative pixels in an image for interactive segmentation, or a partial annotation of an object on the first frame of a clip for video object segmentation. Few-shot learning Few-shot learning BID7 BID15 holds the promise of data efficiency: in the extreme case, one-shot learning requires only a single annotation of a new concept. The present wave of methods BID14 BID22 BID27 BID28 BID1 BID8 BID20 BID26 frame it as direct optimization for the few-shot setting: they synthesize episodes by sampling supports and queries, define a task loss, and learn a task model for inference of the queries given the support supervision. While these works address a setting with a fixed, small number of examples and classes at meta-test time, we explore settings where the number of annotations and classes is flexible. Our approach is most closely related to episodically optimized metric learning approaches. We design a novel, efficient segmentation architecture for metric learning, inspired by Siamese networks BID5 BID11 and few-shot metric methods BID14 BID27 BID26 that learn a distance to retrieve support annotations for the query. In contrast to existing meta-learning schemes, we examine how a meta-learned model generalizes across task families with a nested structure, such as performing semantic segmentation after meta-learning on instance segmentation tasks. Segmentation There are many kinds of segmentation, and many current directions for deep learning techniques BID10. We take up semantic BID6 BID17, interactive BID13 BID2, and semi-supervised video object segmentation as challenge problems for our unified view with guidance. See FIG0 for summaries of these tasks. For semantic segmentation BID23 proposes a one-shot segmentor (OSLSM), which requires few but densely annotated images, and must independently infer one annotation and class at a time. Our guided segmentor can segment from sparsely annotated pixels and perform multi-way inference. For video object segmentation one-shot video object segmentation (OSVOS) by achieve high accuracy by fine-tuning during inference, but this online optimization is too costly in time and fails with sparse annotations. Our guided segmentor is feed-forward, hence quick, and segments more accurately from extremely sparse annotations. BID4 impressively achieve state-of-the-art accuracy and real-time, interactive video object segmentation by replacing online optimization with offline metric learning and nearest neighbor inference on a deep, spatiotemporal embedding; however, they focus exclusively on video segmentation. We consider a variety of segmentation tasks, and investigate how guidance transfers across semantic and instance tasks and how it scales with more annotation. For interactive segmentation, BID29; BID18 learn state-of-the-art interactive object segmentation, and BID18 only needs four annotations per object. However, these purely interactive methods infer each task in isolation and cannot pool supervision across tasks and images without optimization, while our guided segmentor quickly propagates supervision non-locally between images. Akin to few-shot learning, we divide the input into an annotated support, which supervises the task to be done, and an unannotated query on which to do the task. The common setting in which the support contains K distinct classes and S examples of each is referred to as K-way, S-shot learning BID15 BID7 BID27. For guided segmentation tasks we add a further pixel dimension to this setting, as we must now consider the number of support pixel annotations for each image, as well as the number of annotated support images. We denote the number of annotated pixels per image as P, and consider the settings of (S, P)-shot learning for various S and P. In particular, we focus on sparse annotations where P is small, as these are more practical to collect, and merely require the annotator to point to the segment(s) of interest. This type of data collection is more efficient than collecting dense masks by at least an order of magnitude BID0. Since segmentation commonly has imbalanced classes and sparse annotations, we consider mixed-shot and semi-supervised supports where the shot varies by class and some points are unlabeled. This is in contrast to the standard few-shot assumption of class-balanced supports. We define a guided segmentation task as the set of input-output pairs (T i, Y i) sampled from a task distribution P, adopting and extending the notation of BID9. The task inputs are DISPLAYFORM0 where S is the number of annotated support images x s, Q is the number of unannotated query images x q, and L s are the support annotations. The annotations are sets of point-label pairs (p, l) with |L s | = P per image, where every label l is one of the K classes or unknown (∅). The task outputs, that is the targets for the support-defined segmentation task on the queries, are DISPLAYFORM1 Our model handles general way K, but for exposition we focus on binary tasks with K = 2, or L = (+, −). We let Q = 1 throughout as inference of each query is independent in our model. Our approach to guided segmentation has two parts: extracting a task representation from the semi-supervised, structured support and segmenting the query given the task representation. We define the task representation as z = g(x, +, −), and the query segmentation guided by that representation asŷ = f (x, z). The design of the task representation z and its encoder g is crucial for guided segmentation to handle the hierarchical structure of images and pixels, the high and variable dimensions of images and their pixelwise annotations, the semi-supervised nature of support with many unannotated pixels, and skewed support distributions. We examine how to best design the guide g and inference f as deep networks. Our method is one part architecture and one part optimization. For architecture, we define branched fully convolutional networks, with a guide branch for extracting the task representation from the support with a novel late fusion technique (Section 4.1), and an inference branch for segmenting queries given the guidance (Section 4.2). For optimization, we adapt episodic meta-learning to image-to-image learning for structured output (Section 4.3), and increase the diversity of episodes past existing practice by sampling within and across segmentation task families like categories and instances. The task representation z must fuse the visual information from the image with the annotations in order to determine what should be segmented in the query. As images with (partial) segmentations, our support is statistically dependent because pixels are spatially correlated, semi-supervised because the full supervision is arduous to annotate, and high dimensional and class-skewed because scenes are sizable and complicated. For simplicity, we first consider a binary task with (1, P)-shot support consisting of one image with an arbitrary number of annotated pixels P, and then extend to multi-way tasks and general (S, P)-shot support. To begin we decompose the support encoder g(x s, + s, − s) across receptive fields indexed by i for local task representations z i = g(x si, + si, − si); this is the same independence assumption made by existing fully convolutional approaches to structured output. See FIG1 for an overview and our novel late global fusion technique. Early Fusion (prior work) Stacking the image and annotations channel-wise at the input makes z si = g early (x, +, −) = φ S (x ⊕ + ⊕ −) with a support feature extractor φ S. This early fusion strategy, employed by BID29, gives end-to-end learning full control of how to fuse. Masking the image by the positive pixels BID23 BID30 instead forces invariance to context, potentially speeding up learning, but precludes learning from the and disturbs input statistics. All early fusion techniques suffer from an inherent modeling issue: incompatibility of the support and query representations. Stacking requires distinct φ S, φ Q while masking disturbs the input distribution. Early fusion is slow, since changes in annotations trigger a full pass through the network, and only one task can be inferred at a time, limiting existing interactive and few-shot segmentors alike BID29 BID18 BID23.Late Fusion (ours) We resolve the learning and inference issues of early fusion by factorizing features and annotations in the guide architecture as z si = g late (x, +, −) = ψ(φ(x), m(+), m(−)). We first extract visual features from the image alone by φ(x), map the annotations into masks in the feature layer coordinates m(+), m(−), and then fuse both by ψ chosen to be element-wise product. This factorization into visual and annotation branches defines the spatial relationship between image and annotations, improving learning sample efficiency and inference computation time. Fixing m to interpolation and ψ to multiplication, the task representation can be updated quickly by only recomputing the masking and not features φ. See FIG1 (center). We do not model a distribution over z, although this is a possible extension of our work for regularization or sampling diverse segmentations. Our late fusion architecture can now share the feature extractor φ for joint optimization through the support and query. Sharing improves learning efficiency with convergence in fewer iterations and task accuracy with 60% relative improvement for video object segmentation. Late fusion reduces inference time, as only the masking needs to be recomputed to incorporate new annotations, making it capable of real-time interactive video segmentation. Optimization-based methods need seconds or minutes to update. Locality We are generally interested in segmentation tasks that are determined by visual characteristics and not absolute location in space or time, i.e. the task is to group pixels of an object and not pixels in the bottom-left of an image. When the support and query images differ, there is no known spatial correspondence, and the only mapping between support and query should be through features. To fit the architecture to this task structure, we merge the local task representations by m P ({z si : ∀i}) for all positions i. Choosing global pooling for m P globalizes the task by discarding the spatial dimensions. The pooling step can be done by averaging, our choice, or other reductions. The effect of pooling in an image with multiple visually similar objects is shown in FIG1 (right).Multi-Shot and Multi-Way The full (S, P)-shot setting requires summarizing the entire support with a variable number of images with varying amounts of pixelwise annotations. Note in this case that the annotations might be divided across the support, for instance one frame of a video may only have positives while a different frame has only negatives, so S-shot cannot always be reduced to 1-shot, as done in prior work BID23. We form the full task representation z S = m S ({z 1, . . ., z S}) simply and differentiably by averaging the shot-wise representations z s. While we have considered binary tasks thus far, we extend guidance to multi-way inference do in our experiments. We construct a separate guide for each class, averaging across all shots containing annotations for that class. Note that all the guides share φ for efficiency and differ only in the masking. Inference in a static segmentation model is simplyŷ = f θ (x) for output y, parameters θ, and input x. Guided inference is a functionŷ = f (x, z) of the query given the guidance extracted from the support. We further structure inference by f (φ(x), z), where φ is a fully convolutional encoder from input pixels to visual features. Multiple forms of conditioning are possible and have been explored for low-dimensional classification and regression problems by the few-shot learning literature. In preliminary experiments we consider parameter regression, nearest neighbor and prototype retrieval, and metric learning on fused features. We select metric learning with feature fusion because it was simple and robust to optimize. Note that feature fusion is similar to siamese architectures, but we directly optimize the classification loss rather than a contrastive loss. In particular we fuse features by m f = φ(x) ⊕ tile(z) which concatenates the guide with the query features, while tiling z to the spatial dimensions of the query. The fused query-support feature is then scored by a small convolutional network f θ that can be interpreted as a learned distance metric for retrieval from support to query. For multi-way guidance, the fusions of the query and each guide are batched for parallel inference. We distinguish between optimizing the parameters of the model during training (learning) and adapting the model during inference (guidance). Thus during training, we wish to "learn to guide." In standard supervised learning, the model parameters θ are optimized according to the loss between predictionŷ = f θ (x) and target y. We reduce the problem of learning to guide to supervised learning by jointly optimizing the parameters of the guidance branch g and the segmentation branch f according to the loss between f θ (x, z) and query target y, see FIG2. For clarity, we distinguish between tasks, a given support and query for segmentation, and task distributions that define a kind of segmentation problem. For example, semantic segmentation is a task distribution while segmenting birds (a semantic class) is a task. We train a guided network for each task distribution by optimizing episodically on sampled tasks. The supports and queries that comprise an episode are synthesized from a fully labeled dataset. We first sample a task, then a subset of images containing that task which we divide into support and query. During training, the target for the query image is available, while for testing it is not. We binarize support and query annotations to encode the task, and spatially sample support annotations for sparsity. Given inputs and targets, we train the network with the pixelwise cross-entropy loss between the predicted and target segmentation of the query. See Sections 7.1 and 7.2 for more details on data processing and network optimization respectively. After learning, the model parameters are fixed, and task inference is determined by guidance. While we evaluate for varying support size S, as described in 4.2, we train with S = 1 for efficiency while sampling P ∼ Uniform. Once learned, our guided networks can operate at different (S, P) shots to address sparse and dense pixelwise annotations with the same model, unlike existing methods that train for particular shot and way. In our experiments, we train with tasks sampled from a single task distribution, but co-or cross-supervision of distributions is possible. Intriguingly, we see some transfer between distributions when evaluating a guided network on a different distribution than it was trained on in Section 5.3. We evaluate our guided segmentor on a variety of problems that are representative of segmentation as a whole: interactive segmentation, semantic segmentation, and video object segmentation. These are conventionally regarded as separate problems, but we demonstrate that each can be viewed as an instantiation of guided segmentation. As a further demonstration of our method, we present for real-time, interactive video segmentation from dot annotations. To better understand the characteristics of guidance, we experiment with cross-task supervision in Section 5.2 and guiding with large-scale supports in Section 5.3.To standardize evaluation we select one metric for all tasks: the intersection-over-union (IU) of the positives averaged across all tasks and masks. This choice allows us to compare scores across the different kinds of segmentation we consider without skew from varying numbers of classes or images. Note that this metric is not equivalent to the mean IU across classes that is commonly reported for semantic segmentation. Please refer to Section 7.3 for more detail. We include fine-tuning and foreground- segmentation as baselines for all problems. Finetuning simply attempts to optimize the model on the support. Foreground- verifies that methods are learning to co-vary their output with the support supervision and sets an accuracy floor. The backbone of our networks is VGG-16 BID25, pre-trained on ILSVRC BID21, and cast into fully convolutional form BID24. This choice is made for fair comparison with existing works across our challenge tasks of semantic, interactive, and video object segmentation without confounds of architecture, pre-training data, and so forth. Interactive Image Segmentation We recover this problem as a special case of guided segmentation when the support and query images are identical. We evaluate on PASCAL VOC BID6 and compare to deep interactive object selection (DIOS) BID29, because it is state-of-the-art and shares our focus on learning for label efficiency and generality. Our approach differs in support encoding: DIOS fuses early while we fuse late and globally. Our guided segmentor is more accurate with extreme sparsity and intrinsically faster to update, as DIOS must do a full forward pass. See Figure 5 (left). From this we decide on late-global guidance throughout. Video Object Segmentation We evaluate our guided segmentor on the DAVIS 2017 benchmark of 2-3 second videos. For this problem, the object indicated by the fully annotated first frame must be segmented across the video. We then extend the benchmark to sparse annotations to gauge how methods degrade. We compare to OSVOS, a state-ofthe-art online optimization method that fine-tunes on the annotated frame and then segments the video frame-by-frame. While BID4 presents impressive on this task and on real-time interactive video segmentation without optimization, their scope is limited to video, and they employ orthogonal improvements that make comparison difficult. We were unable to reproduce their in our own experimental framework. See Figure 6 (left) for a comparison of accuracy, speed, and annotation sparsity. In the dense regime our method achieves 33.3% accuracy for 80% relative improvement over OSVOS in the same time envelope. Given (much) more time fine-tuning significantly improves in accuracy, but takes 10+min/video. Guidance is ∼ 200× faster at 3sec/video. Our method handles extreme sparsity with little degradation, maintaining 87% of the dense accuracy with only 5 points for positive and negative. Fine-tuning struggles to optimize over so few annotations. Interactive Video Segmentation By dividing guidance and inference, our guided segmentor can interactively segment video in real time. As an initial evaluation, we simulate interactions with randomly-sampled dot annotations. We define a benchmark by fixing the amount of annotation and measuring accuracy as the annotations are given. The accuracy-annotation tradeoff curve is plotted in Figure 6 (right). Our guided segmentor improves with both dimensions of shot, whether images (S) or pixels (P). Our guided architecture is feedforward and fast, and faster still to update for changes to the annotations. Semantic Segmentation Semantic segmentation is a challenge for learning from little data due to the high intra-class variance of appearance. For this problem it is crucial to evaluate on not only held-out inputs, but held-out classes, to be certain the guided learner has not covertly learned to be an unguided semantic segmentor. To do so we follow the experimental protocol of BID23 and score by averaging across four class-wise splits of PASCAL VOC BID6, with has 21 classes (including ), and compare to OSLSM.Our approach achieves state-of-the-art sparse that rival the most accurate dense with just two labeled pixels: see Figure 5 (right). OSLSM is incompatible with missing annotations, as it does early fusion by masking, and so is only defined for {0, 1} annotations. To evaluate it we map all missing annotations to negative. Foreground- is a strong baseline, and we were unable to improve on it with fine-tuning. The oracle is trained on all classes (nothing is held-out). (1 We carry out a novel examination of meta-learning with cross-task supervision. In the language of task distributions, the distribution of instance tasks for a given semantic category are nested in the distribution of tasks for that category. We investigate whether meta-training on the sub-tasks (instances) can address the super-tasks (classes). This tests whether guidance can capture an enumerative definition of a semantic class as the union of instances in that category. To do so, we meta-train our guided segmentor on interactive instance segmentation tasks draw from all classes of PASCAL VOC BID6, and then evaluate the model on semantic segmentation tasks from all categories. We experiment with (S, 1) support from semantic annotations, where S varies from one image to all the images in the training set, shown in the plot to the right. We compare to foreground as a class-agnostic accuracy floor, and a standard semantic segmentation net trained with semantic labels as an oracle. Increasing the amount of semantic annotations for guidance steadily increases accuracy. Thus far we have considered guidance in a variable but constrained scale of annotations, ranging from a single pixel in a single image to a few fully annotated images. We meta-learned our guided networks over episodes with such support sizes, and they perform accordingly well in this regime. Here we consider a much wider spectrum of support sizes, with the goal of understanding how guidance compares to standard supervised learning at both ends of the spectrum. To the best of our knowledge, this is the first evaluation of how few-shot learning scales to many-shot usage for structured output. For this experiment we compare guidance and supervised learning on a transfer task between disjoint semantic categories. We take the classes of PASCAL VOC BID6 as source classes, and take the non-intersecting classes of COCO BID16 as the target classes. We divide COCO 2017 validation into class-balanced train/test halves to look at transfer from a practical amount of annotation (thousands instead of more than a hundred thousand images). Our guided segmentor is meta-trained with semantic tasks sampled from the source classes, then guided with 5,989 densely annotated semantic masks from the target classes. For fair comparison, the supervised learner is first trained on the source classes, and then fine-tuned on the same annotated target data. Both methods share the same ILSVRC pre-training, backbone architecture, and (approximate) number of parameters. In this many-shot regime, guidance achieves 95% of supervised learning performance. A key point of this is to shed light on the spectrum of supervision that spans few-shot and many-shot settings, and encourage future work to explore bridging the two. Guided segmentation unifies annotation-bound segmentation problems. Guided networks reconcile task-driven and interactive inference by extracting guidance, a latent task representation, from any amount of supervision given. With guidance our segmentor revolver can learn and infer tasks without optimization, improve its accuracy near-instantly with more supervision, and once-guided can segment new images without the supervisor in the loop. 7.1 DATA PREPARATION Semantic Segmentation and Interactive Segmentation on PASCAL We use PASCAL VOC 2012 BID6 with the additional annotations of SBDD BID12. We define the training set to be the union of the VOC and SBDD training sets, and take the validation set to be the union of VOC and SBDD validation sets, excluding the images in VOC val that overlap with SBDD train. We sparsify the dense masks with random sampling, which we found ed in performance about equal to the more complex sampling strategies of BID29. Thus for a given P, we sample P points randomly from each of the objects to be segmented, as well as the . Labels for classes or instances that are not part of the task are relabeled to . The process of sampling a task and sparsifying and remapping the ground truth labels is illustrated in FIG2.For few-shot semantic segmentation, we follow the experimental protocol of BID23. We test few-shot performance on held-out classes by dividing the 20 classes of PASCAL into 4 sets of 5 classes. Images that contain both held-out and training classes are placed in the held-out set. We subsample splits with more images to ensure that each split contains the same number of images. For each split, we meta-train a guided segmentor with binary tasks sampled from the 15 training classes. We then compute the average performance across 1000 binary tasks sampled from the 5 held-out classes, and average across all four splits. We use the DAVIS 2017 benchmark of 2-3s video clips. We meta-train on the training videos and report average performance on the validation videos. During training, we synthesize tasks by sampling any two frames from the same video and treating one as the support and the other as the query. During testing, the support consists of all labeled frames, while the remaining frames comprise the query. For the video object segmentation benchmark, the first frame is densely labeled. For interactive video segmentation, varying numbers of frames are labeled with varying numbers of pixelwise labels. The backbone of our guided networks as well as our baseline networks is VGG-16 BID25, pre-trained on ILSVRC BID21, and cast into fully convolutional form BID24. We largely follow the optimization procedure for FCNs detailed in BID24: we optimize our guided nets by SGD with a learning rate of 1e −5, batch size 1, high momentum 0.99, and weight decay of 5e −4. The interpolation weights in the decoder are fixed to bilinear and not learned. Note that we normalize the loss by the number of pixels in each image in order to simplify learning rate selection across different datasets with varying image dimensions. Intersection-over-union (IU) is a standard metric for segmentation, but different families of segmentation tasks choose different forms of the metric. We report the IU of positives averaged across all tasks and masks, defined as i tpi i tpi+f pi+f ni where i ranges over ground truth segment masks. This choice makes performance comparable across tasks, because it is independent of the number of classes. We choose not to include negatives in the metric because it adds no information, given the binary nature of the scoring, even for multi-class predictions and ground truth since these are handled as a set of binary tasks by the metric. Note that this metric is not directly comparable to the mean IU across classes typically reported for semantic segmentation benchmarks. As a point of comparison, the 0.62 mean IU achieved by FCN-32s on the PASCAL segmentation benchmark corresponds to 0.45 positive IU.

Represent the short summary for finding the corresponding scientific article (long)

We propose a meta-learning approach for guiding visual segmentation tasks from varying amounts of supervision.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.


Question: Represent the input.

Question: Training generative adversarial networks requires balancing of delicate adversarial dynamics. Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes. In this work, we introduce a new form of latent optimisation inspired by the CS-GAN and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator. We develop supporting theoretical analysis from the perspectives of differentiable games and stochastic approximation. Our experiments demonstrate that latent optimisation can significantly improve GAN training, obtaining state-of-the-art performance for the ImageNet (128 x 128) dataset. Our model achieves an Inception Score (IS) of 148 and an Frechet Inception Distance (FID) of 3.4, an improvement of 17% and 32% in IS and FID respectively, compared with the baseline BigGAN-deep model with the same architecture and number of parameters. Generative Adversarial Nets (GANs) are implicit generative models that can be trained to match a given data distribution. GANs were originally proposed and demonstrated for images by. As the field of generative modelling has advanced, GANs have remained at the frontier, generating high-fidelity images at large scale . However, despite growing insights into the dynamics of GAN training, most recent advances in large-scale image generation come from architectural improvements , or regularisation focusing on particular parts of the model . Inspired by the compressed sensing GAN (CS-GAN;), we further exploit the benefit of latent optimisation in adversarial games using natural gradient descent to optimise the latent variable z at each step of training, presenting a scalable and easy to implement approach to improve the dynamical interaction between the discriminator and the generator. For clarity, we unify these approaches as latent optimised GANs (LOGAN). To summarise our contributions: 1. We present a novel analysis of latent optimisation in GANs from the perspective of differentiable games and stochastic approximation , arguing that latent optimisation can improve the dynamics of adversarial training. 2. Motivated by this analysis, we improve latent optimisation by taking advantage of efficient second-order updates. 3. Our algorithm improves the state-of-the-art BigGAN-deep model by a significant margin, without introducing any architectural change or additional parameters, ing in higher quality images and more diverse samples (Figure 1 and 2). We use θ D and θ G to denote the vectors representing parameters of the generator and discriminator. We use x for images, and z for the latent source generating an image. The prime is used to denote. p(x) and p(z) denote the data distribution and source distribution respectively. E p(x) [f (x)] indicates taking the expectation of function f (x) over the distribution p(x). A GAN consists of a generator that generates image x = G(z; θ G) from a latent source z ∼ p(z), and a discriminator that scores the generated images as D(x; θ D) . Training GANs involves an adversarial game: while the discriminator tries to distinguish generated samples x = G (z; θ G) from data x ∼ p(x), the generator tries to fool the discriminator. This procedure can be summarised as the following min-max game: The exact form of h(·) depends on the choice of loss function (; ;). To simplify our presentation and analysis, we use the Wasserstein loss , so that h D (t) = −t and h G (t) = t. Our experiments with BigGANdeep uses the hinge loss , which is identical to this form in its linear regime. Our analysis can be generalised to other losses as in previous theoretical work (e.g., Arora et al. 2017). To simplify notation, we abbreviate, which may be further simplified as f (z) when the explicit dependency on θ D and θ G can be omitted. Training GANs requires carefully balancing updates to D and G, and is sensitive to both architecture and algorithm choices . A recent milestone is BigGAN (and BigGAN-deep, Brock et al. 2018), which pushed the boundary of high fidelity image generation by scaling up GANs to an unprecedented level. BigGANs use an architecture based on residual blocks , in combination with regularisation mechanisms and self-attention (; ;). Here we aim to improve the adversarial dynamics during training. We focus on the second term in eq. 1 which is at the heart of the min-max game, with adversarial losses for D and G, which can be written as Computing the gradients with respect to θ D and θ G obtains the following gradient, which cannot be expressed as the gradient of any single function : The fact that g is not the gradient of a function implies that gradient updates in GANs can exhibit cycling behaviour which can slow down or prevent convergence. , vector fields of this form are referred to as the simultaneous gradient. Although many GAN models use alternating update rules (e.g., Goodfellow et al. 2014; Brock et al. 2018), following the gradient with respect to θ D and θ G alternatively in each step, they can still suffer from cycling, so we use the simpler simultaneous gradient (eq. 3) for our analysis. Inspired by compressed sensing , introduced latent optimisation for GANs. We call this type of model latent-optimised GANs (LOGAN). Latent optimization has been shown to improve the stability of training as well as the final performance for medium-sized models such as DCGANs and Spectral Normalised GANs . Latent optimisation exploits knowledge from D to guide updates of z. Intuitively, the gradient ∂z points in the direction that satisfies the discriminator D, which implies better samples. Therefore, instead of using the randomly sampled z ∼ p(z), uses the optimised latent in eq. 1 for training 1. The general algorithm is summarised in Algorithm 1 and illustrated in Figure 3 a. We develop the natural gradient descent form of latent update in Section 4. and use it to obtain ∆z from eq. 4 (GD) or eq. 12 (NGD) Optimise the latent z ← [z + ∆z], [·] indicates clipping the value between −1 and 1 Compute generator loss L Update θ D and θ G with the gradients until reaches the maximum training steps 3 ANALYSIS OF THE ALGORITHM To understand how latent optimisation improves GAN training, we analyse LOGAN as a 2-player differentiable game following;;. The appendix provides a complementary analysis that relates LOGAN to unrolled GANs and stochastic approximation . An important problem with gradient-based optimization in GANs is that the vector-field generated by the losses of the discriminator and generator is not a gradient vector field. It follows that gradient descent is not guaranteed to find a local optimum and can cycle, which can slow down convergence or lead to phenomena like mode collapse and mode hopping.; proposed Symplectic Gradient Adjustment (SGA) to improve the dynamics of gradient-based methods in adversarial games. For a game with gradient g (eq. 3), define the Hessian as the second order derivatives with respect to the parameters, H = ∇ θ g. SGA uses the adjusted gradient g * = g + λ A T g where λ is a positive constant and A = 1 2 (H − H T) is the anti-symmetric component of the Hessian. Applying SGA to GANs yields the adjusted updates (see Appendix B.1 for details): Compared with g in eq. 3, the adjusted gradient g * has second-order terms reflecting the interactions between D and G. SGA has been shown to significantly improve GAN training in basic examples , allowing faster and more robust convergence to stable fixed points (local Nash equilibria). Unfortunately, SGA is expensive to scale because computing the second-order derivatives with respect to all parameters is expensive. Explicitly computing the gradients for the discriminator and generator at z after one step of latent optimisation (eq. 4) obtains In both equations, the first terms represent how f (z) depends on the parameters directly and the second terms represent how f (z) depends on the parameters via the optimised latent source. For the second equality, we substitute ∆z = α ∂z as the gradient-based update of z and use ∂z. The original GAN's gradient (eq. 3) does not include any second-order term, since ∆z = 0 without latent optimisation. In LOGAN, these extra terms are computed by automatic differentiation when back-propagating through the latent optimisation process (see Algorithm 1). The SGA updates in eq. 6 and the LOGAN updates in eq. 8 are strikingly similar, suggesting that the latent step used by LOGAN reduces the negative effects of cycling by introducing a symplectic gradient adjustment into the optimization procedure. The role of the latent step can be formalized in terms of a third player, whose goal is to help the generator, see appendix B for details. Crucially, latent optimisation approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second order terms involving parameters of both the discriminator and the generator -which are extremely expensive to compute -are not used. For latent z's with dimensions typically used in GANs (e.g., 128-256, orders of magnitude less than the number of parameters), these can be computed efficiently. In short, latent optimisation efficiently couples the gradients of the discriminator and generator, as prescribed by SGA, but using the much lower-dimensional latent source z which makes the adjustment scalable. An important consequence of reducing the rotational aspect of GAN dynamics is that it is possible to use larger step sizes during training which suggests using stronger optimisers to fully take advantage of latent optimisation. Latent optimisation can improve GAN training dynamics further by allowing larger single steps ∆z towards the direction of ∂f (z) ∂z without overshooting. Appendix B further explains how LOGAN relates to unrolled GANs and stochastic approximation. Our main finding is that latent optimisation accelerates the speed of updating D relative to that of G, facilitating convergence according to In particular, the generator requires less update compared with D to achieve the same reduction of loss, because latent optimisation "helps" G. Although our analysis suggests using strong optimisers for optimising z, only used basic gradient descent (GD) with a fixed step-size. This choice limits the size ∆z can take: in order not to overshoot when the curvature is large, the step size would be too conservative when the curvature is small. We hypothesis that GD is more detrimental for larger models, which have complex loss surfaces with highly varying curvatures. Consistent with this hypothesis, we observed only marginal improvement over the baseline using GD (section 5.3, Table In this work, we instead use natural gradient descent (NGD, Amari 1998) for latent optimisation. NGD can be seen as an approximate second-order optimisation method , and has been applied successfully in many domains. By using the positive semidefinite (PSD) Gauss-Newton matrix to approximate the (possibly negative definite) Hessian, NGD often works even better than exact second-order methods. NGD is expensive in high dimensional parameter spaces, even with approximations . However, we demonstrate it is efficient for latent optimisation, even in very large models. Given the gradient of z, g = ∂f (z) ∂z, NGD computes the update as where the Fisher information matrix F is defined as The log-likelihood function ln p(t|z) typically corresponds to commonly used error functions such as cross entropy loss. This correspondence is not necessary when NGD is interpreted as an approximate second-order method, as has long been used in practice . Nevertheless, Appendix C provides a Poisson log-likelihood interpretation for the hinge loss commonly used in GANs . An important difference between latent optimisation and commonly seen senarios using NGD is that the expectation over the condition (z) is absent. Since each z is only responsible for generating one image, it only minimises the loss L G (z) for this particular instance. Computing per-sample Fisher this way is necessary to approximate SGA (see Appendix B.1 for details). More specifically, we use the empirical Fisher F with Tikhonov damping, as in TONGA F = g · g T + β I F is cheaper to compute compared with the full Fisher, since g is already available. The damping factor β regularises the step size, which is important when F only poorly approximates the Hessian or when the Hessian changes too much across the step. Using the Sherman-Morrison formula, the NGD update can be simplified into the following closed form: which does not involve any matrix inversion. Thus, NGD adapts the step size according to the curvature estimate c = 1 Figure 4 a illustrates the scaling for different values of β. NGD automatically smooths the scale of updates by down-scaling the gradients as their norm grows, which also contributes to the smoothed norms of updates (Figure 4 b). Since the NGD update remains proportional to g, our analysis based on gradient descent in section 3 still holds. We focus on large scale GANs based on BigGAN-deep trained on 128 × 128 size images from the ImageNet dataset . In Appendix E, we present from applying our algorithm on Spectral Normalised GANs trained with CIFAR dataset , which obtains state-of-the-art scores on this model. We used the standard BigGAN-deep architecture with three minor modifications: 1. We increased the size of the latent source from 128 to 256, to compensate the randomness of the source lost when optimising z. 2. We use the uniform distribution U(−1, 1) instead of the standard normal distribution N for p(z), to be consistent with the clipping operation (Algorithm 1). 3. We use leaky ReLU instead of ReLU as the non-linearity for smoother gradient flow for ∂z. Consistent with detailed findings in that these changes have limited effect, our experiment with this baseline model obtains only slightly better scores compared with those in (Table 1, see also Figure 8). The FID and IS are computed as in , and IS values are computed from checkpoints with the lowest FIDs. The means and standard deviations are computed from 5 models with different random seeds. To apply latent optimisation, we use a damping factor β = 5.0 combined with a large step size of α = 0.9. As an additional way of damping, we only optimise 50% of z's dimensions. Optimising the entire population of z was unstable in our experiments. Similar to , we found it was helpful to regularise the Euclidean norm of weight-change ∆z, with a regulariser weight of 300.0. All other hyper-parameters, including learning rates and a large batch size of 2048, remain the same as in BigGAN-deep; we did not optimise these hyper-parameters. We call this model LOGAN (NGD). Employing the same architecture and number of parameters as the BigGAN-deep baseline, LOGAN (NGD) achieved better FID and IS (Table 1). As observed by , BigGAN training always eventually collapsed. Training with LOGAN also collapsed, perhaps due to higher-order dynamics beyond the scope we have analysed, but took significantly longer (600k steps versus 300k steps with BigGAN-deep). During training, LOGAN was 2 − 3 times slower per step compared with BigGAN-deep because of the additional forward and backward pass. We found that optimising z during evaluation did not improve sample scores (even up to 10 steps), so we do not optimise z for evaluation. Therefore, LOGAN has the same evaluation cost as original BigGAN-deep. To help understand this behaviour, we plot the change from ∆z during training in Figure 5 a. Although the movement in Euclidean space ∆z grew until training collapsed, the movement in D's output space, measured as f (z + ∆z) − f (z), remained unchanged (see Appendix D for details). As shown in our analysis, optimising z improves the training dynamics, so LOGANs work well after training without requiring latent optimisation. We verify our theoretical analysis in section 3 by examining key components of Algorithm 1 via ablation studies. First, we experimented with using basic GD to optimising z, as in , and call this model LOGAN (GD). A smaller step size of α = 0.0001 was required; larger values were unstable and led to premature collapse of training. As shown in Table 1, the scores from LOGAN (GD) were worse than LOGAN (NGD) and similar to the baseline model. We then evaluate the effects of removing those terms depending on ∂f (z) ∂z in eq. 8, which are not in the ordinary gradient (eq. 3). Since these terms were computed when back-propagating through the latent optimisation procedure, we removed them by selectively blocking back-propagation with "stop gradient" operations (e.g., in TensorFlow Abadi et al. 2016). T ∂f (z) ∂z and removing both terms. As predicted by our analysis (section 3), both terms help stabilise training; training diverged early for all three ablations. Truncation is a technique introduced by to illustrate the trade-off between the FID and IS in a trained model. For a model trained with z ∼ p(z) from a source distribution symmetric around 0, such as the standard normal distribution N and the uniform distribution U(−1, 1), down-scaling (truncating) the sourcez = s · z with 0 ≤ s ≤ 1 gives samples with higher visual quality but reduced diversity. This observation is quantified as higher IS and lower FID when evaluating samples from truncated distributions. Figure 2 b show higher quality compared with those in a (e.g., the interfaces between the elephants and ground, the contours around the pandas). In this work we present the LOGAN model which significantly improves the state-of-the-art on large scale GAN training for image generation by online optimising the latent source z. Our illustrate improvements in quantitative evaluation and samples with higher quality and diversity. Moreover, our analysis suggests that LOGAN fundamentally improves adversarial training dynamics. We therefore expect our method to be useful in other tasks that involve adversarial training, including representation learning and inference , text generation , style learning , audio generation and video generation (; A ADDITIONAL SAMPLES AND Figure 6 and 7 provide additional samples, organised similarly as in Figure 1 and 2. Figure 8 shows additional truncation curves. In this section we present three complementary analyses of LOGAN. In particular, we show how the algorithm brings together ideas from symplectic gradient adjustment, unrolled GANs and stochastic approximation with two time scales. To analyse LOGAN as a differentiable game we treat the latent step ∆z as adding a third player to the original game played by the discriminator and generator. The third player's parameter, ∆z, is optimised online for each z ∼ p(z). Together the three players (latent player, discriminator, and generator) have losses averaged over a batch of samples: where η = 1 N (N is the batch size) reflects the fact that each ∆z is only optimised for a single sample z, so its contribution to the total loss across a batch is small compared with θ D and θ G which are directly optimised for batch losses. This choice of η is essential for the following derivation, and has important practical implication. It means that the per-sample loss L G (z), instead of the loss summed over a batch N n=1 L G (z n), should be the only loss function guiding latent optimisation. Therefore, when using natural gradient descent (Section 4), the Fisher information matrix should only be computed using the current sample z. The ing simultaneous gradient is , we can write the Hessian of the game as: The presence of a non-zero anti-symmetric component in the Hessian implies the dynamics have a rotational component which can cause cycling or slow down convergence. Since η 1 for typical batch sizes (e.g., Symplectic gradient adjustment (SGA) counteracts the rotational force by adding an adjustment term to the gradient to obtain g * ← g + λ A T g, which for the discriminator and generator has the form: The gradient with respect to ∆z is ignored since the convergence of training only depends on θ D and θ G. If we drop the last terms in eq.17 and 18, which are expensive to compute for large models with high-dimensional θ D and θ G, and use ∂z, the adjusted updates can be rewritten as Because of the third player, there are still the terms depend on to adjust the gradients. Efficiently computing is non-trivial (e.g., Pearlmutter 1994). However, if we introduce the local approximation then the adjusted gradient becomes identical to 8 from latent optimisation. In other words, automatic differentiation by commonly used machine learning packages can compute the adjusted gradient for θ D and θ G when back-propagating through the latent optimisation process. Despite the approximation involved in this analysis, both our experiments in section 5 and the from verified that latent optimisation can significantly improve GAN training. Latent optimisation can be seen as unrolling GANs in the space of the latent, rather than the parameters. Unrolling in the latent space has the advantages that: 1. LOGAN is more scalable than Unrolled GANs because it avoids second-order derivatives over a potentially very large number of parameters. 2. While unrolling the update of D only affects the parameters of G (as in Metz et al. 2016), latent optimisation effects both D and G as shown in eq. 8. We next formally present this connection by showing that SGA can be seen as approximating Unrolled GANs . For the update θ D = θ D + ∆θ D, we have the Taylor expansion approximation at θ D: Here p(t = 1; z, D, G) is the probability that the generated image G(z) can fool the discriminator D. The original GAN's discriminator can be interpreted as outputting a Bernoulli distribution In this case, if we parameterise β G = D (G(z)), the generator loss is the negative log-likelihood Bernoulli, however, is not the only valid choice as the discriminator's output distribution. Instead of sampling "1" or "0", we assume that there are many identical discriminators that can independently vote to reject an input sample as fake. The number of votes k in a given interval can be described by a Poisson distribution with parameter λ with the following PMF: The probability that a generated image can fool all the discriminators is the probability of G(z) receiving no vote for rejection Therefore, we have the following negative log-likelihood as the generator loss if we parameterise λ = −D (G(z)): This interpretation has a caveat that when D (G(z)) > 0 the Poisson distribution is not well defined. However, in general the discriminator's hinge loss pushes D (G(z)) < 0 via training. For a temporal sequence x 1, x 2,..., x T (changes of z or f (z) at each training step in this paper), to normalise its variance while accounting for the non-stationarity, we process it as follows. We first compute the moving average and standard deviation over a window of size N: Then normalise the sequence as: The in Figure 5 a is robust to the choice of window size. Our experiments with N from 10 to 50 yielded visually similar plots. To test if latent optimisation works with models at more moderate scales, we applied it on SN-GANs . Although our experiments on this model are less thorough than in the main paper with BigGAN-deep, we hope to provide basic guidelines for researchers interested in applying latent optimisation on smaller models. The experiments follows the same basic setup and hyper-parameter settings as the CS-GAN in. There is no class conditioning in this model. For NGD, we found a smaller damping factor β = 0.1, a z regulariser weight of 3.0 (the same as in Wu et al. 2019), combined with optimising 70% of the latent source (instead of 50% for BigGAN-deep) worked best for SN-GANs. In addition, we found running extra latent optimisation steps benefited evaluation, so we use ten steps of latent optimisation in evaluation for in this section, although the models were still trained with a single optimisation step. We reckon that smaller models might not be "over-parametrised" enough to fully amortise the computation from optimising z, which can then further exploit the architecture in evaluation time. On the other hand, the overhead from running multiple iterations of latent optimisation is relatively small at this scale. We aim to further investigate this difference in future studies. Table 2 shows the FID and IS alongside SN-GAN and CS-CAN which used the same architecture. Here we observe similarly significant improvement over the baseline SN-GAN model, with an improvement of 16.8% in IS and 39.6% in FID. Figure 9 shows random samples from these two models. Overall, samples from LOGAN (NGD) have higher contrasts and sharper contours.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

Latent optimisation improves adversarial training dynamics. We present both theoretical analysis and state-of-the-art image generation with ImageNet 128x128.

represent the tldr for finding the corresponding scientific article

Three factors (batch size, learning rate, gradient noise) change in predictable way the properties (e.g. sharpness) of minima found by SGD.