Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: We present a new latent model of natural images that can be learned on large-scale datasets. The learning process provides a latent embedding for every image in the training dataset, as well as a deep convolutional network that maps the latent space to the image space. After training, the new model provides a strong and universal image prior for a variety of image restoration tasks such as large-hole inpainting, superresolution, and colorization. To model high-resolution natural images, our approach uses latent spaces of very high dimensionality (one to two orders of magnitude higher than previous latent image models). To tackle this high dimensionality, we use latent spaces with a special manifold structure (convolutional manifolds) parameterized by a ConvNet of a certain architecture. In the experiments, we compare the learned latent models with latent models learned by autoencoders, advanced variants of generative adversarial networks, and a strong baseline system using simpler parameterization of the latent space. Our model outperforms the competing approaches over a range of restoration tasks. Learning good image priors is one of the core problems of computer vision and machine learning. One promising approach to obtaining such priors is to learn a deep latent model, where the set of natural images is parameterized by a certain simple-structured set or probabilistic distribution, whereas the complexity of natural images is tackled by a deep ConvNet (often called a generator or a decoder) that maps from the latent space into the space of images. The best known examples are generative adversarial networks (GANs) and autoencoders BID4.Given a good deep latent model, virtually any image restoration task can be solved by finding a latent representation that best corresponds to the image evidence (e.g. the known pixels of an occluded image or a low-resolution image). The attractiveness of such approach is in the universality of the learned image prior. Indeed, applying the model to a new restoration task can be performed by simply changing the likelihood objective. The same latent model can therefore be reused for multiple tasks, and the learning process needs not to know the image degradation process in advance. This is in contrast to task-specific approaches that usually train deep feed-forward ConvNets for individual tasks, and which have a limited ability to generalize across tasks (e.g. a feed-forward network trained for denoising cannot perform large-hole inpainting and vice versa).At the moment, such image restoration approach based on latent models is limited to low-resolution images. E.g. BID16 showed how a latent model trained with GAN can be used to perform inpainting of tightly-cropped 64 × 64 face images. Below, we show that such models trained with GANs cannot generalize to higher resolution (eventhough GAN-based systems are now able to obtain high-quality samples at high resolutions BID9). We argue that it is the limited dimensionality of the latent space in GANs and other existing latent models that precludes them from spanning the space of high-resolution natural images. To scale up latent modeling to high-resolution images, we consider latent models with tens of thousands of latent dimensions (as compared to few hundred latent dimensions in existing works). We show that training such latent models is possible using direct optimization BID1 and that it leads to good image priors that can be used across a broad variety of reconstruction tasks. In previous models, the latent space has a simple structure such as a sphere or a box in a Euclidean space, or a full Euclidean space with a Gaussian prior. Such choice, however, is not viable in our Figure 1: Restorations using the same Latent Convolutional Model (images 2,4,6) for different image degradations (images 1,3,5). At training time, our approach builds a latent model of non-degraded images, and at test time the restoration process simply finds a latent representation that maximizes the likelihood of the corrupted image and outputs a corresponding non-degraded image as a restoration .case, as vectors with tens of thousands of dimensions cannot be easily used as inputs to a generator. Therefore, we consider two alternative parameterizations of a latent space. Firstly, as a baseline, we consider latent spaces parameterized by image stacks (three-dimensional tensors), which allows to have "fully-convolutional" generators with reasonable number of parameters. Our full system uses a more sophisticated parameterization of the latent space, which we call a convolutional manifold, where the elements of the manifold correspond to the parameter vector of a separate ConvNet. Such indirect parameterization of images and image stacks have recently been shown to impose a certain prior BID15, which is beneficial for restoration of natural images. In our case, we show that a similar prior can be used with success to parameterize high-dimensional latent spaces. To sum up, our contributions are as follows. Firstly, we consider the training of deep latent image models with the latent dimensionality that is much higher than previous works, and demonstrate that the ing models provide universal (w.r.t. restoration tasks) image priors. Secondly, we suggest and investigate the convolutional parameterization for the latent spaces of such models, and show the benefits of such parameterization. Our experiments are performed on CelebA BID11 (128x128 resolution), SUN Bedrooms BID17 (256x256 resolution), CelebA-HQ BID9 ) (1024x1024 resolution) datasets, and we demonstrate that the latent models, once trained, can be applied to large hole inpainting, superresolution of very small images, and colorization tasks, outperforming other latent models in our comparisons. To the best of our knowledge, we are the first to demonstrate how "direct" latent modeling of natural images without extra components can be used to solve image restoration problems at these resolutions (Figure 1).Other related work. Deep latent models follow a long line of works on latent image models that goes back at least to the eigenfaces approach BID14. In terms of restoration, a competing and more popular approach are feed-forward networks trained for specific restoration tasks, which have seen rapid progress recently. Our approach does not quite match the quality of e.g. BID6, that is designed and trained specifically for the inpainting task, or the quality of e.g. BID18 that is designed and trained specifically for the face superresolution task. Yet the models trained within our approach (like other latent models) are universal, as they can handle degradations unanticipated at training time. Our work is also related to pre-deep learning ("shallow") methods that learn priors on (potentiallyoverlapping) image patches using maximum likelihood-type objectives such as BID12 BID8 BID21. The use of multiple layers in our method allows to capture much longer correlations. As a , our method can be used successfully to handle restoration tasks that require exploiting these correlations, such as large-hole inpainting. Let {x 1, x 2, . . ., x N} be a set of training images, that are considered to be samples from the distribution X of images in the space X of images of a certain size that need to be modeled. In latent modeling, we introduce a different space Z and a certain distribution Z in that space that is used to re-parameterize X. In previous works, Z is usually chosen to be a Euclidean space with few dozen to few hundred dimensions, while our choice for Z is discussed further below. Figure 2: The Latent Convolutional Model incroprorates two sequential ConvNets. The smaller ConvNet f (red) is fitted to each training image and is effectively used to parameterize the latent manifold. The bigger ConvNet g (magenta) is used as a generator, and its parameters are fitted to all training data. The input s to the pipeline is fixed to a random noise and not updated during training. The deep latent modeling of images implies learning the generator network g θ with learnable parameters θ, which usually has convolutional architecture. The generator network maps from Z to X and in particular is trained so that g θ (Z) ≈ X. Achieving the latter condition is extremely hard, and there are several approaches that can be used. Thus, generative adversarial networks (GANs) train the generator network in parallel with a separate discriminator network that in some variants of GANs serves as an approximate ratio estimator between X and X+g θ (Z) over points in X. Alternatively, autoencoders BID4 and their variational counter-parts BID10 train the generator in parallel with the encoder operating in the reverse direction, ing in a more complex distribution Z. Of these two approaches, only GANs are known to be capable of synthesizing high-resolution images, although such ability comes with additional tricks and modifications of the learning formulation BID9. In this work, we start with a simpler approach to deep latent modeling BID1 known as the GLO model. GLO model optimizes the parameters of the generator network in parallel with the explicit embeddings of the training examples {z 1, z 2, . . ., z N}, such that g θ (z i) ≈ x i by the end of the optimization. Our approach differs from and expands BID1 ) in three ways: (i) we consider a much higher dimensionality of the latent space, (ii) we use an indirect parameterization of the latent space discussed further below, (iii) we demonstrate the applicability of the ing model to a variety of image restoration tasks. Scaling up latent modeling. Relatively low-dimensional latent models of natural images presented in previous works are capable of producing visually-compelling image samples from the distribution BID9, but are not actually capable of matching or covering a rather high-dimensional distribution X. E.g. in our experiments, none of GAN models were capable of reconstructing most samples x from the hold-out set (or even from the training set; this observation is consistent with BID1 and also with BID20). Being unable to reconstruct uncorrupted samples clearly suggests that the learned models are not suitable to perform restoration of corrupted samples. On the other hand, autoencoders and the related GLO latent model BID1 were able to achieve better reconstructions than GAN on the hold-out sets, yet have distinctly blurry reconstructions (even on the training set), suggesting strong underfitting. We posit that existing deep latent models are limited by the dimensionality of the latent space that they consider, and aim to scale up this dimensionality significantly. Simply scaling up the latent dimensionality to few tens of dimensions is not easily feasible, as e.g. the generator network has to work with such a vector as an input, which would make the first fully-connected layer excessively large with hundreds of millions of parameters 1.To achieve a tractable size of the generator, one can consider latent elements z to have a threedimensional tensor structure, i.e. to be stacks of 2D image maps. Such choice of structure is very natural for convolutional architectures, and allows to train "fully-convolutional" generators with the first layer being a standard convolutional operation. The downside of this choice, as we shall see, is that it allows limited coordination between distant parts of the images x = g θ (z) produced by the generator. This drawback is avoided when the latent space is parameterized using latent convolutional manifolds as described next. Latent convolutional manifolds. To impose more appropriate structure on the latent space, we consider structuring these spaces as convolutional manifolds defined as follows. Let s be a stack of maps of the size W s × H s × C s and let {f φ | φ ∈ Φ} be a set of convolutional networks all sharing the same architecture f that transforms s to different maps of size DISPLAYFORM0 Various choices of φ then span a manifold embedded into this space, and we refer to it as the convolutional manifold. A convolutional manifold C f,s is thus defined by the ConvNet architecture f as well as by the choice of the input s (which in our experiments is always chosen to be filled with uniform random noise). Additionally, we also restrict the elements of vectors φ to lie within the [−B; B] range. Formally, the convolutional manifold is defined as the following set: DISPLAYFORM1 where φ serves as a natural parameterization and N φ is the number of network parameters. Below, we refer to f as latent ConvNet, to disambiguate it from the generator g, which also has a convolutional structure. The idea of the convolutional manifold is inspired by the recent work on deep image priors BID15. While they effectively use convolutional manifolds to model natural images directly, in our case, we use them to model the latent space of the generator networks ing in a fully-fledged learnable latent image model (whereas the model in BID15) cannot be learned on a dataset of images). The work BID15 demonstrates that the regularization imposed by the structure of a very high-dimensional convolutional manifold is beneficial when modeling natural images. Our intuition here is that similar regularization would be beneficial in regularizing learning of high-dimensional latent spaces. As our experiments below reveal, this intuition holds true. Learning formulation. Learning the deep latent model (Figure 2) in our framework then amounts to the following optimization task. Given the training examples {x 1, x 2, . . ., x N}, the architecture f of the convolutional manifold, and the architecture g of the generator network, we seek the set of the latent ConvNet parameter vectors {φ 1, φ 2, . . ., φ N} and the parameters of the generator network θ that minimize the following objective: DISPLAYFORM2 with an additional box constraints φ j i ∈ [−0.01; 0.01] and s being a random set of image maps filled with uniform noise. Following BID1, the norm in is taken to be the Laplacian-L1: DISPLAYFORM3 where L j is the jth level of the Laplacian image pyramid BID2. We have also found that adding an extra MSE loss term to the Lap-L1 loss term with the weight of 1.0 speeds up convergence of the models without affecting the by much. The optimization is performed using stochastic gradient descent. As an outcome of the optimization, each training example x i gets a representation z i = f φi on the convolutional manifold C f,s.Importantly, the elements of the convolutional manifold then define a set of images in the image space (which is the image of the convolutional manifold under learned generator): Top Choice for Colorization Figure 3: Results (perceptual metrics -lower is better -and user preferences) for the two datasets (CelebA -left, Bedrooms -right) and three tasks (inpainting, super-resolution, colorization). For the colorization task the perceptual metric is inadequate as the grayscale image has the lowest error, but is shown for completeness. While not all elements of the manifold I f,s,θ will correspond to natural images from the distribution X, we have found out that with few thousand dimensions, the ing manifolds can cover the support of X rather well. I.e. each sample from the image distribution can be approximated by the element of I f,s,θ with a low approximation error. This property can be used to perform all kinds of image restoration tasks. DISPLAYFORM4 Image restoration using learned latent models. We now describe how the learned latent model can be used to perform the restoration of the unknown image x 0 from the distribution X, given some evidence y. Depending on the degradation process, the evidence y can be an image x 0 with masked values (inpainting task), the low-resolution version of x 0 (superresolution task), the grayscale version of x 0 (colorization task), the noisy version of x 0 (denoising task), a certain statistics of x 0 computed e.g. using a deep network (feature inversion task), etc. We further assume, that the degradation process is described by the objective E(x|y), which can be set to minus log-likelihood E(x|y) = − log p(y|x) of observing y as a of the degradation of x. E.g. for the inpainting task, one can use E(x|y) = (x − y) m, where m is the 0-1 mask of known pixels and denotes element-wise product. For the superresolution task, the restoration objective is naturally defined as E(x|y) = ↓ (x) − y, where ↓ (·) is an image downsampling operator (we use Lanczos in the experiments) and y is the low-resolution version of the image. For the colorization task, the objective is defined as E(x|y) = gray(x) − y, where gray(·) denotes a projection from the RGB to grayscale images (we use a simple averaging of the three color channels in the experiments) and y is the grayscale version of the image. Using the learned latent model as a prior, the following estimation combining the learned prior and the provided image evidence is performed: DISPLAYFORM5 In other words, we simply estimate the element of the image manifold that has the highest likelihood. The optimization is performed using stochastic gradient descent over the parameters φ on the latent convolutional manifold. For the baseline models, which use a direct parameterization of the latent space, we perform analogous estimation using optimization in the latent space: DISPLAYFORM6 In the experiments, we compare the performance of our full model and several baseline models over a range of the restoration tasks using formulations and. Datasets. The experiments were conducted on three datasets. The CelebA dataset was obtained by taking the 150K images from BID11 (cropped version) and resizing them from 178×218 to 128 × 128. Note that unlike most other works, we have performed anisotropic rescaling rather than additional cropping, leading to the version of the dataset with larger portions and higher variability (corresponding to a harder modeling task). The Bedrooms dataset from the LSUN BID17 is another popular dataset of images. We rescale all images to the 256 × 256 size. Finally, the CelebA-HQ dataset from BID9 Figure 5: Qualitative comparison on SUN Bedrooms for the tasks of inpainting (rows 1-2), superresolution (rows 3-4), colorization (rows 5-6). The LCM method performs better than most methods for the first two tasks. Tasks. We have compared methods for three diverse tasks. For the inpainting task, we have degraded the input images by masking the center part of the image (50 × 50 for CelebA, 100 × 100 for Bedrooms, 400 × 400 for CelebA-HQ). For the superresolution task, we downsampled the images by a factor of eight. For the colorization task, we have averaged the color channels obtaining the gray version of the image. We have performed extensive comparisons with other latent models on the two datasets with smaller image size and lower training times (CelebA and Bedrooms). The following latent models were compared:• Latent Convolutional Networks (LCM -Ours): Each f φi has 4 layers (in CelebA), 5 layers (in Bedrooms) or 7 layers (in CelebA-HQ) and takes as input random uniform noise. The Generator, g θ has an hourglass architecture. The latent dimensionality of the model was 24k for CelebA and 61k for Bedrooms.• GLO: The baseline model discussed in the end of Section 2 and inspired by BID1, where the generator network has the same architecture as in LCM, but the convolutional space is parameterized by a set of maps. The latent dimensionality is the same as in LCM (and thus much higher than in BID1). We have also tried a variant reproduced exactly from BID1 with vectorial latent spaces that feed into a fully-connected layers (for the dimensionalities ranging from 2048 to 8162 -see Appendix B), but invariably observed underfitting. Generally, we took extra care to find the optimal parameterization that would be most favourable to this baseline.• DIP: The deep image prior-based restoration BID15. We use the architecture proposed by the authors in the paper. DIP can be regarded as an extreme version of our paper with the generator network being an identity. DIP fits 1M parameters to each image for inpainting and colorization and 2M parameters for super-resolution.• GAN: For CelebA we train a WGAN-GP BID5 with the DCGAN type generator and a latent space of 256. For Bedrooms we use the pretrained Progressive GAN (PGAN) models with the latent space of dimensionality 512 published by the authors of BID9. During restoration, we do not impose prior on the norm of z since it worsens the underfitting problem of GANs (as demonstrated in Appendix C).• AE: For the CelebA we have also included a standard autoencoder using the Lap-L1 and MSE reconstruction metrics into the comparison (latent dimensionality 1024). We have also tried the variant with convolutional higher-dimensional latent space, but have observed very strong overfitting. The variational variant (latent dimensionality 1024) lead to stronger underfitting than the non-variational variant. As the experiments on CelebA clearly showed a strong underfitting, we have not included AE into the comparison on the higher-resolution Bedrooms dataset. For Bedrooms dataset we restricted training to the first 200K training samples, except for the DIP (which does not require training) and GAN (we used the progressive GAN model trained on all 3M samples). All comparisons were performed on hold-out sets not used for training. Following BID1, we use plain SGD with very high learning rate of 1.0 to train LCM and of 10.0 to train the GLO models. The exact architectures are given in Appendix D. We have used quantitative and user study-based assessment of the . For the quantitative measure, we have chosen the mean squared error (MSE) measure in pixel space, as well as the mean squared distance of the VGG16-features BID13 between the original and the reconstructed images. Such perceptual metrics are known to be correlated with the human judgement BID7 BID19. We have used the [relu1_2, relu2_2, relu3_3, relu4_3, relu5_3] layers contributing to the distance metric with equal weight. Generally, we observed that the relative performance of the methods were very similar for the MSE measure, for the individual VGG layers, and for the averaged VGG metrics that we report here. When computing the loss for the inpainting task we only considered the positions corresponding to the masked part. Quantitative metrics however have limited relevance for the tasks with big multimodal conditional distributions, i.e. where two very different answers can be equally plausible, such as all three tasks that we consider (e.g. there could be very different colorizations of the same bedroom image).In this situation, human judgement of quality is perhaps the best measure of the algorithm performance. To obtain such judgements, we have performed a user study, where we have picked 10 random images for each of the two datasets and each of the three tasks. The of all compared methods alongside the degraded inputs were shown to the participants (100 for CelebA, 38 for Bedrooms). For each example, each subject was asked to pick the best restoration variant (we asked to take into account both realism and fidelity to the input). The were presented in random order (shuffled independently for each example). We then just report the percentage of user choices for each method for a given task on a given dataset averaged over all subjects and all ten images. Results. The of the comparison are summarized in Figure 3 and TAB2 with representative examples shown in FIG0 and Figure 5. "Traditional" latent models (built WGAN/PGAN and AE) performed poorly. In particular, GAN-based models produced that were both unrealistic and poorly fit the likelihood. Note that during fitting we have not imposed the Gaussian prior on the latent space of GANs. Adding such prior did not in considerable increase of realism and lead to even poorer fit to the evidence (see Appendix C).The DIP model did very well for inpainting and superresolution of relatively unstructured Bedrooms dataset. It however performed very poorly on CelebA due to its inability to learn face structure from data and on the colorization task due to its inability to learn about natural image colors. Except for the Bedrooms-inpainting, the new models with very large latent space produced that were clearly favoured by the users. LCM performed better than GLO in all six user comparisons, while in terms of the perceptual metric the performance of LCM was also better than GLO for inpainting and superresolution tasks. For the colorization task, the LCM is unequivocally better in terms of user preferences, and worse in terms of the perceptual metric. We note that, however, perceptual metric is inadequate for the colorization task as the original grayscale image scores better than the of all evaluated methods. We therefore only provide the in this metric for colorization for the sake of completeness (finding good quantitative measure for the highly-ambiguous colorization task is a well-known unsolved problem).Additional on CelebA and Bedrooms dataset are given in Appendices A, F, G. Figure 6: A comparision of optimization over the convolutional manifold (column "OptConv"), the z-space (column "OptZ") and the Progressive GAN BID9 latent space (column "PGAN") on the CelebA-HQ dataset BID9. For the CelebA-HQ, we have limited comparison of the LCM model to the pretrained progressive GAN model BID9 published by the authors (this is because proper tuning of the parameters of other baselines would take too much time). On this dataset, LCM uses a latent space of 135k parameters. Additionally, we use CelebA-HQ to highlight the role of the convolutional manifold structure in the latent space. Recall that the use of the convolutional manifold parameterization is what distinguish the LCM approach from the GLO baseline. The advantage of the new parameterization is highlighted by the experiments described above. One may wonder, if the convolutional manifold constraint is needed at testtime, or if during the restoration process the constraint can be omitted (i.e. if can be used instead of with the generator network g trained with the constraint). Generally, we observed that the use of the constraint at testtime had a minor effect on the CelebA and Bedrooms dataset, but was very pronounced on the CelebA-HQ dataset (where the training set is much smaller and the resolution is much higher).In Figure 6 and TAB4, we provide qualitative and quantitative comparison between the progressive GAN model BID9, the LCM model, and the same LCM model applied without the convolutional manifold constraint for the task of inpainting. The full LCM model with the convolutional manifold performed markedly better than the other two approaches. Progressive GAN severely underfit even the known pixels. This is even despite the fact that the training set of BID9 included the validation set (since their model was trained on full CelebA-HQ dataset). Unconstrained LCM overfit the known pixels while providing implausible inpaintings for the unknown. Full LCM model obtained much better balance between fitting the known pixels and inpainting the unknown pixels. The in this work suggest that high-dimensional latent spaces are necessary to get good image reconstructions on desired hold-out sets. Further, it shows that parametrizing these spaces using ConvNets imposes further structure on them that allow us to produce good image restorations from a wide variety of degradations and at relatively high resolutions. More generally, this method can easily be extended to come up with more interesting parametrizations of the latent space, e.g. by interleaving the layers with image-specific and dataset-specific parameters. The proposed approach has several limitations. First, when trained over very large datasets, the LCM model requires long time to be trained till convergence. For instance, training an LCM on 150k samples of CelebA at 128 × 128 resolution takes about 14 GPU-days. Note that the GLO model of the same latent dimensionality takes about 10 GPU-days. On the other hand, the universality of the models means that they only need to be trained once for a certain image type, and can be applied to any degradations after that. The second limitation is that both LCM and GLO model require storing their latent representations in memory, which for large datasets and large latent spaces may pose a problem. Furthermore, we observe that even with the large latent dimensionalities that we use here, the models are not able to fit the training data perfectly suffering from such underfitting. Our model also assumes that the (log)-likelihood corresponding to the degradation process can be modeled and can be differentiated. Experiments suggests that however such modeling needs not be very accurate, e.g. simple quadratic log-likelihood can be used to restore JPEG-degraded images (Appendix H). Finally, our model requires lengthy optimization in latent space, rather than a feedforward pass, at test time. The number of iterations however can be drastically reduced using degradation-specific or universal feed-forward encoders from image-space to the latent space that may provide a reasonable starting point for optimization. This work has been supported by FIG1, we show comparison on the "extreme" task of half-image inpainting. FIG2 gives a comparison for the task of inpainting where 95% of pixel values are occluded at random. In both cases, the LCM model achieves the best balance of fitting the known evidence and the inpainting quality of known pixels. As a baseline in the main text, we have used the variant of the GLO model BID1, where the latent space is organized as maps leading to "fully-convolutional" generator. The latent dimensionality is picked the same as for the the LCM model. Here, we provide evidence that using the original GLO implementation with vectorial-structured latent space, followed by a fully-convolutional layer gives worse . In particular, we have tried different dimensionality of the latent space (up to 8192, after which we ran out of memory due to the size of the generator). The for vector-space GLO in comparison with the GLO baseline used in the main text are in Figure 9 and TAB6. The vector based GLO model, despite being trained on latent vector with relatively high dimensionality, clearly underfits. Figure 9: Image inpainting using GLO models with latent spaces of different dimension and structure. The GLO baseline from the main text achieves the best fit to the known pixels and arguably the best inpaintings of the unknown pixels. Figure 10: Image reconstruction using the WGAN-GP with gradually increasing penalties on the norm of the latent representation z as justified by the probabilistic model behind GANs. Increasing the weight of this penalty (shown above) leads to worse underfitting without improving the quality of the reconstruction. Therefore the comparisons in the main text use the variant without such penalty. Most GAN implementations (including ours) use Gaussian prior when sampling in the latent space. In principle, such prior should be imposed during the restoration process (in the form of an additional term penalizing the squared norm of z). We however do not impose such prior in the comparisons in the main text, since it makes the underfitting problem of GANs even worse. In TAB7 we demonstrate that the fitting error for the images from the train set indeed gets worse as the penalty weight is increased. In Figure 10, this effect is demonstrated qualitatively. The architecture details for the components of the LCM model are as follows:• Generator Network g θ: The generator network g θ has an hourglass architecture in all three datasets. In CelebA the map size varies as follows: 32 × 32 → 4 × 4 → 128 × 128 and the generator has a total of 38M parameters. In Bedrooms the map size varies as: 64 × 64 → 4 × 4 → 256 × 256 and the generator has a total of 30M parameters. In CelebAHQ the map size varies as 256 × 256 → 32 × 32 → 1024 × 1024 and the generator has a total of 40M parameters. All the generator networks contain two skip connections within them and have a batch-norm and the LeakyReLU non-linearity after every convolution layer.• Latent Network f φi: The latent network used in CelebA128 consists of 4 convolutional layers with no padding. The latent network used in Bedrooms and CelebA-HQ consists of 5 and 7 convolutional layers respectively with no padding. The code of our implementation is available at the project website. For the sake of completeness we provide losses of LCM and GLO models on the training and test set. We additionally provide the loss if the LCM is optimized over the z-space (i.e the output of f φ) instead of the parameters of f φ (The shown in row "LCM Z-Space"). In general, the full LCM model has higher loss for train and for test sets, as being more constrained than the other two methods. The additional constraints however allow the LCM model to perform better at image reconstruction tasks. In FIG3, we provide additional inpainting and superresolution on the Bedrooms dataset for the compared methods. In this section we show the of performing linear interpolations on the latent space of the convolutional GLO, and LCM and compare it to a linear cross-fade performed in the image space. We start by first finding the best fitting latent parameters (we optimize over φ for LCM and over z for convolutional GLO) for the source and target images and then perform linear interpolation between them. As can be seen in Figure 12, interpolations in the LCM latent space seem to be smoother and a lot more faithful to the training data distribution than interpolations in convolutional GLO latent space. Figure 12: Interpolations in the latent space of the LCM model (top row), the convolutional GLO model (middle row). For the reference, we also provide linear cross-fade in the image pixel space in the bottom row. In the case of our model, the interpolation is performed between φ 1 and φ 2, i.e. along the convolutional manifold. Arguably, LCM interpolations are more plausible, with faces rotating smoothly and with more plausible detailes (e.g. noses) in the case of LCM. Generally, there is noticeably less "double-vision" artefacts. Electronic zoom-in recommended. In this section we perform JPEG image restoration using a squared error negative log-likelihood function as a loss. As in the case of inpainting, super-resolution and colorization we perform the optimization over φ keeping the generator fixed. Results in Figure 13 suggest that LCMs can be used to restore images even when application-specific likelihood function is unknown/hard to model. Figure 13: Image restoration from heavy JPEG compression. Left -the input, middle -restored, right -ground truth. Rather than modeling JPEG degradation with a specific likelihood function, we used a simple quadratic (log)-likelihood potential (corresponding to Gaussian noise corruption). In this section, we show the of unconditional sampling from the LCM latent space. A random subset of m = 30k trained latent ConvNet parameter vectors {φ 1, . . ., φ m} are first mapped to a 512-dimensional space using PCA. We then fit a GMM with 3 components and a full covariance matrix on these 512-dim vectors and sample from it. FIG0 shows the of the sampling procedure. FIG0: Unconditional Image Generation. We first project the latent parameters, the φ's, to a lower dimensional space using PCA and then sample from it. The details are given in the text.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Fewshot example: "Data augmentation and adversarial training are very effective for disentangling correlated speaker and noise, enabling independent control of each attribute for text-to-speech synthesis." == "To leverage crowd-sourced data to train multi-speaker text-to-speech (TTS) models that can synthesize clean speech for all speakers, it is essential to learn disentangled representations which can independently control the speaker identity and noise in generated signals. However, learning such representations can be challenging, due to the lack of labels describing the recording conditions of each training example, and the fact that speakers and recording conditions are often correlated, e.g. since users often make many recordings using the same equipment. This paper proposes three components to address this problem by: formulating a conditional generative model with factorized latent variables, using data augmentation to add noise that is not correlated with speaker identity and whose label is known during training, and using adversarial factorization to improve disentanglement. Experimental demonstrate that the proposed method can disentangle speaker and noise attributes even if they are correlated in the training data, and can be used to consistently synthesize clean speech for all speakers. Ablation studies verify the importance of each proposed component. Recent development of neural end-to-end TTS models BID26 BID1 enables control of both labelled and unlabelled speech attributes by conditioning synthesis on both text and learned attribute representations BID27 BID21 BID10 BID0 BID5 BID9. This opens the door to leveraging crowd-sourced speech recorded under various acoustic conditions BID18 to train a high-quality multi-speaker TTS model that is capable of consistently producing clean speech. To achieve this, it is essential to learn disentangled representations that control speaker and acoustic conditions independently. However, this can be challenging for two reasons. First, the underlying acoustic conditions of an utterance, such as the type and level of noise and reverberation, are difficult to annotate, and therefore such labels are often unavailable. This hinders the use of direct conditioning on the acoustic condition labels in a way similar to conditioning on one-hot speaker labels BID1. Second, speaker identity can have strong correlations with recording conditions, since a speaker might make most of their recordings in the same location using the same device. This makes it difficult to learn a disentangled representation by assuming statistical independence BID6.We address this scenario by introducing three components: a conditional generative model with factorized latent variables to control different attributes, data augmentation by adding noise to training utterances in order to counteract the inherent speaker-noise correlation and to create ground truth noisy acoustic condition labels, and adversarial training based on the generated labels to encourage disentanglement between latent variables. We utilize the VCTK speech synthesis dataset BID23, and noise signals from the CHiME-4 challenge BID24 to synthesize a dataset containing correlated speaker and noise conditions for controlled experiments. We extensively evaluate disentanglement performance on the learned latent representations as well as the synthesized samples. Experimental identify the contribution of each component, and demonstrate the ability of the proposed model to disentangle noise from speakers and consistently synthesize clean speech for all speakers, despite the strong correlation in the training data. We base our TTS model on Tacotron 2 BID20, which takes a text sequence as input, and outputs a sequence of mel spectrogram frames. To control speech attributes other than text, two additional latent variables, z s and z r, are introduced to condition the generative process, where the former models speaker identity, and the latter models residual unlabelled attributes (e.g. acoustic conditions). Prior distributions for both variables are defined to be isotropic Gaussian. The full TTS model can be written as a conditional generative model with two latent variables: p(speech | z s, z r, text).Two variational distributions are introduced: q(z s | speech) and q(z r | speech), to approximate the intractable posteriors of the latent variables, following the variational autoencoder (VAE) framework BID14. Each distribution is defined to be diagonal-covariance Gaussian, whose mean and variance are parameterized by a neural network encoder. Note that z s, z r, and text are assumed to be conditionally independent given speech, in order to simplify inference. In contrast to learning an embedding for each speaker, learning an inference model for z s can be used to infer speaker attributes for previously unseen speakers. To factorize speaker and residual information, an auxiliary speaker classifier that takes z s as input is trained jointly with the TTS model. This encourages information that is discriminative between speakers to be encoded in z s, and leaves residual information to z r. A simple fully-connected network is used for the speaker classifier. When acoustic conditions are correlated with speakers, information about e.g. noise level can be used to discriminate between speakers, and therefore can be encoded into z s. To counteract such behavior, one can decorrelate these factors by leveraging prior knowledge that adding noise should not affect speaker identity. We propose to augment the original training set with a noisy copy that mixes each utterance with a randomly selected piece of noise at a randomly sampled signal-to-noise ratio (SNR), but reuses the same transcript and speaker label as the original utterance. This operation can be seen as flattening the SNR distribution of each speaker, in order to make SNRs less discriminative about speakers. To increase the degree of disentanglement, it is also useful to proactively discourage z s from encoding acoustic condition information. If the ground truth acoustic condition labels are available, domain adversarial training BID3 can be applied directly to encourage z s not to be informative about the acoustic condition. Nevertheless, such labels are often unavailable in crowdsourced datasets such as BID18.In order to utilize adversarial training in such a scenario, we propose to use the augmentation label (original/augmented) to replace the acoustic condition label (clean/noisy). This augmentation label can be seen as a noisy acoustic condition label: an augmented utterance must be noisy, but an original one can be either. If z s is invariant to acoustic conditions, then it is also invariant to augmentation labels, implying that the latter is a necessary condition for the former. Following BID3, invariance of z s to augmentation is measured using the empirical H-divergence between the z s distribution of the augmented data and that of the original data, given a hypothesis class H that is a set of binary classifiers. The empirical H-divergence measures how well the best classifier in the hypothesis class can distinguish between samples drawn from different distributions. However, it is generally hard to compute the empirical H-divergence. Following BID2 BID3, we approximate it with the Proxy A-distance: 2(1 − 2), where is a generalization error of an augmentation classifier trained to predict if z s is inferred from an augmented utterance. A simple fully-connected network is used for the augmentation classifier. The complete model is illustrated in FIG0, composed of three modules: a synthesizer, p(speech | z s, z r, text), an inference network with two encoders, q(z s | speech) and q(z r | speech), and an adversarial factorization module with speaker and augmentation classifiers, p(y s | z s) and p(y a | z r), where y s and y a denotes speaker and augmentation labels. The parameters of the synthesizer, the two encoders, the speaker classifier, and the augmentation classifiers are accordingly denoted as θ, φ s, φ r, ψ s, and ψ a, respectively. Training of the proposed model aims to maximize the conditional likelihood and the information z s contains about speakers, while minimizing the H-divergence between the z s inferred from the original utterances and that from the augmented ones. The H-divergence is approximated with the Proxy A-distance obtained from the augmentation classifier. The objective function can be formulated as combining an evidence lower bound (ELBO) with a domain adversarial training BID3 objective: DISPLAYFORM0 where λ 1, λ 2 > 0 are the loss weights for the two classifiers, and ELBO(θ, φ s, φ r ; speech, text) is formulated as: DISPLAYFORM1 Note that the augmentation classifier is optimized with a different objective than the rest of the model. To train the entire model jointly, a gradient reversal layer BID3 is inserted after the input to the augmentation classifier, which scales the gradient by −λ 2. Our formulation of a TTS model with latent variables are closely related to BID27 BID21 BID0 BID5 BID9, which focus on modeling unlabeled speech attributes. In contrast to this work, BID27 BID21 BID0 BID5 do not address disentangling attributes to enable independent control when different attributes are highly correlated in the training data, while BID9 learns to disentangle speaker attributes from the rest by encoding those with small within-speaker variance to z s.The proposed augmentation-adversarial training combines data augmentation for speech BID11 with domain adversarial neural networks (DANNs) BID3 for disentangling correlated attributes. These two methods have been mainly applied for training robust discriminative models BID7 BID22 BID24 BID19, and are less studied in the context of building generative models. In addition, our method provides two advantages. First, while DANNs require domain labels, our proposed method enables adversarial training even when the ground truth domain labels are unavailable. Second, domain adversarial training aims to remove domain information while preserving target attribute information; however, if domain and target attribute have very strong correlations, the two objectives conflict with each other, and one of the them will be compromised. Our proposed method alleviates such issues by using data augmentation to decorrelate the two factors. Learning disentangled representations for deep generative models has gain much interest recently BID8 BID28. Several studies also explored adversarial training for disentanglement, such as using maximum mean discrepancy BID15 and generative adversarial network BID17. We particularly emphasize disentangling statistically correlated attributes, and apply H-divergence based adversarial training on latent variables. We artificially generated a noisy speech dataset with correlated speaker and noise conditions using the VCTK corpus BID23 and noise from the CHiME-4 challenge BID24. The motivation here is to simulate real noisy data while evaluating the model under carefully controlled conditions. VCTK contains 44 hours of clean English speech from 109 speakers. We downsample the signals to 16 kHz to match the noise sample rate, and split it into training and testing sets in a 9:1 ratio. The CHiME-4 corpus contains 8.5 hours of noise recorded in four different locations (bus, cafe, pedestrian area, and street junction), which we split into three partitions: train, test, and aug. To simulate speaker-correlated noise, we randomly selected half the speakers to be noisy, and mixed all of their train and test utterances with noise sampled from train and test respectively, at SNRs ranging from 5 -25 dB. As described in Section 2.2, we generated an augmented set by mixing every (potentially noisy) training utterance with a noise signal sampled from aug at SNRs ranging from 5 -25 dB. Utterances in the augmented set are annotated with y a = 1, and those in the original noisy training set are annotated with y a = 0. We strongly encourage readers to listen to the samples on the demo page. The synthesizer network use the sequence-to-sequence Tacotron 2 architecture BID20, with extra input z s and z r concatenated and passed to the decoder at each step. If not otherwise mentioned, z s is 64-dim and z r is 8-dim. The generated speech is represented as a sequence of 80-dim mel-scale filterbank frames, computed from 50ms windows shifted by 12.5ms. We represent input text as a sequence of phonemes, since learning pronunciations from text is not our focus. The speaker and the residual encoders both use the same architecture which closely follow the attribute encoder in BID9. Each encoder maps a variable length mel spectrogram to two vectors parameterizing the mean and log variance of the Gaussian posterior. Both classifiers are fully-connected networks with one 256 unit hidden layer followed by a softmax layer to predict the speaker or augmentation posterior. The synthesizer, encoders, and speaker classifier are trained to maximize Eq with λ 1 = λ 2 = 1, and the augmentation classifier is trained to maximize Eq. The entire model is trained jointly with a batch size of 256, using the Adam optimizer BID13, configured with an initial learning rate of 10 −3, and an exponential decay that halves the learning rate every 12.5k steps, starting at 50k steps. We quantify the degree of disentanglement by training speaker and noise classifiers on z s and z r separately. The classification accuracy on a held-out set is used to measure how much information a latent variable contains about the prediction targets. A simple linear discriminative analysis classifier is used for all four tasks. If the classifier input contains no information about the target, the best a classifier can do is to predict the highest prior probability class. Since the distributions of both speaker and acoustic conditions are close to uniform, a speaker-uninformative input should in about 1% accuracy, and a noise-uninformative input should in about 50%.Results are shown in TAB1, comparing the full proposed model with two alternative models: one which removes adversarial training, denoted as "-adv,", and a second which further removes data augmentation, denoted as "-adv -aug." Without data augmentation and adversarial training, the second alternative completely fails to disentangle speaker from noise, i.e. its speaker encoding z s can infer both, while its residual encoding z s cannot infer either. The first alternative learns to encode acoustic condition into z r, reaching 96.5% accuracy on noise prediction; however, part of such information still leaks to z s, as indicated by the 85% noise prediction accuracy. The full proposed model achieves the highest noise prediction accuracy using z r, and the lowest accuracy using z s, implying the best allocation of acoustic information. Nevertheless, adversarial training also in slight degradation of speaker information allocation, where the speaker prediction accuracy using z r increases from 1.4% to 2.3%. We further analyze the latent space of the proposed model by visualizing the learned speaker and residual representations using t-SNE BID16, which is a technique for projecting high-dimensional vectors to a two-dimensional space. Results are shown in Figure 2, where each point corresponding to a projected z r (left column) or z s (right column) inferred from a single utterance. Points are color-coded according to speaker, gender, and accent labels in each row. In the left column, projected z r are clearly separated by acoustic condition, but not by gender or speaker. In contrast, projected z s shown in the right column forms many small clusters, with one speaker each cluster; Moreover, as shown in the middle row, clusters of speakers are further separated according to their genders. In the lower right panel, projected z s of noisy utterances and clean utterances are overlaid, demonstrating that z s have similar distributions conditioning on different acoustic conditions. To evaluate how well the two latent variables, z s and z r, can control the synthesized speech, we sample five clean speakers and five noisy speakers, and select one testing utterance for each speaker with duration ≥ 3s. For each of the ten utterances, the two latent variables are inferred using the corresponding encoders. We construct an evaluation set of 100 phrases that does not overlap with the VCTK corpus, and synthesize them conditioned on each combination of z r and z s, including those inferred from different utterances. The total 10,000 synthesized samples are divided into four groups, depending on the set of speakers (clean/noisy) z r and z s are inferred from. To quantify the ability to control noise, we use waveform amplitude distribution analysis (WADA) BID12 to estimate an SNR without a clean reference signal. We compare to a baseline multi-speaker Tacotron model, which removes the residual encoder and replaces the speaker encoder with a lookup table of 64-D speaker embeddings. The upper half of TAB2 presents the estimated SNRs of synthesized speech using this baseline, conditioning on the same five clean speakers and the five noisy speakers mentioned above. The difference in SNR between clean and noisy speakers indicates that the acoustic condition is tied to speaker identity in this baseline model. Results of the proposed model and the two alternatives mentioned in Section 4.2 are shown in the lower half of TAB2. By conditioning on z r inferred from clean utterances, the proposed model is able to synthesize clean speech even for noisy speakers whose training utterances all had noise. Moreover, when conditioning on the same set of z r, the proposed achieves the smallest discrepancy in SNR between different z s sets. On the other hand, the "-adv" variant has a larger discrepancy between different z s sets, indicating worse disentanglement comparing to the full model, while the "-adv-aug" variant fails to control noise through z r. These are in line with the noise prediction using z s and z r shown in TAB1. Figure 3 illustrates synthesized samples for a noisy speaker, comparing the baseline to our proposed model. Our model is capable of controlling noise using z r, and can generate clean speech for the noisy speaker, while the baseline output always contains noise. We next examine if z s can control the speaker identity of synthesized speech, using a text-independent speaker verification system BID25 to is compute speaker discriminative embeddings, called d-vectors BID4, from the reference and synthesized speech samples. The system is trained to optimize a generalized end-to-end speaker verification loss, so that the embeddings of two utterances are close to each other if they are from the same speaker, and far way if from different speakers. We build a nearest-neighbor classifier, which assigns an input signal the speaker label of the reference signal whose d-vector is closest to that of the input, measured using Euclidean distance. To prevent noise from affecting d-vector quality, we only evaluate synthesized samples conditioned on z r from clean utterances. TAB3 shows that the synthesized samples closely resemble the speaker characteristics of their corresponding reference samples, regardless of z r used for conditioning. The indicate that speaker identity is controlled by z s, while being invariant to change in z r. To quantify fidelity, we rely on crowd-sourced mean opinion scores (MOS), which rates the naturalness of the synthesized samples by natives speakers using headphones, with scores ranging from 1 to 5 in 0.5 increments. To quantify fidelity, we rely on crowd-sourced mean opinion scores (MOS), which rates the naturalness of the synthesized samples by natives speakers using headphones, with scores ranging from 1 to 5 in 0.5 increments. Results shown in TAB4 compares the baseline and the proposed model conditioning on z r from clean utterances. When conditioning on z r from clean utterances, the proposed model achieves a higher MOS score than the baseline. In contrast, the MOS drops significantly when conditioning on z r inferred from noisy utterances. The indicate that disentangling speaker and noise improves the naturalness of the generated speech, and the proposed model can synthesize more natural speech with less noise than the baseline when conditioning on z r inferred from clean signals. Finally, we study the sensitivity of disentanglement performance with respect to the choice of speaker encoding dimensions. As shown in the previous two sections, good latent space disentanglement translates to good performance in terms of control of speaker identity and acoustic conditions for synthesis. In this section, we only evaluate latent space disentanglement when changing the dimension of z s TAB5 compares performance of the proposed model when the dimensionality of z s is 32, 64, 128, and 256. Variants without data augmentation or adversarial training fail to disentangle in all configurations. When the dimension of z s increases, both the proposed model and "-adv" report worse separation of information, as indicated by increased noise prediction accuracy using z s. Specifically, the "-adv" fails to encode noise information in z r when z s has 128 dimensions, which could from a bad initialization of model parameters; however, such a behavior also indicates that when adversarial training is not applied, the disentanglement performance may rely heavily on the model initialization. On the other hand, the proposed model is least sensitive to the change of z s dimensionality. It always achieves the highest noise prediction accuracy using z r, and the lowest noise prediction accuracy using z s. We build a neural network TTS model which incorporates conditional generative modeling, data augmentation, and adversarial training to learn disentangled representations of correlated and partially unlabeled attributes, which can be used to independently control different aspects of the synthesized speech. Extensive studies on a synthetic dataset verify the effectiveness of each element of the proposed solution, and demonstrate the robustness to the choice of hyperparameters. The proposed methods for disentangling correlated attributes is general, and can potentially be applied to other pairs of correlated factors, such as reverberation and speaker, or to other modalities, such as controllable text-to-image generation. In addition, for future work, we would also like to investigate the capability of the proposed method to disentangle pairs of attributes which are both unsupervised.6 Acknowledgement"

We present a new deep latent model of natural images that can be trained from unlabeled datasets and can be utilized to solve various image restoration tasks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the article for finding a single-sentence tldr

Question: Automatic Essay Scoring (AES) has been an active research area as it can greatly reduce the workload of teachers and prevents subjectivity bias. Most recent AES solutions apply deep neural network (DNN)-based models with regression, where the neural neural-based encoder learns an essay representation that helps differentiate among the essays and the corresponding essay score is inferred by a regressor. Such DNN approach usually requires a lot of expert-rated essays as training data in order to learn a good essay representation for accurate scoring. However, such data is usually expensive and thus is sparse. Inspired by the observation that human usually scores an essay by comparing it with some references, we propose a Siamese framework called Referee Network (RefNet) which allows the model to compare the quality of two essays by capturing the relative features that can differentiate the essay pair. The proposed framework can be applied as an extension to regression models as it can capture additional relative features on top of internal information. Moreover, it intrinsically augment the data by pairing thus is ideal for handling data sparsity. Experiment shows that our framework can significantly improve the existing regression models and achieve acceptable performance even when the training data is greatly reduced. Automatic Essay Scoring (AES) is the technique to automatically score an essay over some specific marking scale. AES has been an eye-catching problem in machine learning due to its promising application in education. It can free tremendous amount of repetitive labour, boosting the efficiency of educators. Apart from automation, computers also prevail human beings in consistency, thus eliminate subjectivity and improve fairness in scoring. Attempts in AES started as early as Project Essay Grade (PEG) (; 2003), when the most prevalent methods relied on hand-crafted features engineered by human experts. Recent advances in neural networks bring new possibilities to AES. Several related works leveraged neural networks and achieved decent (; ; ;). As is shown in Figure 1, these approaches generally follow the'representation + regression' scheme where a neural network reads in the text embeddings and generates a high level representation that will be fed to some regression model for a score. However, such model requires a large amount of expert-rated essays for training. In reality, collecting such dataset is expensive. Therefore, data sparsity remains a knotty problem to be solved. Inspired by the observation that human raters usually score an essay by comparing it to a set of references, we propose to leverage the pairwise comparisons for scoring instead of regression. The goal of the model is shifted from predicting the score directly to comparing two essays, and the final score will be determined by comparing new essays with known samples. In order to achieve this, we designed a Siamese network called Referee Network (RefNet) and corresponding scoring algorithms. RefNet is a framework so that it can use various representation encoders as backbones. What's more, though this model is designed to capture mutual features, it can also benefit from essay internal information via transfer learning. Scoring essays by comparison has various benefits. First, RefNet is incredibly strong in dealing with data sparsity problem. Essays are paired with each other to form the training data for RefNet, which significantly augmented the data size. Experiments show that our model achieve acceptable performance even when the training data is radically reduced, while regression models are subject to drastic performance degeneration. Second, unlike end-to-end black-box models, our system scores an essay by comparing it with a set of labeled anchors, providing transparency to a certain degree during inference process. Last but not least, with information in both internal and mutual perspective, RefNet can have better insight into the quality of essays. Our contributions can be summarized as follows: • We designed Referee Network (RefNet), a simple but effective model to compare two essays, and Majority Probability Voting Algorithm to infer the score from pairwise comparison . To the best of our knowledge, it is the first time a Siamese neutral network is used in AES. • Our model intrinsically solves the problem of data sparsity. It achieves acceptable performance even when the training data is greatly reduced, while regression models are impaired a lot. Its efficacy in few-shot learning makes it an ideal solution for real applications where labelled data is usually limited. • RefNet exploits a new realm of information, mutual relationship, by pairwise comparison. With transfer learning, it also leverages internal features captured by regression. Moreover, RefNet can be applied as an extension to various regression models and consistently improve the performance. Existing AES solutions fall into two categories depending on how essays are represented: featureengineered models and end-to-end models. , 9 handcrafted features are fed into a Support Vector Machine(SVM). Those features range from simple ones like script length to elaborated ones like phrase structure rules and grammatical relation distance measure. However, no matter how many features are used, it cannot develop expressive representations of the essays. Besides, extracting handcrafted features often relies on other models, ing in highly coupled systems that are slow and sophisticated. Recent models based on neural networks can capture the features automatically. uses a self-trained look-up table for embedding and investigates a variety of neural networks, among which LSTM yields the best performance. Several variations of neural network were also explored: applies CNN with attention pooling on sentence level and LSTM with attention pooling on essay level. attached LSTM to a special network architecture called SkipFlow, which models the relationship between snapshots of hidden states of LSTM as it reads. All of these methods achieved decent , which proved the effectiveness of neural network in this problem. However, several problems remind to be solved. The first one is that no significant improvement over vanilla ones has been observed. The second problem is the scarcity of labeled data casts doubt on whether elaboration of neural networks has reached its bottleneck. A stark contrast to researchers' delicate designs in generating better representations is the rudimentary regression methods used to obtain the final prediction. Of course, a regression layer satisfies the fundamental need of this task: mapping a representation to a scalar. However, considering how complicated the relationship between essay contents and its score can be, the capability of such simple method is questionable. Pairwise difference between essays, on the contrary, is intuitively more understandable and, according to , outperforms mere regression in experiment. Instead of directly mapping to the grade, learning rank preference is used to explicitly model the grade relationships between essays. Yannakoudakis et al. used a special kind of SVM which outputs a real number with the -intensive loss function. However, its only distinction from regression is including the degree of misclassification as part of the loss function so that the SVM can maximize the difference between closely-ranked data pairs. It is more an amendment for regression than a new approach and cannot fully release the power the of comparison based methods. In the last section, we argue that the potential of ranking preference methods in AES has not yet been fully exploited. In this paper, we will try to exhaust the information in representations using a novel approach. As depicted in Figure 1, to score an essay in our system, Referee Network will compare it with known samples. Then probability majority voting will be used to infer the score from pairwise rank preference. Details will be elaborated in this section. We use the same scheme as to obtain the embeddings at sentence level. It is proven that Transformers, an attention mechanism that learns the contextual relations between words, can effectively make use of the textual information. The model Bidirectional Encoder Representations from Transformers (BERT) which is built on this this approach achieved state-of-the-art on various downstream tasks such as reading comprehension . Therefore, we directly retrieve the word embeddings from pre-trained BERT model. For a sentence with l words, BERT will output a set of low-dimensional representations s = {w 0, w 1, ..., w l, w l+1}, where w i (1 ≤ i ≤ l) denotes the embedding for the i th word, w 0 is a special tag CLS for classification tasks and w l+1 is a SEP tag for separating sentences. We use the average of the hidden states of the penultimate layer along the time axis to represent a sentence: Here the superscript -2 means that the embedding is retrieved from the second last layer in BERT. We use the penultimate layer instead of the final one because according to , the representations in the final layer fit too closely to the pre-training tasks including masked language modelling and next sentence prediction. Embeddings in the penultimate layer are flexible enough for the model to fit to our AES task while at the same time semantically meaningful. 3.2.1 NETWORK ARCHITECTURE As is mentioned above, we will not built a model that directly infers the score via any kind of regression. Instead, we built a model that compares essays, i.e takes two essays as input and outputs the rank preference. In order to achieve this goal, we borrowed the idea from and designed the Referee Network (RefNet). As shown in Figure 2, RefNet is actually embarrassingly simple. The pair of essays, which are both in the form of a set of sentences embeddings, will be encoded into essay representations through a backbone network. The backbone network can be anything that outputs a vector from a matrix with certain dimension, in our case, the size of BERT outputs. In this paper, we will try the following backbones: • Average Embeddings. For each input essay e = {s 1, s 2, s 3, ..., s m}, it will just compute the average of all sentences r = (• One layer of Simple Recurrent Neural Networks (RNN). • One layer of Long Short-Term Memory (LSTM). Due to the prevalence of RNN and LSTM, we shall skip their descriptions here. Details of these two network architectures can be found in. Note that for a pair of essays [e 1, e 2] ∈ R 2m×d, the RNN and LSTM backbones will generate two sets of hidden states {{h m}}, and we will use only use the state at the final time step because its superiority over average state over time has been presented by. Then the representations of the two essays will be concatenated along their first dimension r = [h After that, one fully connected layer with leaky linear rectifier activation will be applied to extract higher level features. Finally, another fully connected layer with two units together with softmax activation will give a rank preference, indicating with essay is better written: In this paper, we denote RefNet as R(e i, e j) in mathematical formulas. To keep things simple, we simplify the output of R(e i, e j) as a scalar in, indicating the probability that essay e i is better written than e j. To train RefNet, we firstly pair each essay with those with different scores within the same prompt to form essay pairs. Pairing is not conducted across prompts because essays from different prompts are not really comparable due to disparate writing requirements and scoring scales. We do not pair the essays with the same score because one essay can hardly be exactly as good as another: among identically scores essays, one can further distinguish one may be better than another. What's more, the inconsistent scoring schemes make concept of equal quality even more vague. In the ASAP dataset, for instance, the scale can be as wide as 0-60 or as narrow as 0-3. As a , two essays with the same score in 0-3 scale may have drastically different scores in 0-60 scale. Therefore, it is foreseeable that requiring RefNet to categorize identically rated essays will frustrate the model during training and impair the performance. After pairing, RefNet is trained by minimizing the cross entropy between model output and true values. We hope to make full advantage of the essay representations by exploiting both their internal and mutual information. And what enables RefNet to use internal information is transfer learning. As Figure 3 presents, we firstly train the backbone network with conventional end-to-end schemes where the model outputs the score directly by regression. After being trained in this pseudo task, the parameters in the LSTM are transferred to RefNet and fine tuned for essay comparison, which is the real task. In this way, what is learned in regression is more or less retained in the comparison model. After comparing the test essay against known samples, we designed an algorithm called probability majority voting to infer its final score. Suppose there are n i anchors for score notch N i. Each anchor is paired with the testing input x and fed into our referee network for comparison. By taking the average referee output over all the anchors, we get the probability of how likely that the essays at Two aspects are considered to form the voting criteria. First, according to the model, the test input is inferior than p i percent of anchors with score label N i. Intuitively, if we choose the anchors with higher scores, the test essay will also be beaten. Therefore, depending on how large p i is, it will more or less add to the likelihood that [N min, N i) contains the correct answer. In specific, votes of value p i will be given to all the scores in [N min, N i). Similarly, the test input is also superior than 1 − p i percent of anchors with score label N i. Thus, a vote with value 1 − p i is added to The second consideration is that we hope the p for the predicted score is close to 0.5, as one score does not appear correct if most of its anchors are better or worse than the input essay. Therefore, we will penalize the distance from 0.5 by subtracting |p i − 0.5| from the notch's own votes. Formally, total pairwise comparison for some essay is {N 1 : p 1, N 2 : p 2, ..., N k : p k}, where k is the total number of score notches. Without loss of generality, we assume that N 1 < N 2 <... < N k. The total votes for score notch N i is: Our final prediction is naturally the score notch with the highest votes. If more than one scores are the highest, we will take the average of all such scores and round to the nearest integer: In this section, we will firstly present some information regarding the dataset and performance metrics. Then we will show the performance of our model in two different tasks and compare with other models. After that, we will conduct ablation studies analyze the . We use Automated Student Assessment Prize(ASAP) dataset 1, which is the standard dataset for developing and evaluating AES systems. It is composed of 12976 essays in 8 prompts (see Table 1 for detailed statistics), where prompt 1,2,7,8 are persuasive, narrative or expository essays while the rest are Source Dependent Response questions. In our experiments, we use the same 5-fold split as did for fair comparisons. In this data split scheme, 60% of data is used for training, 20% for validation and the rest for testing. Similar to (; ; ;), we use Quadratic Weighted Kappa (QWK), which is the official standard for ASAP dataset, as our evaluation metric. The QWKs are computed for each prompt in its original scale respectively. All the experiments are conducted over all 5 folds and the the average of each prompt's QWKs over all folds is calculated as the performance measure for the model. We firstly trained our model on the whole dataset and observed that RefNet tends to overfit when all the training data is used. It is not supervising as the total training data is amplified by approximately 300 times after pairing, which means that the model will see each essay around 300 times within each epoch. As a , the model can overfit before the first epoch is finished. To solve this problem, training samples are dropped randomly before training. To minimize the potentially negative impact, we set up different dropping rate for different essay pairs. As Figure 4 suggests, when training on the whole dataset, RefNet can easily give accurate comparisons between two essays with contrasting scores. However, distinguishing the pairs that have similar scores is much more challenging. Therefore, the pairs with larger score difference are more likely to be dropped while those with smaller difference is more likely to be kept. Different dropping rate are shown in Table 4.2. After the data adjustment, approximately only 15% of training set are kept. The first three blocks of Table 3 compares the of regression and RefNet with the same backbone. One can clearly see that RefNet constantly outperforms conventional regression approaches. Notice that our method offers the greatest improvement in for RNN backbone, which is the weakest of the three. That is because scoring by comparison is not very sensitive to the quality of representations. As long as the representation makes sense, RefNet is able to boost the performance to a high level. We also compare our model with the ensembled neural networks in and SkipFlow LSTM networks proposed by. The show that we achieved the state-of-the-art average QWK score, and our edge is particularly large in prompt 8 which has much less data but more score notches than other prompts. In this task, we created two mini-ASAP datasets by excerpting 25% and 10% of essays from the original training and validation dataset respectively while testing data remains the same. Noticing that the essays under each prompt are distributed evenly in the original train test split, we gave special attention to this issue and our mini-ASAP dataset is balanced among different prompts. And we will not use the drop training data as the excessive data problem no longer exists. Table 4 we can see that with regression, the model will suffer from major performance degradation when the training data is reduced. On the contrary, RefNet is much more robust to scarce data. Even after 90% of data is dropped, RefNet can still offer high quality predictions. Several ablation tests are conducted to study the effects of individual components. In the first experiment, we remove the transfer learning and try to train RefNet from starch to test its efficacy. In the second experiment, data adjustment is disabled and RefNet is trained on the whole dataset. Finally, we hope to compare the performance of regression and pairwise methods under the same representations. We fix the transferred backbone parameters and train the fully connected comparison layers in RefNet only and see how QWK varies. All the ablation tests use LSTM as backbone. Table 5, the pairwise ranking approach achieved 0.754 on the fixed embeddings learnt from regression task, which is higher than pure regression performance 0.729 in Table 3. It shows that by taking mutual information into account, scoring by pairwise comparison can consistently improve the performance. Second, internal information is also exploited by transfer learning. With information from both internal and mutual perspective, RefNet can have better insight into the quality of essays. Besides, considering the complexity of the task, it is hard to train RefNet from scratch. Results in Table 5 show that the performance of RefNet without transfer learning degenerates by 2.48%. Third, RefNet is naturally invulnerable to cross-prompt noise. In reality, dataset of a single prompt is rarely large enough, so hybrid dataset such as ASAP dataset are commonly used. However, the hybrid dataset consisting of essays of different scoring range, written by students at different levels or with different s may be not self-consistent. For regression, the scores from different score range should be carefully aligned before training on the whole dataset because end-to-end method is sensitive to labels. However, such alignment can hardly be achieved. Current works just linearly rescale whatever original score range to, bringing noise to the system. In contrast, RefNet sees only the relative relation between essays within the same prompt, avoiding possible cross-prompt noise. RefNet shows even larger edge over regression in few-shot learning problems. On one hand, RefNet is intrinsically suitable for few-shot learning problems as the pairing operation can amplify the training data by one or multiple levels of magnitude. What's more, unlike some common data augmentation techniques such as random transforms, no noise is introduced in pairing. On the other hand, the'representation + regression' mechanism is highly data demanding. First, though both approaches needs to somehow extract the features from texts, predicting by regression imposes higher requirement in expressing those features in a numeric and explicit form. Second, since basic models such as LSTM and the features that can be learnt by those models have been well exploited, researchers may have to resort to deeper and more elaborated features to push the performance, ing in more complicated models with more parameters. Both factors makes endto-end model unable to be fully trained in small datasets. In this paper we present Referee Network, a framework for automatic essay scoring using pairwise comparisons. We demonstrate that RefNet is expert in solving data sparsity problems. It can retain the performance at a high level even when the training data is significantly reduced, which outperforms regression models by a significant margin. We also show that RefNet can improve conventional regression models by leveraging the additional mutual information between representations. With only vanilla backbones, our model is able to obtain state-of-the-art . Even if the essay representations are fixed and have mediocre quality, our model can still boost up the scoring accuracy. Furthermore, the capacity of RefNet can go far beyond this context as it is an extendable framework that can be used with any kind of representation encoders. Besides the simple backbones we tried in this paper, one can by all means utilize more complicated and better performing models as backbones. In this way, the performance of AES systems can always be pushed to a new record.

Represent the short summary for finding the corresponding scientific article (long)

Automatically score essays on sparse data by comparing new essays with known samples with Referee Network.