Question: represent the text to find a very short summary (1-2 sentences max)

Question: Most deep neural networks (DNNs) require complex models to achieve high performance. Parameter quantization is widely used for reducing the implementation complexities. Previous studies on quantization were mostly based on extensive simulation using training data. We choose a different approach and attempt to measure the per-parameter capacity of DNN models and interpret the to obtain insights on optimum quantization of parameters. This research uses artificially generated data and generic forms of fully connected DNNs, convolutional neural networks, and recurrent neural networks. We conduct memorization and classification tests to study the effects of the number and precision of the parameters on the performance. The model and the per-parameter capacities are assessed by measuring the mutual information between the input and the classified output. We also extend the memorization capacity measurement to image classification and language modeling tasks. To get insight for parameter quantization when performing real tasks, the training and test performances are compared. Deep neural networks (DNNs) have achieved impressive performance on various machine learning tasks. Several DNN architectures are known, and the most famous ones are fully connected DNNs (FCDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).It is known that neural networks do not need full floating-point precision for inference BID10 BID16 BID23. A 32-bit floating-point parameter can be reduced to 8-bit, 4-bit, 2-bit, or 1-bit, but this can incur performance degradation. Therefore, precision should be optimized, which is primarily conducted by extensive computer simulations using training data. This not only takes much time for optimization but also can incorrectly predict the performance in real environments when the characteristics of input data are different from the training data. In this study, we attempt to measure the capacity of DNNs, including FCDNN, CNN, and RNN, using a memorization and classification task that applies random binary input data. The per-parameter capacities of various models are estimated by measuring the mutual information between the input data and the classification output. Then, the fixed-point performances of the models are measured to determine the relation between the quantization sensitivity and the per-parameter capacity. The memorization capacity analysis are extended to real models for performing image classification and language modeling, by which the parameter quantization sensitivity is compared between memorization and generalization tasks. The contributions of this paper are as follows.• We experimentally measure the memorization capacity of DNNs and estimate the perparameter capacity. The capacity per parameter is between 2.3 bits to 3.7 bits, according to the network structure, which is FCDNN, CNN, or RNN. The value is fairly independent of the model size.• We show that the performance of the quantized networks is closely related to the capacity per parameter, and FCDNNs show the most resilient quantization performance while RNNs suffer most from parameter quantization. The network size hardly effects the quantization performance when DNN models are trained to use full capacity.• We explain that severe quantization, such as binary or ternary weights, can be employed without much performance degradation when the networks are in the over-parameter region.• We suggest the sufficient number of bits for representing weights of neural networks, which are approximately 6 bits, 8 bits, and 10 bits for FCDNNs, CNNs, and RNNs, respectively. This estimate of the number of bits for implementing neural networks is very important considering that many accelerators are designed without any specific training data or applications.• The study with real-models shows that neural networks are more resilient to quantization when performing generalization tasks than conducting memorization. Thus, the optimum bits obtained with the memorization tasks are conservative and safe estimate when solving real problems. The paper is organized as follows. In Section 2, previous works on neural network capacity and fixedpoint optimization are briefly presented. Section 3 explains the capacity measurement methods for DNN models. Section 4 presents parameter capacity measurement for FCDNNs, CNNs, and RNNs. The quantization performances measured on DNNs are presented in Section 5. Concluding remarks follow in Section 6. The capacity of neural networks has been studied since the early days of DNN research. Although the capacity can be defined in many ways, it is related to the learnability of networks. The capacity of networks is shown as the number of uncorrelated random samples that can be memorized BID8. A single-layer perceptron with n parameters can memorize at least 2n random samples BID11. In other words, the network can always construct a hyperplane with n parameters that divides 2n samples. Additionally, the capacity of a three-layer perceptron is proportional to the number of parameters BID0. Recently, RNNs were trained with random data to measure the capacity per parameter BID6. Our study is strongly motivated by this research, and extends it to the quantization performance interpretation of generic DNN models, including FCDNN, CNN, and RNN. Recent studies have showed that neural networks have a generalization ability even if the expressive capacity of the model is sufficiently large BID33 BID2. In this paper, we also discuss the effect of network quantization when performing generalization tasks. Early works on neural network quantization usually employed 16-bit parameters obtained by directly quantizing the floating-point numbers BID10. Recently, a retraining technique was developed to improve the performance of quantized networks BID16 BID23. Retraining-based quantization was applied to CNN and RNN models, showing superior performance compared to directly quantized ones BID1 BID31. Many studies attempting extreme quantization have been published, such as 2-bit ternary BID16 BID21 BID35, 1-bit binary weight quantization, and XNOR networks BID7 BID28. Some aggressive model compression techniques also employed vector quantization or table look-up BID12 BID4. However, not all CNNs show the same quantization performance. For example, AlexNet BID20 shows almost the same performance with only 1-bit quantized parameters. However, the same quantization technique incurs a very severe performance loss when applied to ResNet BID28. A previous study shows that large sized networks are more resilient to severe quantization than smaller ones. Theoretical works and many practical implementation optimization techniques have been studied BID13 BID17 BID19 BID30 BID24. Recent work increases the number of network parameters to preserve the performance under low-precision quantization BID26. Our works are not targeted to a specific data or model, but introduce the general understanding of parameter quantization. We assess the network capacity of DNN models using a random data memorization and classification task BID6. In this task, N random binary vectors, X, are generated and each is randomly and uniformly assigned to the output label Y. The size of the binary vector depends on the DNN model. For FCDNN, the input X is a one dimensional vector whose size is determined by the hidden layer dimension. In CNN, the input needs to be a 2-D or 3-D tensor. Input samples of CNNs are generated by concatenating and reshaping random binary vectors. During the training process, the DNN is trained to correctly predict the label, which is 0 or 1, of the random input X. As the number of input data size, N, increases, the classification accuracy drops because of the limited memorization capacity. Note that the accuracy for the memorization task refers to the training performance after convergence because there is no proper test dataset for random training samples. The capacity is measured using the mutual information, defined as a measure of the amount of information that one random variable contains about another random variable (Cover & BID9 . The mutual information of a trained network with N input samples is calculated as follows: DISPLAYFORM0 where p is the mean classification accuracy for all samples under trained parameter θ. The network capacity is defined as DISPLAYFORM1 The accuracy, p, may vary depending on the training method of the model. We find N and p that maximize the mutual information of the networks by iteratively training the models. This optimization employs both grid search-and Bayesian optimization-based hyper-parameter tuning BID5 . The optimization procedure consists of three stages. First, we try to find the largest input data size whose accuracy is slightly lower than 1. Second, we perform a grid search to determine the boundary values of the hyper-parameters. The searched hyper-parameters can include initialization, optimizer, initial learning rate, learning rate decay factor, batch size, and optimizer variables. Finally, we conduct hyper-parameter tuning within the search space using Scikit-learn library BID27 . We add the number of training samples N as a hyper-parameter and use the mutual information of Eq. FORMULA0 as the metric for the optimization. Quantization of model parameters perturbs the trained network, therefore, fixed-point training or retraining with full-precision backpropagation is usually needed BID16 BID21 BID7 BID34 . However, the performance of the quantized networks does not always meet that of the floating-point models, even after retraining. This suggests that model capacity is reduced by quantization, especially when the number of bits used is very small. In this research, we observe the memorization capacity degradation caused by quantization in generic FCDNN, CNN, and RNN models. The uniform quantization is used for the sake of convenient arithmetic, and the same step size is assigned to each layer in the FCDNN, each kernel in the CNN, or each weight matrix in the LSTM layer. The bias values are not quantized, because they have a large dynamic range. It is important to note that the weights connected to the output are not quantized, because their optimum bit-widths depend on the number of labels in the output. Quantization is performed from floating-point to 8-bit, 6-bit, 5-bit, 4-bit, 3-bit, and 2-bit precision, in sequence. Retraining is performed after every quantization, but requires only a small number of epochs, because only fine-tuning is needed BID16 . We compare the generalization performance of floating-point and fixed-point DNNs by visualizing the loss surface. Loss is measured by applying Gaussian random noise to the parameters of the trained network as shown in Eq.. DISPLAYFORM0 Here, L(θ) is the loss according to the network parameters. The distribution of weights may vary depending on the model size and learning method. We apply the normalized filter noise to the θ noise for fair comparison on different models BID22.We employ two real networks, one is for image classification with CIFAR-10 dataset and the other is language modeling with Penn Tree Bank (PTB) dataset. One large and one small model are trained for these networks. We quantize those networks with the precision of 8, 6, 4, and 2 bits and analyze the variation of the surface according to the precision. θ noise is added to the quantized parameters. In order to reduce the error due to randomness of noise, all loss values are measured with 10 different trials and the average values are plotted. The capacities of FCDNNs, CNNs, and RNNs are measured via the memorization task explained in Section 3.1. The models used for the test employ floating-point parameters. The training data for FCDNNs is a 1-D vector of size n in. N input data are used as for the training data. The output, Y, is the randomly assigned label, either 0 or 1, for each input. Thus, inputs, X and Y, are represented as X ∈ {0, 1} N ×nin and Y ∈ {0, 1} N, respectively. The input data dimension, n in, should be larger than log 2 N so that no overlapped data is contained among N input data. In the experiments for FCDNNs, the input vector dimension, n in, is chosen to be equal to the number of units in the hidden layer. We conduct experiments for FCDNNs with hidden layer dimensions of 32, 64, 128, and 256, and with hidden layer depths of 1, 2, 3, and 4. The initialization method chosen is the'He' initialization BID14 and gradients are updated following the rule in SGD, with momentum, which shows the best performance in our grid search. The initial learning rate for hyper parameter tuning is chosen between 0.001 and 0.05 on the log scale. The decay factor and momentum are set to have even distance values in the linear scale between 0.1 and 0.5 and between 0.6 and 0.99, respectively. For each model, experiments are conducted to measure the accuracy of memorization while increasing the size of the input data, N. Note that only the training error is measured in this memorization task, because there is no unseen data. Experimental are based upon the best accuracy obtained when attempted with different hyper parameters. The capacity of the model is estimated according to Eq., where p is the training accuracy. The experimentally obtained memorization capacities of the FCDNN models are presented in FIG1, where depths of 1, 2, 3, and 4, and widths of 32, 64, 128, and 256 are used. When the number of hidden layers is the same, the amount of data that can be almost perfectly memorized quadruples when the dimension of the hidden layer is doubled. This means that the memorization capacity is linearly proportional to the number of parameters. Similarly, the FCDNN models with 2, 3, or 4 hidden layer depths can memorize 2, 3, or 4 times the input data as compared to the single layer DNN, respectively. FIG2 shows the memorization accuracy and the mutual information obtained using Eq. on the FCDNN. The model is composed of three layers and the hidden layer of size 64. Here, we find that the amount of mutual information steadily increases as the input data size grows. However, it begins to drop as the input size grows farther, and the memorization accuracy drops. By analyzing the accuracy trend of the model, it is possible to distinguish the input data size into three regions: the over-parameterized, the maximum performance, and the under-parameterized sections, as shown in FIG2. For example, if the model is trained to memorize only 10,000 data, it can be regarded as over-parameterized. The number of data that can be memorized by maximally utilizing all the parameters is between 30,000 and 40,000. In over-parameterized regions, performance can be maintained, even if the capacity of the networks is reduced. The per-parameter capacity of FCDNNs is shown in FIG2. Regardless of the width or depth, one parameter has a capacity of 1.7 to 2.5 bits, and FCDNNs have an average of 2.3-bit capacity per parameter. This is consistent with theoretical study BID11 BID0. The total capacity of the model may be interpreted as an optimal storage that can store a maximum of random binary samples BID11 BID3. The capacity of CNNs is also measured via a similar memorization task. CNNs can have a variety of structures according to the number of channels, the size of the kernels, and the number of layers. The kernel size of CNNs in this test are either (3 × 3) or (5 × 5), which are the same for all layers, the number of convolution layers from 3 to 9. The dimensions of the inputs are n height = n width = 32 and n channel = 1 for all experiments. Three max-pooling operations are applied to reduce the number of parameters in the fully connected layer. CNN structures used in our experiments are shown in Supplementary materials. The CNN models contain not only convolution layers but also fully connected layers. Thus, the per-parameter capacity for convolution layers is calculated after subtracting the capacity for fully connected layers from the measured total capacity. We assume the per-parameter capacity of the fully connected layer as 2.3 bits to calculate the capacity for convolution layers. As shown in FIG2, the convolution layers have the per-parameter capacity of between 2.86 and 3.09 except the smallest model, which is higher than that of FCDNNs. The average capacity per parameter of the tested models is 3.0 bits. Results show that the per-parameter capacity of CNNs is higher than that of FCDNNs, even when CNNs memorize uncorrelated data. Note that one parameter of FCDNNs is used only once for each inference. However, the parameter of CNNs is used multiple times. This parameter-sharing nature of CNNs seems to increase the amount of information that one parameter can store. It has been shown that the various structures of RNNs all have similar capacity per parameter of 3 to 6 bits BID6. We train RNNs with a dataset with no sequence correlation to show the capacity of the parameters. The random input dataset is composed of inputs, X ∈ {0, 1} N ×nseq×nin and labels Y ∈ {0, 1} N, which are uniformly set to 0 or 1. The training loss is calculated using the cross-entropy of the label at the output of the last step. We train RNNs with a single LSTM layer of 32-D. The input dimension, n in, is also 32-D and the amount of unrolling sequence, n seq, is five-step. It has been reported that unrolling of five-step almost saturates the performance in this setup BID6. We apply 5 input random vectors, X 0, X 1, X 2, X 3, and X 4, each with 32-D, and assign one label to this 160-D vector at the last time step. The error propagates from the last step only, and the outputs at intermediate time-steps are ignored. The number of parameters in the network is 8,386. In this case, the maximum mutual information is obtained when the number of samples is 32K, and the memorization accuracy is 99.52 %. Therefore, the per-parameter capacity of the model is 3.7 bits. The RNN shows higher per-parameter capacity than FCDNNs and CNNs. We have shown that FCDNNs, CNNs, and RNNs have different per-parameter capacities. According to the parameter-data ratio, a trained DNN can be an over-parameterized, max-capacity, or underparameterized model. Thus, we can assume that the DNN performance under quantization would depend on not only the network structure, such as FCDNN, CNN, or RNN, but also the parameter-data ratio. The experiments are divided into two cases. The first is to measure performance degradation via quantization precision when each model is in the maximum capacity region. The second analyzes performance when the models are in the over-parameterized region. When the FCDNN, CNN, and RNN are trained to have the maximum memorization capacity, the performances with parameter quantization are shown in Fig. 3(a). The FCDNN, CNN, and RNN models are shown. The fixed-point performances of two FCDNNs, two CNNs, and two RNNs are illustrated. With 6-bit parameter quantization, the FCDNN shows no accuracy drop. However, those for CNNs and RNNs are 5 % and 18 %, respectively. Because the RNN contains the largest amount of information at each parameter, the loss caused by parameter quantization seems to be the most severe. We also find that there is no decline in performance until the parameter precision is lowered to 6-bit for FCDNNs, 8-bit for CNNs, and 10-bit for RNNs, even when all models use full capacity. Next, we show the fixed-point performance of DNNs when they are trained to be in the overparameterized region. Note that the per-parameter capacity is lowered in the over-parameterized region. We conducted simulation with half size of the maximum number of data that can be memorized. For example, an FCDNN used for the measurement has 3 hidden layers with a hiddenlayer dimension of 128; the capacity of the corresponding model is about 2 17 bits. The network is FORMULA6 FC FORMULA0 FC FORMULA1 FC FC FORMULA0 FC FORMULA1 (a) LSTM FORMULA0 LSTM FORMULA1 LSTM FORMULA0 (c) over-parameterized when the number of memorized samples is 2 16. Fig. 3(b) shows that the FCDNN model memorizes all samples even with 4-bit parameter quantization when the model uses half of the capacity. Also, over-parmeterized model is less sensitive to bit-precision on CNNs and RNNs. The performances of fixed-point DNNs with the number of samples are shown in Fig. 4. The shows that DNNs are more robust when the networks are more over-parameterized.2 We have assessed the required precision of networks for performing memorization tasks. The memorization test only uses the training data that are artificially generated. However, most neural networks should conduct more than memorization because the test data are not seen during the training. In this section, we analyze the effects of network quantization for performing real tasks. We train two different sized CNN models with CIFAR-10 data. The structures of the two models are as follows: DISPLAYFORM0 The size of kernels of both models is (3×3), 16C represents a convolution layer with 16 channels and 128F C means a fully connected layer with 128-dimension. The number of parameters is 0.22M for the small model and 3.5M for the large one. Both models were trained with the same hyper-parameter setting. To analyze the impact of network quantization on the test performance, we plot the loss and accuracy surfaces of floating-point and quantized CNN models in Fig. 5. For simplicity, the of floatingpoint and 2-bit fixed-point CNN are given. Please refer to the Supplementary materials for other . When applying the training data that may have been memorized during the training phase, the large model shows indifferent performance surface regardless of the parameter precision. But, for the small model, the 2-bit model shows quite degraded performance when compared to the floating-point network. However, the test accuracy of small 2-bit model is not much lowered. We can notice that the loss surface of the 2-bit model shown in Fig. 5 is much wider than that of the floating-point model. This is consistent with recent studies on generalization BID18 BID22. This observation suggests that the quantized networks are more resilient when performing generalization tasks. Thus, the required precision of the network obtained with the memorization task can be considered a conservative estimate. Fig. 6 shows the training and test data based performance of fixed-point CNN and RNN on real data. T iny model has the same structure as the small model, but reducing the number of channels by half and the size of the fully connected layers by a quarter. The RNNs are trained for language modeling with Penn Treebank corpus BID25, and the models consists of two LSTM layers with the same dimension. Here, both for CNN and RNN models, we can confirm that large networks are more robust to quantization. Also, the networks need more parameter precision when conducting memorization tasks using the training set, rather than solving real problems using the test set. We have measured fixed-point DNN performance on real tasks and are shown in FIG7. FCDNN models are trained with MNIST dataset, ResNet BID15 models are trained using ILSVRC-2012 dataset BID29 and RNN based word-level language models (WLMs) are designed using PTB dataset. Experimented FCDNNs and RNNs are composed of two FC layers and two LSTM layers with the same dimension, respectively. 4-bit quantized FCDNNs shows almost same performance compared to the floating-point networks even when the number of neurons are only 8. Performances are preserved up to 6 bits on ResNets and RNN WLMs. Their resiliency to quantization increases as networks become larger. Quantization of parameters is a straightforward way of reducing the complexity of DNN implementations, especially when VLSI or special purpose neural processing engines are used. Our study employed simulations on varying sizes of generic forms of DNN models. Memorization tests using random binary input data were conducted to determine the total capacity by measuring the mutual information. Our simulation show that the per-parameter capacity is not sensitive to the model size, but is dependent on the structure of the network models, such as FCDNN, CNN, and RNN. The maximum per-parameter capacities of FCDNNs, CNNs, and RNNs are approximately 2.3 bits, 3.0 bits, and 3.7 bits per parameter. Thus, RNNs have the tendency of demanding more bits when compared to FCDNNs. We quantized DNNs under various capacity-utilization regions and showed that the capacity of parameters are preserved up to 6 bits, 8 bits, and 10 bits on FCDNNs, CNN, and RNNs, respectively. The performance of the quantized networks was also tested with image classification and language modeling tasks. The show that networks need more parameter precision when conducting memorization tasks, rather than inferencing with unseen data. Thus, the precision obtained through the memorization test can be considered a conservative estimate in implementing neural networks for solving real problems. This research not only gives valuable insights on the capacity of parameters but also provides practical strategies for training and optimizing DNN models. DISPLAYFORM0 In FCDNNs, each weight matrix, W l between two layers, demands |h l−1 | × |h l | weights, where |h| is the number of units for the layer, l. CNNs, which are popular for image processing, usually receive 2-dimensional (D) or 3-D input data, whose size is much larger than the filter or kernel size. The set of weights between layers is referred to as the'kernel' and the output is referred to as the'feature map'. Because the input size is usually much larger than the kernel size, the CNN parameters are reused many times. A kernel slides over the input feature map and produces an output feature map, and the sliding step is determined by the stride, s. The convolution weights is denoted as W l ∈ R k l,h ×k l,w ×n l−1 ×n l and feature map of the layer as C l ∈ R c l,h ×c l,w ×n l. k l,h and k l,w are height and width of each kernel and c l,h, c l,w are height and width of the feature map in layer l, respectively. n l is the number of feature map in the layer l. CNNs can have a variety of structures according to the number of channels, the size of the kernels, and the number of layers. We attempted to produce a general setting for CNNs. The experiments models are shown in Table. 1. We constructed CNNs to have twice the number of feature maps and half the height/width after pooling. Also, to minimize side-effects by fully connected layers, the output feature of the last convolution layer is flattened and directly propagated to the sof tmax layer. The capacity per parameter is measured only for parameters in convolution layers, by subtracting capacity of the fully connected layer. RNNs have a feedback structure that reflects the information in the previous steps when processing sequence data. RNNs are composed of one or multiple recurrent layers, and each layer computes the output, y t, and the hidden state, h t, using the previous hidden state, h t−1, and the input, x t.We use LSTM as the recurrent layers, showing stable performance in various applications. The mutual information equation of a network can be obtained as follows. We first re-write our random variables X, Y, andŶ. DISPLAYFORM0 DISPLAYFORM1 where f (θ, X i) is the predict of a network when the input is X i. Under our experimental setting, both X and Y have uniform random distribution. Note that X and Y are independent as well as Y i and Y j when i = j. Therefore, DISPLAYFORM2 And we use the network's average accuracy p as a probability of Y i =Ŷ i, so that DISPLAYFORM3 Finally, the equation is derived as: DISPLAYFORM4 The loss surfaces of fixed-point CNNs are shown in Fig. 8 and Fig. 9. The models are trained with CIFAR-10 dataset. The loss surfaces of RNN LMs for PTB dataset are also shown in FIG1 and FIG1. The performance degradation on test dataset is lower than the degradation on training dataset in all experiments.

represent the tldr for finding the corresponding scientific article! E.g. Tackling inverse design via genetic algorithms augmented with deep neural networks.  == Challenges in natural sciences can often be phrased as optimization problems. Machine learning techniques have recently been applied to solve such problems. One example in chemistry is the design of tailor-made organic materials and molecules, which requires efficient methods to explore the chemical space. We present a genetic algorithm (GA) that is enhanced with a neural network (DNN) based discriminator model to improve the diversity of generated molecules and at the same time steer the GA. We show that our algorithm outperforms other generative models in optimization tasks. We furthermore present a way to increase interpretability of genetic algorithms, which helped us to derive design principles The design of optimal structures under constraints is an important problem spanning multiple domains in the physical sciences. Specifically, in chemistry, the design of tailor-made organic materials and molecules requires efficient methods to explore the chemical space. Purely experimental approaches are often time consuming and expensive. Reliable computational tools can accelerate and guide experimental efforts to find new materials faster. We present a genetic algorithm (GA) (; ; ;) for molecular design that is enhanced with two features: Inverse design is the systematic development of structures with desired properties. In chemistry , the challenge of inverse design has been tackled as an optimization problem, among others in the form of variational autoencoders (VAEs), generative adversarial networks (GANs) and genetic algorithms. Variational autoencoders and generative adversarial networks VAEs are a widely used method for direct generation of molecular string or graph representations (Gómez-). They encode discrete representations into a continuous (latent) space. Molecules resembling a known structure can be found by searching around the region of the encoded point. Making using of the continuous latent representation, it is possible to search via gradients or Bayesian Optimization (BO). However, the generation of semantically and syntactically valid molecules is a challenging task. Thus, several follow up works to the VAE have been proposed for inverse design in chemistry. Among them, CVAE (Gómez-), GVAE and SD-VAE work directly on string molecular representations. Alternatively, JT-VAE (a) as well as work on molecular graphs. Unlike latent space property optimization, the policy network (PN) based GCPN model proposes a reinforcement learning (RL)-based method for direct optimization on molecular graphs. ORGAN demonstrate training string-based generative adversarial networks (GANs) via RL. Another method based on adversarial training is VJTNN (b). first introduced molecule generating models based on language models and reinforcement learning, where actions in an environment are taken to construct a molecule, receiving reward from an external scoring function. This model has also shown strong performance in the GuacaMol benchmark . In all the approaches mentioned above, generative models are trained to mimic the reference data set distributions, thus limiting the exploration ability of VAEs and GANs. There exist several examples of GA based molecule optimization algorithms in literature (; ; ; ; ;) ). While some of these examples pre-define mutations on a SMILES level to ensure validity of the molecules, other approaches use fragmentbased assembly of molecules. GAs are likely to get trapped in regions of local optima . Thus, for selection of the best molecules, report multiple restarts upon stagnation. We include the aforementioned models as baselines for numerical comparison. 3 GA-D ARCHITECTURE 3.1 OVERVIEW Our approach is illustrated in Figure 1. Our generator is a genetic algorithm with a population of molecules m. In each generation, the fitness of all molecules is evaluated as a linear combination of molecular properties J(m) and the discriminator score D(m): Random mutations of high fitness (best performing) molecules replace inferior members, while bestperforming molecules continue to a subsequent generation. The probability of replacing a molecule is evaluated using a smooth logistic function based on a ranking of fitness among the molecules of a generation. At the end of each generation, a neural network based discriminator is trained jointly on molecules generated by the GA and a reference data set. The fitness evaluation accounts for the discriminator predictions for each molecule. Therefore, the discriminator plays a role in the selection of the subsequent population. Mutation of molecules to populate subsequent generations is an important element of the GA. A low degree of mutation can lead to a slow exploration of the chemical space, causing stagnation Figure 1: Overview of our hybrid structure, which augments genetic algorithms with ML based neural networks. of the fitness function. Robustness of SELFIES allows us to do random mutations to molecular strings while preserving their validity. Thus, our mutations only include 50% insertions or 50% replacements of single SELFIES characters. To accelerate the exploration of the GA, we add one domain-specific mutation rule -the direct addition of phenyl groups in approximately 4% of cases. Character deletion is implicitly taken into account in SELFIES mutations, and we do not use crossover rules. The fundamental role of the discriminator is to increase molecular diversity by removing longsurviving molecules. Consider the realistic scenario in which a GA has found a molecule close to a local optimum, where all mutations are lowering the fitness. As a , this molecule survives for multiple generations while occupying a large fraction of the population. During such periods, the GA has limited view of the chemical space as it repeatedly explores mutations of the same high-fitness molecule. A straightforward solution would be the addition of a linear penalty in the fitness function that accounts for the number of successive generations a molecule survives. However, this method assigns independent scores to similar-looking molecules, which again in less variety. Our solution is the addition of an adaptive penalty (in our case a neural network based discriminator), thus resolving the problem of stagnation. Molecules with similar representation receive similar classification scores. Furthermore, long-surviving molecules are trained longer and receive weaker scores, ing in decreasing fitness -reducing the chance of long periods of stagnation (illustrated in Figure 2). The task of the discriminator thus is to memorize families of high performing molecules and penalize their fitness to force the GA to explore different regions in chemical space. Comparing to the literature standard, we aim at maximizing the penalized logP objective J(m) proposed by (Gómez-). For molecule m, the penalized logP function is defined as where logP indicates the water-octanol partition coefficient, SA represents the synthetic accessibility and prevents the formation of chemically unfeasible molecules, and RingPenalty linearly penalizes the presence of rings of size larger than 6. Our reference data set consists of 250,000 commercially available molecules extracted from the ZINC database . All three quantities in the above equation are normalized based on this data set. 5.6 ± 0.5 RNN GCPN 12.61 ± 0.81 GA GA + D (here) 13.31 ± 0.63 20.72 ± 3.14) GA + DNN 1 average of three best molecules after performing Bayesian optimization on the latent representation of a trained VAE 2 two best molecules of a single run 3 averaged over 10 runs with 20 molecules per generation with a molecular weight (excl. hydrogen) smaller than 39.15 ± 3.50 g/mol run for 50 generations 4 averaged over 10 runs with 500 molecules up to 81 characters per generation and 100 generations 5 averaged over 5 runs with 1000 generations each We define our fitness function according to Eq. 1 and 2 with β = 0 and 10. The algorithm is run for 100 generations with a population size of 500. All generated molecules are constrained to a canonical smile length of 81 characters (as in ). We train the discriminator (fully connected neural network with ReLU activation and sigmoid output layer, the input is a vector of chemical and geometrical properties characterizing the molecules) at the end of each generation for 10 epochs on 500 molecules proposed by the GA and 500 randomly drawn molecules from the ZINC data set. We report maximum J(m) scores, averaged over 10 independent runs. The highest J(m) achieved by our approach are 13.31 ± 0.63 (β = 10) and 12.61 ± 0.81 (β = 0), respectively, which is almost twice as high es the highest literature value of 7.87±0.07 (See Table 1). Furthermore, we compare to 50,000 random valid SELFIES strings, which surprisingly outperforms some existing generative models. The GA-D(t) are explained in the next section. In Figure 3, we show the of runs where we use a time-dependent adaptive penalty. During periods of saturation, the weight of the discriminator predictions is switched from 0 to 1000 until stagnation is overcome. The genetic algorithm is hence forced to propose new families of molecules to increase J(m). As it can be observed in Figure 3, even after steep decreases in max J(m), the scores recover and potentially reach values higher than in previous plateaus. We observe that this approach significantly outperforms all previous methods in maximizing the objective J(m). Visual inspection of the highest performing molecules shows that the GA is exploiting deficiencies of the (penalized) logP metric by generating chemically irrelevant motifs such as sulfur chains. While being of limited relevance for application, this nonetheless shows the that the GA is very efficient in exploring the chemical space and finding solutions for a given task. Analysis of these solutions will help us to better understand and eventually improve objective functions. At the same time, surprising solutions found by an unbiased algorithm can lead to unexpected discovery or boost human creativity . While exploitative tasks that follow some reference database are significant for questions involving drug design, more explorative behaviour could be beneficial in domains such as solar cell design , flow battery design , and in particular for questions where datas sets are not available, such as design of targets in molecular beam interferometry . Figure 4 shows classes of molecules explored by the GA in a trajectory with 1000 generations and a generation size of 500 (see trajectories in Figure 3). A K-means clustering analysis based in the RDKit fingerprint bit-vectors with 20 clusters was used to automatically generate labels for the 50 best performing molecules (in terms of their properties J(m)) in each generation. We find that the algorithm starts with a class of relatively small molecules (≈ 40 generations, see class 10), after which the algorithm explores several different classes of large molecules with e.g. aromatic rings and conjugates carbon chains (see class 15) and long linear sulfur chains (see class 1). Class 1 includes the molecules with the highest J(m) scores found in this work, exceeding values of 20. The clustering analysis furthermore helps to analyze the large number of generated molecules. This allows us to learn from the data and derive design rules to achieve high target properties. In case of the penalized logP score, we find that representative examples (short distance to the cluster center) of high performing classes contain aromatic rings, conjugated carbon chains and linear sulfur chains, which can act as design rules for molecules with high penalized logP scores. Using these design rules to construct molecules of the maximally allowed length consisting of only sulfur chains, conjugated carbon chains and chains of benzene rings with sulfur bridges yields J(m) scores of 31.79, 8.58 and 10.02, respectively. The J(m) score reached by the linear sulfur chain outperforms the best scores found by the GA and other generative models. To visualize the explorative behaviour of the GA, we did a two-dimensional principal component analysis (PCA) of the molecular fingerprint vectors of all molecules generated by the GA-D(t) in the trajectory shown in Figure 4. Five snapshots of the trajectory are shown in Figure 5, colored according to their chemical family. The property score J(m) of the trajectory is visualized in the images. We find that the GA sequentially explores different parts of the 2D projection of the chemical space, finding different families of high performing molecules, while the discriminator prevents the algorithm from searching in the same space repeatedly. In the previous sections, we aimed to maximize the penalized logP objective. Here, we consider two different tasks: firstly, generating molecules of specific chemical interest and secondly, modifying existing molecules to increase penalized logP scores. We used β = 0 throughout Section 4.4. We run 250 instances of randomly selected sets of target properties (logP in the interval from -5 to 10, SA scores in the range from 1 to 5 and ring penalty for 0 to 3). The fitness function is modified to minimize the summed squared difference between the actual and desired properties. We compute the number of experiments that successfully proposed molecules with a squared difference less than 1.0. Each run is constrained to run for 100 generations with a maximum canonical SMILES length of 81 characters. In 90.0% of the cases, our approach proposes the desired molecules. Improvement Success Improvement Success JT-VAE (a) 0.84 ± 1.45 83.6% 0.21 ± 0.71 46.4% GCPN 2.49 ± 1.30 100.0% 0.79 ± 0.63 100.0% MMPA (b) 3.29 ± 1.12 -1.65 ± 1.44 -DEFactor 3.41 ± 1.67 85.9% 1.55 ± 1.19 72.6% VJTNN (b) 3.55 ± 1.67 -2.33 ± 1.17 -GA (here) 5.93 ± 1.41 100.0% 3.44 ± 1.09 99.8% Figure 6: Distributions of logP and QED for a) ZINC and b) GuacaMol data set compared to molecules generated using the GA. c) Examples of molecules generated by the GA with high logP and QED scores. In this experiment, we follow the experimental setup proposed by. We optimize the penalized logP score of 800 low-scoring molecules from the ZINC data set. Our genetic algorithm is initiated with a molecule from the data set, and we run each experiment for 20 generations and a population size of 500 without the discriminator. For each run, we report the molecule m that increases the penalized logP the greatest, while possessing a similarity sim(m, m) > δ with the respective reference molecules m. We calculate molecular similarity based on Morgan Fingerprints of radius 2. To ensure generation of molecules possessing a certain similarity, for molecule m we modify the fitness to: Here, SimilarityPenalty(m) is 0 if sim(m, m) > δ and −10 6 otherwise. In Table 2, we report the average improvement for each molecule. Success is determined when the GA successfully improve upon the penalized logP score, while not violating the similarity constraint. Figure S1 shows examples of improved molecular structures. To show the performance of the GA on a drug-discovery task , we modified the objective function to include the drug-likeness score QED . Solubility metric logP and drug-likeness cannot be maximized at the same time, which is shown in Figure 6 at the example of the ZINC and the GuacaMol data set . The experiment shows that the GA is able to efficiently and densely sample the edge of the property distributions of both data sets. Example molecules that simultaneously maximize logP and QED are shown in Figure 6c. The definition of the fitness function used in this work (see Equation 1) has a free parameter (β), that balances the relative importance of molecular target properties J(m) and discriminator score Figure 7 illustrates the of this test, where each curve is obtained from averaging five independent runs with equal settings. In Figure 7a, we show the average property score J(m) for 11 different values β and observe, as expected, that low values of β on average lead to higher J(m) score while very high values of β limit the algorithm to stay inside the reference data set distribution, which, per definition, has a mean J(m) score of 0. Figure 7b illustrates the low average discriminator scores encountered after a few generations in case of low values of β. The curves shown in Figure 7a and b suggest that β allows us to interpolate between high and low J(m) and high and low resemblance with the reference data set. However, as shown in Figure 7c, this is not the case. At values of β ≈ 20, there is a rather fast transition from molecules with high values of J(m) of approximately 12-13 to a J(m) distribution centered around 0 which is similar to the J(m) distribution of the reference data set. The same abrupt change is visible in the distributions of the discriminator scores. Only at β ≈ 18, we find a slightly wider distribution of intermediate J(m) values, which we assume is partially related to the fact that the β = 18 curve shown in Figure 7a is not converged yet. Figure 7d shows examples of molecules generated with β = 0 (upper panel) and β = 50 (lower panel). While in the first case, the GA finds large molecules with many aromatic rings and long linear chains, that do not resemble the molecules in the reference data set, the latter case shows molecules with structures and properties resembling the reference data set. In this case, the distribution of J(m) is comparable to that of the data set (with 0 mean and unit standard deviation), while the discriminator encounters difficulty in correctly classification of the GA molecules, indicated by a mean discriminator score varying around 0.5 across many generations. We presented a hybrid GA and ML-based generative model and demonstrated its application in molecular design. The model outperforms literature approaches in generating molecules with desired properties. A detailed analysis of the data generated by the genetic algorithm allowed us to interpret the model and learn rules for the design of high performing molecules. This human expert design inspired from GA molecules outperformed all molecules created by generative models. For computationally more expensive property evaluations, we will extend our approach by the introduction of an on-the-fly trained ML property evaluation method, which will open new ways of solving the inverse design challenge in chemistry and materials sciences. Our approach is independent of domain knowledge, thus applicable to design questions in other scientific disciplines beyond chemistry. We therefore plan to generalize the GA-D approach to make it a more general concept of generative modelling. 6 SUPPLEMENTARY INFORMATION Figure S1 shows examples of the molecules optimized in Section 4.4. Figure S1: Molecular modifications ing in increased penalized logP scores under similarity constraint sim(m, m) > 0.4, 0.6. We show the molecules that ed in largest score improvement. Figures S2-S4 show comparisons between the property distributions observed in molecule data sets such as the ZINC and the GuacaMol data set and property distributions of molecules generated using random SELFIES (Figure S2), GA generated molecules with the penalized logP objective (Figure S3) and GA generated molecules with an objective function which includes logP and QED (Figure S4). While the average logP scores of average SELFIES are low, the tail of the distribution reaches to high values, explaining the surprisingly high penalized logP scores shown in Table 1. The QED and weight distributions of molecules optimized using the penalized logP objective significantly differ from the distributions of the ZINC and the GuacaMol data set (see Figure S3). As soon as the QED score is simultaneously optimized, the distributions of GA generated molecules and molecules from the reference data sets become more similar (see Figure S4). Figure 6 shows that the GA can simultaneously optimize logP and QED. Figure S2: Distributions of a) logP, b) SA, c) QED and d) molecular weight for randomly generated SELFIES, molecules from the ZINC data set and molecules from the GuacaMol data set. Figure S3: Distributions of a) logP, b) SA, c) QED and d) molecular weight for GA generated SELFIES (penalized logP objective function), molecules from the ZINC data set and molecules from the GuacaMol data set. Figure S4: Distributions of a) logP, b) SA, c) QED and d) molecular weight for GA generated SELFIES (logP and QED as objective function), molecules from the ZINC data set and molecules from the GuacaMol data set.

We suggest the sufficient number of bits for representing weights of DNNs and the optimum bits are conservative when solving real problems.

Represent the short summary for finding the corresponding scientific article (long)

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Inspired by the combination of feedforward and iterative computations in the visual cortex, and taking advantage of the ability of denoising autoencoders to estimate the score of a joint distribution, we propose a novel approach to iterative inference for capturing and exploiting the complex joint distribution of output variables conditioned on some input variables. This approach is applied to image pixel-wise segmentation, with the estimated conditional score used to perform gradient ascent towards a mode of the estimated conditional distribution. This extends previous work on score estimation by denoising autoencoders to the case of a conditional distribution, with a novel use of a corrupted feedforward predictor replacing Gaussian corruption. An advantage of this approach over more classical ways to perform iterative inference for structured outputs, like conditional random fields (CRFs), is that it is not any more necessary to define an explicit energy function linking the output variables. To keep computations tractable, such energy function parametrizations are typically fairly constrained, involving only a few neighbors of each of the output variables in each clique. We experimentally find that the proposed iterative inference from conditional score estimation by conditional denoising autoencoders performs better than comparable models based on CRFs or those not using any explicit modeling of the conditional joint distribution of outputs. Based on response timing and propagation delays in the brain, a plausible hypothesis is that the visual cortex can perform fast feedforward BID21 inference when an answer is needed quickly and the image interpretation is easy enough (requiring as little as 200ms of cortical propagation for some object recognition tasks, i.e., just enough time for a single feedforward pass) but needs more time and iterative inference in the case of more complex inputs BID23. Recent deep learning research and the success of residual networks BID9 BID8 point towards a similar scenario BID16: early computation which is feedforward, a series of non-linear transformations which map low-level features to high-level ones, while later computation is iterative (using lateral and feedback connections in the brain) in order to capture complex dependencies between different elements of the interpretation. Indeed, whereas a purely feedforward network could model a unimodal posterior distribution (e.g., the expected target with some uncertainty around it), the joint conditional distribution of output variables given inputs could be more complex and multimodal. Iterative inference could then be used to either sample from this joint distribution or converge towards a dominant mode of that distribution, whereas a unimodaloutput feedfoward network would converge to some statistic like the conditional expectation, which might not correspond to a coherent configuration of the output variables when the actual conditional distribution is multimodal. This paper proposes such an approach combining a first feedforward phase with a second iterative phase corresponding to searching for a dominant mode of the conditional distribution while tackling the problem of semantic image segmentation. We take advantage of theoretical BID0 on denoising autoencoder (DAE), which show that they can estimate the score or negative gradient of the energy function of the joint distribution of observed variables: the difference between the reconstruction and the input points in the direction of that estimated gradient. We propose to condition the autoencoder with an additional input so as to obtain the conditional score, Given an input image x, we extract a segmentation candidate y and intermediate feature maps h by applying a pre-trained segmentation network. We add some noise to y and train a DAE that takes as input both y and h by minimizing Eq. 3. Training scenario 1 (a) yields the best and uses the corrupted prediction as input to the DAE during training. Training scenario 2 (b) corresponds to the original DAE prescription in the conditional case, and uses a corruption of the ground truth as input to the DAE during training.i.e., the gradient of the energy of the conditional density of the output variables, given the input variables. The autoencoder takes a candidate output y as well as an input x and outputs a valueŷ so thatŷ − y estimates the direction DISPLAYFORM0. We can then take a gradient step in that direction and update y towards a lower-energy value and iterate in order to approach a mode of the implicit p(y|x) captured by the autoencoder. We find that instead of corrupting the segmentation target as input of the DAE, we obtain better by training the DAE with the corrupted feedforward prediction, which is closer to what will be seen as the initial state of the iterative inference process. The use of a denoising autoencoder framework to estimate the gradient of the energy is an alternative to more traditional graphical modeling approaches, e.g., with conditional random fields (CRFs) BID14 BID10, which have been used to model the joint distribution of pixel labels given an image BID13. The potential advantage of the DAE approach is that it is not necessary to decide on an explicitly parametrized energy function: such energy functions tend to only capture local interactions between neighboring pixels, whereas a convolutional DAE can potentially capture dependencies of any order and across the whole image, taking advantage of the state-of-the-art in deep convolutional architectures in order to model these dependencies via the direct estimation of the energy function gradient. Note that this is different from the use of convolutional networks for the feedforward part of the network, and regards the modeling of the conditional joint distribution of output pixel labels, given image features. The main contributions of this paper are the following: 1. A novel training framework for modeling structured output conditional distributions, which is an alternative to CRFs, inspired by denoising autoencoder estimation of energy gradients. 2. Showing how this framework can be used in an architecture for image pixel-wise segmentation, in which the above energy gradient estimator is used to propose a highly probable segmentation through gradient descent in the output space. 3. Demonstrating that this approach to image segmentation outperforms or matches classical alternatives such as combining convolutional nets with CRFs and more recent state-of-theart alternatives on the CamVid dataset. In this section, we describe the proposed iterative inference method to refine the segmentation of a feedforward network. As pointed in section 1, DAE can estimate a density p(y) via an estimator of the score or negative gradient − ∂E ∂y of the energy function E BID25 BID24 BID0. These theoretical analyses of DAE are presented for the particular case where the corruption noise added to the input is Gaussian. Results show that DAE can estimate the gradient of the energy function of a joint distribution of observed variables. The main is the following: DISPLAYFORM0 where σ 2 is the amount of Gaussian noise injected during training, y is the input of the autoencoder and r(y) is its output (the reconstruction). The approximation becomes exact as σ → 0 and the autoencoder is given enough capacity, training examples and training time. The direction of (r(y) − y) points towards more likely configurations of y. Therefore, the DAE learns a vector field pointing towards the manifold where the input data lies. In our case, we seek to rapidly learn a vector field pointing towards more probable configurations of y|x. We propose to extend the summarized in subsection 2.1 and condition the autoencoder with an additional input. If we condition the autoencoder with features h, which are a function of x, the DAE framework with Gaussian corruption learns to estimate DISPLAYFORM0, where y is a segmentation candidate, x an input image and E is an energy function. Gradient descent in energy can thus be performed in order to iteratively reach a mode of the estimated conditional distribution: DISPLAYFORM1 with step size. In addition, whereas Gaussian noise around the target y true would be the DAE prescription for the corrupted input to be mapped to y true, this may be inefficient at visiting the configurations we really care about, i.e. those produced by our feedforward predictor, which we use to obtain a first guess for y, as initialization of the iterative inference towards an energy minimum. Therefore, we propose that during training, instead of using a corrupted y true as input, the DAE takes as input a corrupted segmentation candidate y and either the input x or some features h extracted from a feedforward segmentation network applied to DISPLAYFORM2, where f k is a non-linear function and l ∈ {1, ..., L} is the index of a layer in the feedforward segmentation network. The output of the DAE is computed asŷ = r(ỹ, h), where r is a non-linear function which is trained to denoise conditionally andỹ is a corrupted form of y. During training,ỹ is y plus noise, while at test time (for inference) it is simply y itself. In order to train the DAE, we extract both y and h from a feedforward segmentation network; we corrupt y intoỹ; and we train the DAE by minimizing the following loss DISPLAYFORM3 where H is the categorical cross-entropy and y true is the segmentation ground truth. Figure 1(a) depicts the pipeline during training. First, a fully convolutional feedforward network for segmentation is trained. In practice, we use one of the state-of-the-art pre-trained networks. Second, given an input image x, we extract a segmentation proposal y and intermediate features h from the segmentation network. Both y and h are fed to a DAE network (adding Gaussian noise to y). The DAE is trained to properly reconstruct the clean segmentation (ground truth y true). FIG0 (b) presents the original DAE prescription, where the DAE is trained by taking as input y true and h. Once trained, we can exploit the trained model to iteratively take gradient steps in the direction of the segmentation manifold. To do so, we first obtain a segmentation proposal y from the feedforward network and then we iteratively refine this proposal by applying the following rule DISPLAYFORM4 For practical reasons, we collapsed the corruption noise σ 2 into the step size. Given an input image x, we extract a segmentation candidate y and intermediate feature maps h by applying a pre-trained segmentation network. We then feed x and h to the trained DAE and iteratively refine y by applying Eq. 4. The final prediction is the last value of y computed in this way. Figure 2 depicts the test pipeline. We start with an input image x that we feed to a pre-trained segmentation network. The segmentation networks outputs some intermediate feature maps h and a segmentation proposal y. Then, both y and h are fed to the DAE to compute the outputŷ. The DAE is used to take iterative gradient steps y = y − (y −ŷ) towards the manifold of segmentation masks, with no noise added at inference time. On one hand, recent advances in semantic segmentation mainly focus on improving the architecture design BID19 BID1 BID4 BID12, increasing the context understanding capabilities BID6 BID26 BID3 BID28 and building processing modules to enforce structure consistency to segmentation outputs BID13 BID3 BID31. Here, we are interested in this last research direction. CRFs are among the most popular choices to impose structured information at the output of a segmentation network, being fully connected CRFs BID13 and CRFs as RNNs BID31 among best performing variants. More recently, an alternative to promote structure consistency by decomposing the prediction process into multiple steps and iteratively adding structure information, was introduced by BID15. Another iterative approach was introduced by BID7 to tackle image semantic segmentation by repeatedly detecting, replacing and refining segmentation masks. Finally, the reinterpretation of residual networks BID16 BID8 was exploited by, in the context of biomedical image segmentation, by iteratively refining learned pre-normalized images to generate segmentation predictions. On the other hand, there has recently been some research devoted to exploit of DAE on different tasks, such as image generation BID18, high resolution image estimation BID20 and semantic segmentation BID27. BID18 propose plug & play generative networks, which, in the best reported , train a fully-connected DAE to reconstruct a denoised version of some feature maps extracted from an image classification network. The iterative update rule at inference time is performed in the feature space. Sønderby et al. FORMULA1 use DAE in the context of image super-resolution to learn the gradient of the density of high resolution images and apply it to refine the output of an upsampled low resolution image. BID27 exploit convolutional pseudo-priors trained on the ground-truth labels in semantic segmentation task. During the training phase, the pseudo-prior is combined with the segmentation proposal from a segmentation model to produce joint distribution over data and labels. At test time, the ground truth is not accessible, thus feedforward segmentation predictions are fed iteratively to the convolutional pseudo-prior network. In this work, we exploit DAEs in the context of image segmentation and extend them in two ways, first by using them to learn a conditional score, and second by using a corrupted feedforward prediction as input during training to obtain better segmentations. The main objective of these experiments is to answer the following questions: Can a conditional DAE be used successfully as the building block of iterative inference for image segmentation? Does our proposed corruption model (based on the feedforward prediction) work better than the prescribed target output corruption? Does the ing segmentation system outperform more classical iterative approaches to segmentation such as CRFs?4.1 CAMVID DATASET CamVid 1 BID2 ) is a fully annotated urban scene understanding dataset. It contains videos that are fully segmented. We used the same split and image resolution as BID1. The split contains 367 images (video frames) for training, 101 for validation and 233 for test. Each frame has a resolution of 360x480 and pixels are labeled with 11 different classes. We experimented with two feedforward architectures for segmentation: the classical fully convolutional network FCN-8 of BID17 and the more recent state-of-the-art fully convolutional densenet (FC-DenseNet103) of BID12, which do not make use of any additional synthetic data to boost their performances. BID17: FCN-8 is a feedforward segmentation network, which consists of a convolutional downsampling path followed by a convolutional upsampling path. The downsampling path successively applies convolutional and pooling layers, and the upsampling path successively applies transposed convolutional layers. The upsampling path recovers spatial information by merging features skipped from the various resolution levels on the downsampling path. BID12: FC-DenseNet is a feedforward segmentation network, that exploits the feature reuse idea of and extends it to perform semantic segmentation. FC-DenseNet103 consists of a convolutional downsampling path, followed by a convolutional upsampling path. The downsampling path iteratively concatenates all feature outputs in a feedforward fashion. The upsampling path applies a transposed convolution to feature maps from the previous stage and recovers information from higher resolution features from the downsampling path of the network by using skip connections. Our DAE is composed of a downsampling path and an upsampling path. The downsampling path contains convolutions and pooling operations, while the upsampling path is built from unpooling with switches (also known as unpooling with index tracking) BID30 BID1 and convolution operations. As discussed in, reverting the max pooling operations more faithfully, significantly improves the quality of the reconstructed images. Moreover, while exploring potential network architectures, we found out that using fully convolutional-like architectures with upsampling and skip connections (between downsampling and upsampling paths) decreases segmentation when compared to unpooling with switches. This is not surprising, since we inject noise to the model's input when training the DAE. Skip connections directly propagate this added noise to the end layers; making them responsible for the data denoising process. Note that the last layers of the model might not have enough capacity to accomplish the denoising task. In our experiments, we use DAE built from 6 interleaved pooling and convolution operations, followed by 6 interleaved unpooling and convolution operations. We start with 64 feature maps in the first convolution and duplicate the number of feature maps in consecutive convolutions in the downsampling path. Thus, the number of feature maps in the networks downsampling path is: 64, 128, 256, 512, 1024 and 2048. In the upsampling path, we progressively reduce the number of feature maps up to the number of classes. Thus, the number of feature maps in consecutive layers of the upsampling path is the following: 1024, 512, 256, 128, 64 and 11 (number of classes). We concatenate the output of 4th pooling operation in downsampling path of DAE together with the feature maps h corresponding to 4th pooling operation in downsampling path of the segmentation network. We train our DAE by means of stochastic gradient descent with RMSprop BID22, initializing the learning rate to 10 −3 and applying an exponential decay of 0.99 after each epoch. All models are trained with data augmentation, randomly applying crops of size 224 × 224 and horizontal flips. We regularize our model with a weight decay of 10 −4. We use a minibatch size of 10. While training, we add zero-mean Gaussian noise (σ = 0.1 or σ = 0.5) to the DAE input. We train the models for a maximum of 500 epochs and monitor the validation reconstruction error to early stop the training using a patience of 100 epochs. At test time, we need to determine the step size and the number of iterations to get the final segmentation output. We select and the number of iterations by evaluating the pipeline on the validation set. Therefore, we try ∈ {0.01, 0.02, 0.05, 0.08, 0.1, 0.5, 1} for up to 50 iterations (iteration ∈ {1, 2, ..., 50}). For each iteration, we compute the mean intersection over union (mean IoU) on the validation set and keep the combination (, number of iterations) that maximizes this metric to evaluate the test set. TAB0 reports our for FCN-8 and FC-DenseNet103 without any post-processing step, applying fully connected CRF BID13, context network BID28 as trained post-processing step, CRF-RNN BID31 trained end-to-end with the segmentation network and DAE's iterative inference. For CRF, we use publicly available implementation of BID13.As shown in the table, using DAE's iterative inference on the segmentation candidates of a feedforward segmentation network (DAE(y)) outperforms state-of-the-art post-processing variants; improving upon FCN-8 by a margin of 3.0% IoU. When applying CRF as a post-processor, the FCN-8 segmentation improve 1.2%. Note that similar improvements for CRF were reported on other architectures for the same dataset (e.g. BID1). Comparable improvements are achieved when using the context module BID28 as post-processor (1.3%) and when applying CRF-RNN (1.7%). It is worth noting that our method does not decrease the performance of any class with respect to FCN-8. However, CRF loses 2.8% when segmenting column poles, whereas CRF-RNN loses 1.1% when segmenting signs. When it comes to more recent state-ofthe-art architectures such as FC-DenseNet103, the post-processing increment on the segmentation metrics is lower, as expected. Nevertheless, the improvement is still perceivable (+ 0.5% in IoU). When comparing our method to other state-of-the-art post-processors, we observe a slight improvement. End-to-end training of CRF-RNN with FC-DenseNet103 did not yield any improvement over FC-DenseNet103.It is worth comparing the performance of the proposed approach DAE(y) with DAE(y true) trained from the ground truth. As shown in the table, DAE(y) consistently outperforms DAE(y true). For FCN-8, the proposed method outperforms DAE(y true) by a margin of 2.2%. For FC-DenseNet103, differences are smaller but still noticeable. In both cases, DAE(y) not only outperforms DAE(y true) globally, but also in the vast majority of classes that exhibit an improvement. Note that the model trained on the ground truth requires a bigger Gaussian noise σ in order to slightly increase the performance of the pre-trained feedforward segmentation networks. It is worth mentioning that training our model end-to-end with the segmentation network didn't improve the , while being more memory demanding. FIG4 (d), the FCN-8 segmentation network fails to properly find the fence in the image, mistakenly classifying it as part of a building (highlighted with a white box on the image). CRF is able to clean the segmentation candidate, for example, by filling in missing parts of the sidewalk but is not able to add non-existing structure (see FIG4 (e)). Our method not only improves the segmentation candidate by smoothing large regions such as the sidewalk, but also corrects the prediction by incorporating missing objects such as the fence on FIG4 In this subsection, we analyze the influence of the two inference parameters of our method, namely the step size and the number of iterations. This analysis is performed on the validation set of CamVid dataset, for the above-mentioned feedforward segmentation networks. For the sake of comparison, we perform a similar analysis on densely connected CRF; by fixing the best configuration and only changing the number of CRF iterations. plot the in the case of FCN-8 and FC-DenseNet103, respectively. As expected, there is a trade-off between the selected step size and the number of iterations. The smaller the, the more iterations are required to achieve the best performance. Interestingly, all within a reasonable range lead to similar maximum performances. We have proposed to use a novel form of denoising autoencoders for iterative inference in structured output tasks such as image segmentation. The autoencoder is trained to map corrupted predictions to target outputs and iterative inference interprets the difference between the output and the input as a direction of improved output configuration, given the input image. Experiments provide positive evidence for the three questions raised at the beginning of Sec. 4: a conditional DAE can be used successfully as the building block of iterative inference for image segmentation, the proposed corruption model (based on the feedforward prediction) works better than the prescribed target output corruption, and the ing segmentation system outperforms state-of-the-art methods for obtaining coherent outputs.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Refining segmentation proposals by performing iterative inference with conditional denoising autoencoders.

represent the short summary for finding the corresponding scientific article (long)

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.


Question: Represent the scientific article to find a short TLDR

Question: Inspired by the success of self attention mechanism and Transformer architecture in sequence transduction and image generation applications, we propose novel self attention-based architectures to improve the performance of adversarial latent code- based schemes in text generation. Adversarial latent code-based text generation has recently gained a lot of attention due to their promising . In this paper, we take a step to fortify the architectures used in these setups, specifically AAE and ARAE. We benchmark two latent code-based methods (AAE and ARAE) designed based on adversarial setups. In our experiments, the Google sentence compression dataset is utilized to compare our method with these methods using various objective and subjective measures. The experiments demonstrate the proposed (self) attention-based models outperform the state-of-the-art in adversarial code-based text generation. Text generation is of particular interest in many natural language processing (NLP) applications such as dialogue systems, machine translation, image captioning and text summarization. Recent deep learning-based approaches to this problem can be categorized into three classes: auto-regressive or maximum likelihood estimation (MLE)-based, generative adversarial network (GAN)-based and reinforcement learning (RL)-based approaches. BID26 ) model the text (language) as an auto-regressive process, commonly using RNNs. RNNs compactly represent the samples history in the form of recurrent states. In these models, text is generated by predicting next token (character, word, etc) based on the previously generated ones BID9. One of the main challenges involved with auto-regressive methods is exposure bias BID3. This problem arises due to discrepancy between the training and generation phases. In fact, ground-truth samples from the past are used in training, while past generated ones are used in generation. A number of solutions have been proposed to address this problem by modifying the training procedure including scheduled sampling BID3, Gibbs sampling BID25, and Professor forcing BID16.Over the past few years, researchers have extensively used GANs BID8 as a powerful generative model for text BID29 BID5, inspired by the great success in the field of image generation. GANs are believed to be capable of solving the exposure bias problem in text generation raised from using MLE. The reason is that they solved a similar issue of blurry image generation in MLE-based variational autoencoders (VAEs). It is belived that the discriminator is able to guide the text generator, through their training exchange, how to generate samples similar to real (training) data. However, there are other challenges involved in GAN-based text generation. A few of these challenges in text generation are inherent to GANs themselves, such as mode collapse and training instability. The mode collapse problem happens when the adversarially trained generator does not produce diverse texts. These issues can be mitigated by using well-known techniques such as feature matching, and entropy regularization BID24. Another challenge is due to the discrete nature of text, which causes the generator sampling to be non-differentiable over the categorical distribution of the words. In this paper, we take advantage of Transformer self-attention mechanism BID27 and incorporate it in two state-of-the-art adversarial latent code-based schemes proposed for text generation. More specifically:• We incorporate the Transformer structure in the design of encoder and decoder blocks of AAE BID17 and ARAE BID14 setups for text generation.• Blocks closely inspired from the Transformer's encoder layers, incorporating self-attention and element-wise fully-connected layers in a residual configuration and with positional encodings, are used along with spectral normalization to propose a novel GAN (both generator and discriminator) structure for AAE and ARAE setups.• The performance improvement obtained from the proposed architectures is demonstrated via objective and subjective measures used in extensive experiments.2 RELATED WORK 2.1 SPECTRAL NORMALIZATION Spectral normalization BID19 ) is a weight normalization method proposed to stabilize the training of GANs. The authors show that the Lipshitz norm of a neural networks can be bounded by normalizing the spectral norm of layer weight matrices. As opposed to local regularizations used in WGAN-GP, etc., the network-wide spectral regularization stabilizes the GAN training, produces more diverse outputs and in higher inception scores. We use spectral normalization in our adversarial setups for the same reasons. In sequence modeling literature, attention was initially proposed by BID2. It recognizes the fixed-length latent representation of the input sequence as the main performance bottleneck in the seq-to-seq models and proposed using soft-attention in the decoder. Using attention, the decoder can also attend to any desired token in the input sequence besides consuming the compressed representation ing at the end of encoding operation. Self-attention was initially proposed for language inference in BID21. The authors named it as "intra-attention" and showed that their structure can be an effective alternative for LSTMs in the task of natural language inference BID4, at the time achieving state of the art performance with much fewer parameters as well as requiring a training time an order of magnitude shorter. Self-attention structures have since been used to set the state of the art in a number of different tasks BID27 BID7 BID22 BID0. They drastically reduce the path length between any two sequence inputs, making the learning of long term dependencies much easier BID27. They are considerably easier to parallelize, reducing the number of operations that are required to be sequential. Recently, applied self attention along with spectral normalization to the task of image generation using GANs. It showed by visualization that using attention, the generator can attend to far neighborhoods of any shape rather than close-by fixed-shape ones at each level in a hierarchical generation. The authors claim that applying spectral normalization to generator as well as discriminator helps training dynamics (stability). Similarly, we also adopt self attention and spectral normalization in our architecture designs. Transformer BID27 extended the use of self attention mechanism and was proved to be the state-of-the-art in sequence transduction applications such as machine translation. It dispenses convolutional and recurrent layers and relies entirely on attention-only layers and element-wise feed forward layers. One of the main challenges of the language generation task originates from the discrete nature of text. Similarly to generating other discrete tokens, the back propagation of error through argmax operator is not well-defined. To address this problem, various approaches have been proposed in the literature including continuous approximation of discrete sampling BID10 BID13, using policy gradient from reinforcement learning BID11 BID24, etc. One of the most successful solutions is based on autoencoders with continuous latent spaces (i.e. latent code-based methods). Various training setups have been proposed for training these autoencoders including adversarial BID14 and variational BID12 setups. A recent paper BID6 ) performs a thorough review of the state-of-the-art latent codebased text generation methods. It studies the performance of a number of code-based text generation schemes and uses a unified rigorous evaluation protocol to evaluate them. We got inspired by their evaluation protocol to demonstrate the strength of our self attention-based approach in the context. They use a broad set of measures to perform a comprehensive study. We adopt forward and reverse perplexity as well as BLEU from their objective measures and fluency from the subjective ones. In this section, we briefly explain two prominent baseline methods using adversarial latent code-based generation techniques and present the technical details in Section 3. Adversarial autoencoder (AAE) BID18 proposes an adversarial setup to train probabilistic autoencoders. It matches the aggregated posterior of the encoder output (latent codes) to an arbitrary distribution that can be easily sampled from. Although authors demonstrate the applications of their setup in semi-supervised learning, style and content disentanglement, etc, AAE decoder can be effectively used as a generative model, converting samples of the arbitrary distribution (noise) to real-like outputs. From application perspective, authors only evaluated AAE performance in vision-related applications. In this paper, we tailor AAE for text generation, following guidelines proposed by BID6 and incorporate self attention and Transformer as novel parts in the model. The adversarially regularized autoencoder (ARAE) BID15 learns an autoencoder with continuous contracted codes that highly correlate with discrete inputs. That is, similar inputs get encoded (mapped) to nearby continuous codes. ARAE aims at exploiting GAN's ability to force the generator to output continuous codes corresponding to the code space obtained from encoding the real text data. By matching the outputs of generator and encoder, ARAE provides an implicit latent code GAN that serves as a generative model for decoding text. In this section, we explain the details of our self attention-based models following the ARAE and AAE setups proposed in BID6. These setups have shown comparable to the state-of-theart in text generation. We select similar setups to provide fair comparisons and report the best techniques/parameters based on our experiments. In our architectures, Transformer BID27 is used in designing all autoencoders. In both encoder and decoder, we use three blocks of Transformer.' Block' and'layer' names are used, respectively, instead of'layer' and'sub-layer' in the original paper. Layer normalization BID1 ) is applied on every layer (multi-head attention, masked multihead attention and feed forward layers) within each Transformer block. Multi-head attentions have eight heads and embedding layers are of size 304 (a multiple of eight). Similarly to BID27, positional encoding is used at the very first layer of the encoder and decoder. The dimensions and encoding place were found empirically for the best objective and subjective performance. For GAN structures, i.e. the generator and discriminator architectures, we use modified Transformer encoder layers combined with spectral normalization, as depicted in FIG1. As in the regular transformer blocks, all connections are residual. Inspired by spectral normalization successes in the Figure 1: SALSA-TEXT generator and discriminator architecture designed using Transformer encoder structure GAN-based image generation, especially proved in SAGAN, we apply it to the weights of the discriminator and the generator in our network. We did not find layer normalization (used in original Transformer) to be useful, when applied along with spectral normalization in the generator and discriminator architectures. Hence, only use spectral normalization in our GAN structures. We use self attention-based structures in two well-known adversarial setups BID17 and BID14 ).AAE We use the AAE-SPH setup used in BID6. It is based on the original setup proposed in BID17. The discriminator forces encoder outputs to follow a uniform distribution on the unit sphere. Similarly to BID17, a two-phase training is used, where there is regular alternation between minimizing reconstruction and adversarial (regularization) costs. The trade-off factor (λ) between reconstruction and adversarial costs is 20 (as in BID6). All over the encoder, decoder and discriminator, input and attention heads are dropped with a probability of 0.1. The general architecture and the proposed (self) attention-based changes are depicted in FIG0.ARAE We use the original setup from BID14 with fixed-size full codes as inputs to the decoder. Inside the encoder and decoder, word and attention head dropout is performed with a probability of 0.1 and a maximum of 3-word shift is applied to input words. The general architecture and the proposed (self) attention-based changes are depicted in FIG1. We study the performance of our self attentive (SALSA) architectures and compare it with that of the code-based setups studied in BID6. The performance of the models is evaluated in sentence generation (sampling), on the Google Sentence Compression (GSC) dataset 1 (as in BID6). Training on this dataset is very challenging as the sentences are relatively long (average of 24.8 words) and diverse in terms of content, grammar, etc. GSC comprises 200, 000 training and 9, 995 test sentences. For all the trained models, we use Google's SentencePiece 2 tokenizer using byte-pair encoding (BPE BID23) as in BID6. − The world's largest car maker, said it will buy back a new US$4 million fine for the first time in three years. − London, May 6 A 48-year-old man has been charged with raping a woman in the face of livelihood on Tuesday. − In a bid to save money, the US government's most expensive land. − The Governors of the city of Caterpillar, who was in talks with the world's most expensive officials. − Israel has launched a new website that would help the Gaza Strip, and the first such incident in a deadly attack on the West Bank and the United States. − Harrington has been found guilty of two counts of driving under the influence of a semi-final at a local court. ARAE − A man has been arrested and charged with sexual assault on an alleged assault and assault for allegedly assaulting him to death and a child. − A man is accused of stealing a child in her own home case of her husband. − A man, who was accused of killing the two-year-old girl in connection with several of them who died from injuries last week. − A man is facing charges of sexual assault for allegedly assaulting a woman and her wife. − A man accused of killing the man in his home is being sentenced to life and is expected to be on the way. − A man has been arrested in connection with her death and then sex with her husband, who was in critical condition and will be released on Monday. − Former PSV Ethanolve said he has "two and the title of his contract," the first-round pick of the season. − The Queen will present a "Curprone of the Donegal Plan" in a new biopic of the forthcoming musical, and "Kalways," the school said in a statement. − This week between the "Daily Show" flights, a year after the airing, a day before the launch of a summer vacation. − However, the Myssenon, is planning to open its first wedding, with a move that will include a number of its customers, but they are not planning to sell their services. − The Dallas Morning News reported that a Houston man is in "a new, a state that is leaving the Houston Rockets. − The Dallas Morning News Corp. said it is to open a subsidiary of Houston, the largest newspaper and its staff, to be in the city. We filter the dataset to only include sentences with a maximum of 50 PBE tokens. This only lowers the average number of words per sentence and total number of sentences to 23.1 and 183739, respectively in the training set. The test dataset is also reduced to 9254 lines with an average of 22.7 words per sentence. Samples of generated sentences from all models is listed in Section 4.3.The input noise to the generator is of size 100 (as in BID6). We upsample the noise to the embedding size of 304 by using a fully connected layer. The same upsampled noise is copied a number of times equal to the maximum number of steps in the sentence. We use T = 50 times in our experiments. The noise is then fed to the generator, where positional encodings are added to each step. The previously mentioned fully connected layer also serves to allow the model to learn to protect the information of the positional encodings from the noise. Positional encodings are also added at the start of each transformer encoder block. As we use fixed size sequences, the attention depth is always fixed (T bpe). Positional encodings are also added to the input of each transformer encoder block, inside of the generator. We use various objective and subjective measures to evaluate the models. As objective measures, we use BLEU BID20, Self-BLEU BID32, forward and reverse perplexity. BLEU BID20 ) is a widely used metric to compute the similarity of a set of generated sentences with a reference dataset. The are described in TAB1.Self BLEU BID32 (Table 3) is a measure of diversity for generated texts. In Self-BLEU, for each generated sentence, we compute the BLEU using the sentence as hypothesis and the rest of the generated sentences as the reference. When averaged over all the references, it gives us a measure of how diverse the sentences are. Lower Self-BLEU scores are better, as high BLEU scores indicate great similarity. In Perplexity evaluation TAB2, the goal is to measure the individual quality of the sentences generated. We train an LSTM language model on the WMT News 2017 Dataset 3 filtered for lines of a maximum of 50 BPE tokens (a total of 200000 sentences). The perplexity of the language model is computed over 100000 generated sentences for each model. Reverse perplexity evaluation TAB2 aims to measure variety of the generated sentences. For each model, we train an LSTM-based language model based on 100000 generated sentences, and then evaluate the perplexity on the GSC test dataset, filtered to a maximum length of 50 BPE. Diverse generated sentences that cover dataset to a good extent would in better (lower) reverse perplexity measures ing from the trained LSTM network (language model).For the subjective evaluation TAB3, we use Amazon mechanical Turk 4 online platform. 18 sentences are sampled from each model, i.e. a total of 162 sentences. We assign 81 randomly selected sentences to 50 native English speakers (among Mechanical Turk Masters with hit approval ratings greater than 75%). The remaining 81 are assigned to another group of 50 people with the same qualifications. Each person was asked to evaluate the assigned 81 sentences in one and a half hours. In the evaluation, the 5-point Likart scale is used to measure grammaticality, semantic consistency and overall (Fluency). The overall reflects both grammar and semantic consistency in addition to other human-specific factors. Hence, it is a good representative of "Fluency" measure used in BID6. In TAB0, we list six generated sentences for each model. As seen, AAE generates rather short sentences, while the corresponding SALSA version (SALSA-AAE) has alleviated the issue to a good extent. Finally, ARAE suffers from extreme mode collapse as opposed to its SALSA counterpart. The of objective and subjective evaluations are presented in Tables 2 to 5. As seen, the proposed self attention-based (SALSA) architectures consistently outperform the non-attention-based benchmarks in terms of diversity (measured by reverse perplexity). Moreover, they often show better performance in terms of output quality (measured by BLEU, self BLEU, preplexity and human evaluations) on the long and complicated sentences of the GSC dataset. As seen in the generated samples TAB0, human evaluation TAB3 ) and objective metrics (Tables 2 to 4), the original AAE and ARAE setups perform very poorly on GSC with long sentences. With reverse perplexities of over 8000 and high self-BLEU scores close to 0.9, they suffer from a high level of mode collapse (repeated sentences).Human evaluations do not account for lack of diversity. The reason is humans are presented with a number of shuffled sentences and asked to evaluate them independently (without knowing which sentence coming from which model). Hence, in our experiments for the original AAE and ARAE, a model can generate similar sentences (maybe due to mode collapse) and still receives high subjective scores. It seems that, in our experiments, the original ARAE model suffers from mode collapse. We can see that it has slightly higher human evaluation scores, but extremely poor diversity metrics, i.e. very high reverse perplexity and self-BLEU scores. It can also be seen in the randomly selected generated sentences TAB0, where all the sentences start with "A man" and invariably mention he is being arrested or accused of grievous crimes. This is likely because the sentences in the GSC dataset are long and that their structure is elaborate. SALSA-ARAE on the other hand reliably produces sentences of quality with great diversity. SALSA-AAE has both considerably higher individual quality metrics than the original AAE and much better diversity metrics. It is the strongest pure adversarial text model. As seen in TAB3, SALSA-AAE provides the best grammaticality, semantic consistency and Fluency performance. In this paper, we introduced SALSA-TEXT, a Transformer-based architecture for adversarial codebased text generation. It incorporates self-attention mechanism by utilizing Transformer architecture in autoencoder and GAN setups. Our extensive experiments demonstrate the better performance of our models compared to the state-of-the-art in adversarial code-based text generation (without self-attention). The proposed architectures provide diverse, long and high quality output sentences as confirmed by objective metrics and human evaluations in extensive experiments. As a future direction, it is beneficial to study the performance of self attention in other text generation methods including variational code-based and reinforcement learning-based approaches. Another interesting direction is to experiment with deeper Transformer-based autoencoders to better capture the underlying language model and perform unsupervised pre-training isnpired by the success of BID0 and Radford et al..

We propose a self-attention based GAN architecture for unconditional text generation and improve on previous adversarial code-based results.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We revisit self-training as a semi-supervised learning method for neural sequence generation problem, and show that self-training can be quite successful with injected noise.