Question: represent the text to find a very short summary (1-2 sentences max)

Question: The ability to decompose complex multi-object scenes into meaningful abstractions like objects is fundamental to achieve higher-level cognition. Previous approaches for unsupervised object-oriented scene representation learning are either based on spatial-attention or scene-mixture approaches and limited in scalability which is a main obstacle towards modeling real-world scenes. In this paper, we propose a generative latent variable model, called SPACE, that provides a uniﬁed probabilistic modeling framework that combines the best of spatial-attention and scene-mixture approaches. SPACE can explicitly provide factorized object representations for foreground objects while also decomposing segments of complex morphology. Previous models are good at either of these, but not both. SPACE also resolves the scalability problems of previous methods by incorporating parallel spatial-attention and thus is applicable to scenes with a large number of objects without performance degradations. We show through experiments on Atari and 3D-Rooms that SPACE achieves the above properties consistently in comparison to SPAIR, IODINE, and GENESIS. Results of our experiments can be found on our project website: https://sites.google.com/view/space-project-page One of the unsolved key challenges in machine learning is unsupervised learning of structured representation for a visual scene containing many objects with occlusion, partial observability, and complex . When properly decomposed into meaningful abstract entities such as objects and spaces, this structured representation brings many advantages of abstract (symbolic) representation to areas where contemporary deep learning approaches with a global continuous vector representation of a scene have not been successful. For example, a structured representation may improve sample efficiency for downstream tasks such as a deep reinforcement learning agent . It may also enable visual variable binding for reasoning and causal inference over the relationships between the objects and agents in a scene. Structured representations also provide composability and transferability for better generalization. Recent approaches to this problem of unsupervised object-oriented scene representation can be categorized into two types of models: scene-mixture models and spatial-attention models. In scenemixture models (; ;), a visual scene is explained by a mixture of a finite number of component images. This type of representation provides flexible segmentation maps that can handle objects and segments of complex morphology. However, since each component corresponds to a full-scale image, important physical features of objects like position and scale are only implicitly encoded in the scale of a full image and further disentanglement is required to extract these useful features. Also, since it does not explicitly reflect useful inductive biases like the locality of an object in the Gestalt principles , the ing component representation is not necessarily a representation of a local area. Moreover, to obtain a complete scene, a component needs to refer to other components, and thus inference is inherently performed sequentially, ing in limitations in scaling to scenes with many objects. In contrast, spatial-attention models can explicitly obtain the fully disentangled geometric representation of objects such as position and scale. Such features are grounded on the semantics of physics and should be useful in many ways (e.g., sample efficiency, interpretability, geometric reasoning and inference, transferability). However, these models cannot represent complex objects and segments that have too flexible morphology to be captured by spatial attention (i.e. based on rectangular bounding boxes). Similar to scene-mixture models, previous models in this class show scalability issues as objects are processed sequentially. In this paper, we propose a method, called Spatially Parallel Attention and Component Extraction (SPACE), that combines the best of both approaches. SPACE learns to process foreground objects, which can be captured efficiently by bounding boxes, by using parallel spatial-attention while decomposing the remaining area that includes both morphologically complex objects and segments by using component mixtures. Thus, SPACE provides an object-wise disentangled representation of foreground objects along with explicit properties like position and scale per object while also providing decomposed representations of complex components. Furthermore, by fully parallelizing the foreground object processing, we resolve the scalability issue of existing spatial attention methods. In experiments on 3D-room scenes and Atari game scenes, we quantitatively and qualitatively compare the representation of SPACE to other models and show that SPACE combines the benefits of both approaches in addition to significant speed-ups due to the parallel foreground processing. The contributions of the paper are as follows. First, we introduce a model that unifies the benefits of spatial-attention and scene-mixture approaches in a principled framework of probabilistic latent variable modeling. Second, we introduce a spatially parallel multi-object processing module and demonstrate that it can significantly mitigate the scalability problems of previous methods. Lastly, we provide an extensive comparison with previous models where we illustrate the capabilities and limitations of each method. In this section, we describe our proposed model, Spatially Parallel Attention and Component Extraction (SPACE). The main idea of SPACE, presented in Figure 1, is to propose a unified probabilistic generative model that combines the benefits of the spatial-attention and scene-mixture models. SPACE assumes that a scene x is decomposed into two independent latents: foreground z fg and z bg. The foreground is further decomposed into a set of independent foreground objects z fg = {z fg i} and the is also decomposed further into a sequence of segments z bg = z bg 1:K. While our choice of modeling the foreground and independently worked well empirically, for better generation, it may also be possible to condition one on the other. The image distributions of the foreground objects and the components are combined together with a pixel-wise mixture model to produce the complete image distribution: Here, the foreground mixing probability α is computed as α = f α (z fg). This way, the foreground is given precedence in assigning its own mixing weight and the remaining is apportioned to the . The mixing weight assigned to the is further sub-divided among the K components. These weights are computed as π k = f π k (z bg 1:k) and k π k = 1. With these notations, the complete generative model can be described as follows. Figure 1: Illustration of the SPACE model. SPACE consists of a foreground module and a module. In the foreground module, the input image is divided into a grid of H × W cells (4 × 4 in the figure). An image encoder is used to compute the z where, z depth, and z pres for each cell in parallel. z where is used to identify proposal bounding boxes and a spatial transformer is used to attend to each bounding box in parallel, computing a z what encoding for each cell. The model selects patches using the bounding boxes and reconstructs them using a VAE from all the foreground latents z fg. The module segments the scene into K components (4 in the figure) using a pixel-wise mixture model. Each component consists of a set of latents z bg = (z m, z c) where z m models the mixing probability of the component and z c models the RGB distribution of the component. The components are combined to reconstruct the using a VAE. The reconstructed and foreground are then combined using a pixel-wise mixture model to generate the full reconstructed image. We now describe the foreground and models in more detail. Foreground. SPACE implements z fg as a structured latent. In this structure, an image is treated as if it were divided into H × W cells and each cell is tasked with modeling at most one (nearby) object in the scene. This type of structuring has been used in (; ;). Similar to SPAIR, in order to model an object, each cell i is associated with a set of latents (z). In this notation, z pres is a binary random variable denoting if the cell models any object or not, z where denotes the size of the object and its location relative to the cell, z depth denotes the depth of the object to resolve occlusions and z what models the object appearance and its mask. These latents may then be used to compute the foreground image component p(x|z fg) which is modeled as a Gaussian distribution N (µ fg, σ 2 fg). In practice, we treat σ 2 fg as a hyperparameter and decode only the mean image µ fg. In this process, SPACE reconstructs the objects associated to each cell having z SPACE imposes a prior distribution on these latents as follows: Here, only z pres i is modeled using a Bernoulli distribution while the remaining are modeled as Gaussian. Since we cannot analytically evaluate the integrals in equation 2 due to the continuous latents z fg and z bg 1:K, we train the model using a variational approximation. The true posterior on these variables is approximated as follows. This is used to derive the following ELBO to train the model using the reparameterization trick and SGD . See Appendix B for the detailed decomposition of the ELBO and the related details. Parallel Inference of Cell Latents. SPACE uses mean-field approximation when inferring the cell latents, so z ) for each cell does not depend on other cells. As shown in Figure 1, this allows each cell to act as an independent object detector, spatially attending to its own local region in parallel. This is in contrast to inference in SPAIR, where each cell's latents auto-regressively depend on some or all of the previously traversed cells in a row-major order i.e., q(z However, this method becomes prohibitively expensive in practice as the number of objects increases. claim that these lateral connections are crucial for performance since they model dependencies between objects and thus prevent duplicate detections, we challenge this assertion by observing that 1) due to the bottom-up encoding conditioning on the input image, each cell should have information about its nearby area without explicitly communicating with other cells, and 2) in (physical) spatial space, two objects cannot exist at the same position. Thus, the relation and interference between objects should not be severe and the mean-field approximation is a good choice in our model. In our experiments, we verify empirically that this is indeed the case and observe that SPACE shows comparable detection performance to SPAIR while having significant gains in training speeds and efficiently scaling to scenes with many objects. Preventing Box-Splitting. If the prior for the bounding box size is set to be too small, then the model could split a large object by multiple bounding boxes and when the size prior is too large, the model may not capture small objects in the scene, ing in a trade-off between the prior values of the bounding box size. To alleviate this problem, we found it helpful to introduce an auxiliary loss which we call the boundary loss. In the boundary loss, we construct a boundary of thickness b pixels along the borders of each glimpse. Then, we restrict an object to be inside this boundary and penalize the model if an object's mask overlaps with the boundary area. Thus, the model is penalized if it tries to split a large object by multiple smaller bounding boxes. A detailed implementation of the boundary loss is mentioned in Appendix C. Our proposed model is inspired by several recent works in unsupervised object-oriented scene decomposition. The Attend-Infer-Repeat (AIR) framework uses a recurrent neural network to attend to different objects in a scene and each object is sequentially processed one at a time. An object-oriented latent representation is prescribed that consists of'what','where', and'presence' variables. The'what' variable stores the appearance information of the object, the'where' variable represents the location of the object in the image, and the'presence' variable controls how many steps the recurrent network runs and acts as an interruption variable when the model decides that all objects have been processed. Since the number of steps AIR runs scales with the number of objects it attends to, it does not scale well to images with many objects. Spatially Invariant Attend, Infer, Repeat (SPAIR) attempts to address this issue by replacing the recurrent network with a convolutional network. Similar to YOLO , the locations of objects are specified relative to local grid cells rather than the entire image, which allow for spatially invariant computations. In the encoder network, a convolutional neural network is first used to map the image to a feature volume with dimensions equal to a pre-specified grid size. Then, each cell of the grid is processed sequentially to produce objects. This is done sequentially because the processing of each cell takes as input feature vectors and sampled objects of nearby cells that have already been processed. SPAIR therefore scales with the pre-defined grid size which also represents the maximum number of objects that can be detected. Our model uses an approach similar to SPAIR to detect foreground objects, but importantly we make the foreground object processing fully parallel to scale to large number of objects without performance degradation. Works based on Neural Expectation Maximization do achieve unsupervised object detection but do not explicitly model the presence, appearance, and location of objects. These methods also suffer from the problem of scaling to images with a large number of objects. For unsupervised scene-mixture models, several recent models have shown promising . MONet leverages a deterministic recurrent attention network that outputs pixel-wise masks for the scene components. A variational autoencoder (VAE) is then used to model each component. IODINE approaches the problem from a spatial mixture model perspective and uses amortized iterative refinement of latent object representations within the variational framework. GENESIS also uses a spatial mixture model which is encoded by component-wise latent variables. Relationships between these components are captured with an autoregressive prior, allowing complete images to be modeled by a collection of components. We evaluate our model on two datasets: 1) an Atari dataset that consists of random images from a pretrained agent playing the games, and 2) a generated 3D-room dataset that consists of images of a walled enclosure with a random number of objects on the floor. In order to test the scalability of our model, we use both a small 3D-room dataset that has 4-8 objects and a large 3D-room dataset that has 18-24 objects. Each image is taken from a random camera angle and the colors of the objects, walls, floor, and sky are also chosen at random. Additional details of the datasets can be found in the Appendix E. Baselines. We compare our model against two scene-mixture models (IODINE and GENESIS) and one spatial-attention model (SPAIR). Since SPAIR does not have an explicit component, we add an additional VAE for processing the . Additionally, we test against two implementations of SPAIR: one where we train on the entire image using a 16 × 16 grid and another where we train on random 32 × 32 pixel patches using a 4 × 4 grid. We denote the former model as SPAIR and the latter as SPAIR-P. SPAIR-P is consistent with the SPAIR's alternative training regime on Space Invaders demonstrated in to address the slow training of SPAIR on the full grid size because of its sequential inference. Lastly, for performance reasons, unlike the original SPAIR implementation, we use parallel processing for rendering the objects from their respective latents onto the canvas 1 for both SPAIR and SPAIR-P. Thus, because of these improvements, our SPAIR implementation can be seen as a stronger baseline than the original SPAIR. The complete details of the architecture used is given in Appendix D. In this section, we provide a qualitative analysis of the generated representations of the different models. For each model, we performed a hyperparameter search and present the for the best settings of hyperparameters for each environment. Figure 2 shows sample scene decompositions of our baselines from the 3D-Room dataset and Figure 3 shows the on Atari. Note that SPAIR does not use component masks and IODINE and GENESIS do not separate foreground from , hence the corresponding cells are left empty. Additionally, we only show a few representative components for IODINE and GENESIS since we ran those experiments with larger K than can be displayed. More qualitative of SPACE can be found in Appendix A. In the 3D-Room environment, IODINE is able to segment the objects and the into separate components. However, it occasionally does not properly decompose objects (in the Large 3D-room , the orange sphere on the right is not reconstructed) and may generate blurry objects. GENESIS is able to segment the walls, floor, and sky into multiple components. It is able to capture blurry foreground objects in the Small 3D-Room, but is not able to cleanly capture foreground objects with the larger number of objects in the Large 3D-Room. In Atari, for all games, both IODINE and GENESIS fail to capture the foreground properly. We believe this is because the objects in Atari games are smaller, less regular and lack the obvious latent factors like color and shape as in the 3D dataset, which demonstrates that detection-based approaches are more appropriate in this case. SPAIR & SPAIR-P. SPAIR is able to detect tight bounding boxes in both 3D-Room and most Atari games (it does not work as well on dynamic games, which we discuss below). SPAIR-P, however, often fails to detect the foreground objects in proper bounding boxes, frequently uses multiple bounding boxes for one object and redundantly detects parts of the as foreground objects. This is a limitation of the patch training as the receptive field of each patch is limited to a 32 × 32 glimpse, hence prohibiting it to detect objects larger than that and making it difficult to distinguish the from foreground. These two properties are illustrated well in Space Invaders, where it is able to detect the small aliens, but it detects the long piece of ground on the bottom of the image as foreground objects. SPACE. In 3D-Room, SPACE is able to accurately detect almost all objects despite the large variations in object positions, colors, and shapes, while producing a clean segmentation of the walls, ground, and sky. This is in contrast to the SPAIR model, while being able to provide similar foreground detection quality, encodes the whole into a single component, which makes the representation less disentangled and the reconstruction more blurry. Similarly in Atari, SPACE consistently captures all foreground objects while producing clean segmentation across many different games. Dynamic Backgrounds. SPACE and SPAIR exhibit some very interesting behavior when trained on games with dynamic s. For the most static game -Space Invaders, both SPACE and SPAIR work well. For Air Raid, in which the building moves, SPACE captures all objects accurately while providing a two-component segmentation, whereas SPAIR and SPAIR-P produce splitting and heavy re-detections. In the most dynamic games, SPAIR completely fails because of the difficulty to model dynamic with a single VAE component, while SPACE is able to perfectly segment the blue racing track while accurately detecting all foreground objects. Foreground vs Background. Typically, foreground is the dynamic local part of the scene that we are interested in, and is the relatively static and global part. This definition, though intuitive, is ambiguous. Some objects, such as the red shields in Space Invaders and the key in Montezuma's Revenge (Figure 5) are detected as foreground objects in SPACE, but are considered in SPAIR. Though these objects are static 2, they are important elements of the games and should be considered as foreground objects. Similar behavior is observed in Atlantis (Figure 7), where SPACE detects some foreground objects from the middle base that is above the water. We believe this is an interesting property of SPACE and could be very important for providing useful representations for downstream tasks. By using a spatial broadcast network which is much weaker when compared to other decoders like sub-pixel convolutional nets , we limit the capacity of module, which favors modeling static objects as foreground rather than . Boundary Loss. We notice SPAIR sometimes splits objects into two whereas SPACE is able to create the correct bounding box for the objects (for example, see Air Raid). This may be attributed to the addendum of the auxiliary boundary loss in the SPACE model that would penalize splitting an object with multiple bounding boxes. In this section we compare SPACE with the baselines in several quantitative metrics 3. We first note that each of the baseline models has a different decomposition capacity (C), which we define as the capability of the model to decompose the scene into its semantic constituents such as the foreground objects and the segmented components. For SPACE, the decomposition capacity is equal to the number of grid cells H × W (which is the maximum number of foreground objects that can be detected) plus the number of components K. For SPAIR, the decomposition capacity is equal to the number of grid cells H × W plus 1 for . For IODINE and GENESIS, it is equal to the number of components K. For each experiment, we compare the metrics for each model with similar decomposition capacities. This way, each model can decompose the image into the same number of components. For a setting in SPACE with a grid size of H × W with K SPACE components, the equivalent settings in IODINE and GENESIS would be with C = (H × W) + K SPACE. The equivalent setting in SPAIR would be a grid size of H × W. Step Latency. The leftmost chart of Figure 4 shows the time taken to complete one gradient step (forward and backward propagation) for different decomposition capacities for each of the models. We see that SPAIR's latency grows with the number of cells because of the sequential nature of its latent inference step. Similarly GENESIS and IODINE's latency grows with the number of components K because each component is processed sequentially in both the models. IODINE is Training Latency Plot Figure 4: Quantitative performance comparison between SPACE, SPAIR, IODINE and GENESIS in terms of batch-processing time during training, training convergence and converged pixel MSE. Convergence plots showing pixel-MSE were computed on a held-out set during training. the slowest overall with its computationally expensive iterative inference procedure. Furthermore, both IODINE and GENESIS require storing data for each of the K components, so we were unable to run our experiments on 256 components or greater before running out of memory on our 22GB GPU. On the other hand, SPACE employs parallel processing for the foreground which makes it scalable to large grid sizes, allowing it to detect a large number of foreground objects without any significant performance degradation. Although this data was collected for gradient step latency, this comparison implies a similar relationship exists with inference time which is a main component in the gradient step. Time for Convergence. The remaining three charts in Figure 4 show the amount of time each model takes to converge in different experimental settings. We use the pixel-wise mean squared error (MSE) as a measurement of how close a model is to convergence. We see that not only does SPACE achieve the lowest MSE, it also converges the quickest out of all the models. Average Precision and Error Rate. In order to assess the quality of our bounding box predictions and the effectiveness of boundary loss, we measure the Average Precision and Object Count Error Rate of our predictions. Our are shown in Table 1. We only report these metrics for 3D-Room since we have access to the ground truth bounding boxes for each of the objects in the scene. All three models have very similar average precision and error rate. Despite being parallel in its inference, SPACE has a comparable count error rate to that of SPAIR. SPACE also achieves better average precision and count error rate compared to its variant without the boundary loss (SPACE-WB), which shows the efficacy of our proposed loss. From our experiments, we can assert that SPACE can produce similar quality bounding boxes as SPAIR while 1) having orders of magnitude faster inference and gradient step time, 2) converging more quickly, 3) scaling to a large number of objects without significant performance degradation, and 4) providing complex segmentation. We propose SPACE, a unified probabilistic model that combines the benefits of the object representation models based on spatial attention and the scene decomposition models based on component mixture. SPACE can explicitly provide factorized object representation per foreground object while also decomposing complex segments. SPACE also achieves a significant speed-up and thus makes the model applicable to scenes with a much larger number of objects without performance degradation. Besides, the detected objects in SPACE are also more intuitive than other methods. We show the above properties of SPACE on Atari and 3D-Rooms. Interesting future directions are to replace the sequential processing of by a parallel one and to improve the model for natural images. Our next plan is to apply SPACE for object-oriented model-based reinforcement learning. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection.: Object detection and segmentation using SPACE on 3D-Room data set with large number of objects. In this section, we derive the ELBO for the log-likelihood log p(x). KL Divergence for the Foreground Latents Under the SPACE's approximate inference, the inside the expectation can be evaluated as follows. KL Divergence for the Background Latents Under our GENESIS-like modeling of inference for the latents, the KL term inside the expectation for the is evaluated as follows. Relaxed treatment of z pres In our implementation, we model the Bernoulli random variable z pres i using the Gumbel-Softmax distribution . We use the relaxed value of z pres in the entire training and use hard samples only for the visualizations. In this section we elaborate on the implementation details of the boundary loss. We construct a kernel of the size of the glimpse, gs × gs (we use gs = 32) with a boundary gap of b = 6 having negative uniform weights inside the boundary and a zero weight in the region between the boundary and the glimpse. This ensures that the model is penalized when the object is outside the boundary. This kernel is first mapped onto the global space via to obtain the global kernel. This is then multiplied element-wise with global object mask α to obtain the boundary loss map. The objective of the loss is to minimize the mean of this boundary loss map. In addition to the ELBO, this loss is also back-propagated via RMSProp (Tieleman & Hinton. ). This loss, due to the boundary constraint, enforces the bounding boxes to be less tight and in lower average precision, so we disable the loss and optimize only the ELBO after the model has converged well. D.1 ALGORITHMS Algorithm 1 and Algorithm 2 present SPACE's inference for foreground and . Algorithm 3 show the details of the generation process of the module. For foreground generation, we simply sample the latent variables from the priors instead of conditioning on the input. Note that, for convenience the algorithms for the foreground module and module are presented with for loops, but inference for all variables of the foreground module are implemented as parallel convolution operations and most operations of the module (barring the LSTM module) are parallel as well.) optimizer with a learning rate of 1 × 10 −3 for the module. We use gradient clipping with a maximum norm of 1.0. For Atari games, we find it beneficial to set α to be fixed for the first 1000-2000 steps, and vary the actual value and number of steps for different games. This allows both the foreground as well as the module to learn in the early stage of training. Atari. For each game, we sample 60,000 random images from a pretrained agent . We split the images into 50,000 for the training set, 5,000 for the validation set, and 5,000 for the testing set. Each image is preprocessed into a size of 128 × 128 pixels with BGR color channels. We present the for the following games: Space Invaders, Air Raid, River Raid, Montezuma's Revenge. We also train our model on a dataset of 10 games jointly, where we have 8,000 training images, 1,000 validation images, and 1,000 testing images for each game. We use the following games: Asterix, Atlantis, Carnival, Double Dunk, Kangaroo, Montezuma Revenge, Pacman, Pooyan, Qbert, Space Invaders. Room 3D. We use MuJoCo to generate this dataset. Each image consists of a walled enclosure with a random number of objects on the floor. The possible objects are randomly sized spheres, cubes, and cylinders. The small 3D-Room dataset has 4-8 objects and the large 3D-Room dataset has 18-24 objects. The color of the objects are randomly chosen from 8 different colors and the colors of the (wall, ground, sky) are chosen randomly from 5 different colors. The angle of the camera is also selected randomly. We use a training set of 63,000 images, a validation set of 7,000 images, and a test set of 7,000 images. We use a 2-D projection from the camera to determine the ground truth bounding boxes of the objects so that we can report the average precision of the different models.

Represent the short summary for finding the corresponding scientific article (long)

We propose a generative latent variable model for unsupervised scene decomposition that provides factorized object representation per foreground object while also decomposing background segments of complex morphology.

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models


Question: We propose a single neural probabilistic model based on variational autoencoder that can be conditioned on an arbitrary subset of observed features and then sample the remaining features in "one shot". The features may be both real-valued and categorical. Training of the model is performed by stochastic variational Bayes. The experimental evaluation on synthetic data, as well as feature imputation and image inpainting problems, shows the effectiveness of the proposed approach and diversity of the generated samples. In past years, a number of generative probabilistic models based on neural networks have been proposed. The most popular approaches include variational autoencoder (VAE) and generative adversarial net (GANs). They learn a distribution over objects p(x) and allow sampling from this distribution. In many cases, we are interested in learning a conditional distribution p(x|y). For instance, if x is an image of a face, y could be the characteristics describing the face (are glasses present or not; length of hair, etc.) Conditional variational autoencoder and conditional generative adversarial nets are popular methods for this problem. In this paper, we consider the problem of learning all conditional distributions of the form p(x I |x U \I), where U is the set of all features and I is its arbitrary subset. This problem generalizes both learning the joint distribution p(x) and learning the conditional distribution p(x|y). To tackle this problem, we propose a Variational Autoencoder with Arbitrary Conditioning (VAEAC) model. It is a latent variable model similar to VAE, but allows conditioning on an arbitrary subset of the features. The conditioning features affect the prior on the latent Gaussian variables which are used to generate unobserved features. The model is trained using stochastic gradient variational Bayes .We consider two most natural applications of the proposed model. The first one is feature imputation where the goal is to restore the missing features given the observed ones. The imputed values may be valuable by themselves or may improve the performance of other machine learning algorithms which process the dataset. Another application is image inpainting in which the goal is to fill in an unobserved part of an image with an artificial content in a realistic way. This can be used for removing unnecessary objects from the images or, vice versa, for complementing the partially closed or corrupted object. The experimental evaluation shows that the proposed model successfully samples from the conditional distributions. The distribution over samples is close to the true conditional distribution. This property is very important when the true distribution has several modes. The model is shown to be effective in feature imputation problem which helps to increase the quality of subsequent discriminative models on different problems from UCI datasets collection . We demonstrate that model can generate diverse and realistic image inpaintings on MNIST , Omniglot and CelebA datasets, and works even better than the current state of the art inpainting techniques in terms of peak signal to noise ratio (PSNR).The paper is organized as follows. In section 2 we review the related works. In section 3 we briefly describe variational autoencoders and conditional variational autoencoders. In section 4 we define the problem, describe the VAEAC model and its training procedure. In section 5 we evaluate VAEAC. Section 6 concludes the paper. Appendix contains additional explanations, theoretical analysis, and experiments for VAEAC. Universal Marginalizer is a model based on a feed-forward neural network which approximates marginals of unobserved features conditioned on observable values. A related idea of an autoregressive model of joint probability was previously proposed in and. The description of the model and comparison with VAEAC are available in section 5.3. propose a GANs-based model called GAIN which solves the same problem as VAEAC. In contrast to VAEAC, GAIN does not use unobserved data during training, which makes it easier to apply to the missing features imputation problem. Nevertheless, it turns into a disadvantage when the fully-observed training data is available but the missingness rate at the testing stage is high. For example, in inpainting setting GAIN cannot learn the conditional distribution over MNIST digits given one horizontal line of the image while VAEAC can (see appendix D.4). The comparison of VAEAC and GAIN on the missing feature imputation problem is given in section 5.1 and appendix D.2. [Appendix F], , , and propose to fill missing data with noise and run Markov chain with a learned transition operator. The stationary distribution of such chains approximates the true conditional distribution of the unobserved features. BID0 consider missing feature imputation in terms of Markov decision process and propose LSTM-based sequential decision making model to solve it. Nevertheless, these methods are computationally expensive at the test time and require fully-observed training data. Image inpainting is a classic computer vision problem. Most of the earlier methods rely on local and texture information or hand-crafted problem-specific features . In past years multiple neural network based approaches have been proposed. , and use different kinds and combinations of adversarial, reconstruction, texture and other losses. focuses on face inpainting and uses two adversarial losses and one semantic parsing loss to train the generative model. GANs are first trained on the whole training dataset. The inpainting is an optimization procedure that finds the latent variables that explain the observed features best. Then, the obtained latents are passed through the generative model to restore the unobserved portion of the image. We can say that VAEAC is a similar model which uses prior network to find a proper latents instead of solving the optimization problem. All described methods aim to produce a single realistic inpainting, while VAEAC is capable of sampling diverse inpaintings. Additionally, Yeh et al., Yang et al. (2017 and have high testtime computational complexity of inpainting, because they require an optimization problem to be solved. On the other hand, VAEAC is a "single-shot" method with a low computational cost. Variational autoencoder (VAE) is a directed generative model with latent variables. The generative process in variational autoencoder is as follows: first, a latent variable z is generated from the prior distribution p(z), and then the data x is generated from the generative distribution p θ (x|z), where θ are the generative model's parameters. This process induces the distribution p θ (x) = E p(z) p θ (x|z). The distribution p θ (x|z) is modeled by a neural network with parameters θ. p(z) is a standard Gaussian distribution. The parameters θ are tuned by maximizing the likelihood of the training data points {x i} N i=1 from the true data distribution p d (x). In general, this optimization problem is challenging due to intractable posterior inference. However, a variational lower bound can be optimized efficiently using backpropagation and stochastic gradient descent: DISPLAYFORM0 Here q φ (z|x) is a proposal distribution parameterized by neural network with parameters φ that approximates the posterior p(z|x, θ). Usually this distribution is Gaussian with a diagonal covariance matrix. The closer q φ (z|x) to p(z|x, θ), the tighter variational lower bound L V AE (θ, φ). To compute the gradient of the variational lower bound with respect to φ, reparameterization trick is used: z = µ φ (x) + εσ φ (x) where ε ∼ N (0, I) and µ φ and σ φ are deterministic functions parameterized by neural networks. So the gradient can be estimated using Monte-Carlo method for the first term and computing the second term analytically: DISPLAYFORM1 So L V AE (θ, φ) can be optimized using stochastic gradient ascent with respect to φ and θ. Conditional variational autoencoder (CVAE) approximates the conditional distribution p d (x|y). It outperforms deterministic models when the distribution p d (x|y) is multi-modal (diverse xs are probable for the given y). For example, assume that x is a real-valued image. Then, a deterministic regression model with mean squared error loss would predict the average blurry value for x. On the other hand, CVAE learns the distribution of x, from which one can sample diverse and realistic objects. Variational lower bound for CVAE can be derived similarly to VAE by conditioning all considered distributions on y: DISPLAYFORM0 Similarly to VAE, this objective is optimized using the reparameterization trick. Note that the prior distribution p ψ (z|y) is conditioned on y and is modeled by a neural network with parameters ψ. Thus, CVAE uses three trainable neural networks, while VAE only uses two. Also authors propose such modifications of CVAE as Gaussian stochastic neural network and hybrid model. These modifications can be applied to our model as well. Nevertheless, we don't use them, because of their disadvantage which is described in appendix C. Let binary vector b ∈ {0, 1} D be the binary mask of unobserved features of the object. Then we describe the vector of unobserved features as x b = {x i:bi=1}. For example, x = (x 2, x 3, x 5). Using this notation we denote x 1−b as a vector of observed features. Our goal is to build a model of the conditional distribution DISPLAYFORM1 for an arbitrary b, where ψ and θ are parameters that are used in our model at the testing stage. However, the true distribution p d (x b |x 1−b, b) is intractable without strong assumptions about p d (x). Therefore, our model p ψ,θ (x b |x 1−b, b) has to be more precise for some b and less precise for others. To formalize our requirements about the accuracy of our model we introduce the distribution p(b) over different unobserved feature masks. The distribution p(b) is arbitrary and may be defined by the user depending on the problem. Generally it should have full support over {0, 1}D so that p ψ,θ (x b |x 1−b, b) can evaluate arbitrary conditioning. Nevertheless, it is not necessary if the model is used for specific kinds of conditioning (as we do in section 5.2).Using p(b) we can introduce the following log-likelihood objective function for the model: DISPLAYFORM2 The special cases of the objective are variational autoencoder (b i = 1 ∀i ∈ {1, . . ., D}) and conditional variational autoencoder (b is constant). The generative process of our model is similar to the generative process of CVAE: for each object firstly we generate z ∼ p ψ (z|x 1−b, b) using prior network, and then sample unobserved features x b ∼ p θ (x b |z, x 1−b, b) using generative network. This process induces the following model distribution over unobserved features: DISPLAYFORM0 We use z ∈ R d, and Gaussian distribution p ψ over z, with parameters from a neural network with weights ψ: DISPLAYFORM1 is parameterized by a function w i,θ (z, x 1−b, b), whose outputs are logits of probabilities for each category: DISPLAYFORM2. Therefore the components of the latent vector z are conditionally independent given x 1−b and b, and the components of x b are conditionally independent given z, x 1−b and b. The variables x b and x 1−b have variable length that depends on b. So in order to use architectures such as multi-layer perceptron and convolutional neural network we consider x 1−b = x • (1 − b) where • is an element-wise product. So in implementation x 1−b has fixed length. The output of the generative network also has a fixed length, but we use only unobserved components to compute likelihood. The theoretical analysis of the model is available in appendix B.1. We can derive a lower bound for log p ψ,θ (x b |x 1−b, b) as for variational autoencoder: DISPLAYFORM0 Therefore we have the following variational lower bound optimization problem: DISPLAYFORM1 We use fully-factorized Gaussian proposal distribution q φ which allows us to perform reparameterization trick and compute KL divergence analytically in order to optimize. During the optimization of objective FORMULA9 the parameters µ ψ and σ ψ of the prior distribution of z may tend to infinity, since there is no penalty for large values of those parameters. We usually observe the growth of z 2 during training, though it is slow enough. To prevent potential numerical instabilities, we put a Normal-Gamma prior on the parameters of the prior distribution to prevent the divergence. Formally, we redefine p ψ (z|x 1−b, b) as follows: DISPLAYFORM0 As a , the regularizers − µ 2 ψ 2σ 2 µ and σ σ (log(σ ψ) − σ ψ ) are added to the model log-likelihood. Hyperparameter σ µ is chosen to be large (10 4) and σ σ is taken to be a small positive number (10 −4). This distribution is close to uniform near zero, so it doesn't affect the learning process significantly. The optimization objective requires all features of each object at the training stage: some of the features will be observed variables at the input of the model and other will be unobserved features used to evaluate the model. Nevertheless, in some problem settings the training data contains missing features too. We propose the following slight modification of the problem in order to cover such problems as well. The missing values cannot be observed so x i = ω ⇒ b i = 1, where ω describes the missing value in the data. In order to meet this requirement, we redefine mask distribution as conditioned on x: p(b) turns into p(b|x) in and. In the reconstruction loss we simply omit the missing features, i. e. marginalize them out: DISPLAYFORM0 The proposal network must be able to determine which features came from real object and which are just missing. So we use additional missing features mask which is fed to proposal network together with unobserved features mask b and object x. The proposed modifications are evaluated in section 5.1. In this section we validate the performance of VAEAC using several real-world datasets. In the first set of experiments we evaluate VAEAC missing features imputation performance using various UCI datasets . We compare imputations from our model with imputations from such classical methods as MICE and MissForest (Stekhoven & Bühlmann, 2011) and recently proposed GANs-based method GAIN . In the second set of experiments we use VAEAC to solve image inpainting problem. We show inpainitngs generated by VAEAC and compare our model with models from papers Pathak et al. FORMULA1, and in terms of peak signal-to-noise ratio (PSNR) of obtained inpaintings on CelebA dataset . And finally, we evaluate VAEAC against the competing method called Universal Marginalizer . Additional experiments can be found in appendices C and D. The code is available at https://github.com/tigvarts/ vaeac. The datasets with missing features are widespread. Consider a dataset with D-dimensional objects x where each feature may be missing (which we denote by x i = ω) and their target values y. The majority of discriminative methods do not support missing values in the objects. The procedure of filling in the missing features values is called missing features imputation. In this section we evaluate the quality of imputations produced by VAEAC. For evaluation we use datasets from UCI repository . Before training we drop randomly 50% of values both in train and test set. After that we impute missing features using MICE , MissForest (Stekhoven & Bühlmann, 2011), GAIN and VAEAC trained on the observed data. The details of GAIN implementation are described in appendix A.4.Our model learns the distribution of the imputations, so it is able to sample from this distribution. We replace each object with missing features by n = 10 objects with sampled imputations, so the size of the dataset increases by n times. This procedure is called missing features multiple imputation. MICE and GAIN are also capable of multiple imputation (we use n = 10 for them in experiments as well), but MissForest is not. For more details about the experimental setup see appendices A.1, A.2, and A.4.In table 1 we report NRMSE (i.e. RMSE normalized by the standard deviation of each feature and then averaged over all features) of imputations for continuous datasets and proportion of falsely classified (PFC) for categorical ones. For multiple imputation methods we average imputations of continuous variables and take most frequent imputation for categorical ones for each object. We also learn linear or logistic regression and report the regression or classification performance after applying imputations of different methods in table 2. For multiple imputation methods we average predictions for continuous targets and take most frequent prediction for categorical ones for each object in test set. As can be seen from the tables 1 and 2, VAEAC can learn joint data distribution and use it for missing feature imputation. The imputations are competitive with current state of the art imputation methods in terms of RMSE, PFC, post-imputation regression R2-score and classification accuracy. Nevertheless, we don't claim that our method is state of the art in missing features imputation; for some datasets MICE or MissForest outperform it. The additional experiments can be found in appendix D.2. The image inpainting problem has a number of different formulations. The formulation of our interest is as follows: some of the pixels of an image are unobserved and we want to restore them in a natural way. Unlike the majority of papers, we want to restore not just one most probable inpainting, but the distribution over all possible inpaintings from which we can sample. This distribution is extremely multi-modal because often there is a lot of different possible ways to inpaint the image. Unlike the previous subsection, here we have uncorrupted images without missing features in the training set, so p(b|x) = p(b).As we show in section 2, state of the art use different adversarial losses to achieve more sharp and realistic samples. VAEAC can be adapted to the image inpainting problem by using a combination of those adversarial losses as a part of reconstruction loss p θ (x b |z, x 1−b, b). Nevertheless, such construction is out of scope for this research, so we leave it for the future work. In the current work we show that the model can generate both diverse and realistic inpaintings. In figures 1, 2, 3 and 4 we visualize image inpaintings produced by VAEAC on binarized MNIST , Omniglot (and CelebA . The details of learning procedure and description of datasets are available in appendixes A.1 and A.3.To the best of our knowledge, the most modern inpainting papers don't consider the diverse inpainting problem, where the goal is to build diverse image inpaintings, so there is no straightforward way to compare with these models. Nevertheless, we compute peak signal-to-noise ratio (PSNR) for one random inpainting from VAEAC and the best PSNR among 10 random inpaintings from VAEAC. One inpainting might not be similar to the original image, so we also measure how good the inpainting which is most similar to the original image reconstructs it. We compare these two metrics computed for certain masks with the PSNRs for the same masks on CelebA from papers Yeh et al. FORMULA1 and. The are available in tables 3 and 4.We observe that for the majority of proposed masks our model outperforms the competing methods in terms of PSNR even with one sample, and for the rest (where the inpaintings are significantly diverse) the best PSNR over 10 inpaintings is larger than the same PSNR of the competing models. Even if PSNR does not reflect completely the visual quality of images and tends to encourage blurry VAE samples instead of realistic GANs samples, the show that VAEAC is able to solve inpainting problem comparably to the state of the art methods. FORMULA1 ) is that it needs the distribution over masks at the training stage to be similar to the distribution over them at the test stage. However, it is not a very strict limitation for the practical usage. Universal Marginalizer (UM) is a model which uses a single neural network to estimate the marginal distributions over the unobserved features. So it optimizes the following objective: DISPLAYFORM0 For given mask b we fix a permutation of its unobserved components: (i 1, i 2, . . ., i |b|), where |b| is a number of unobserved components. Using the learned model and the permutation we can generate objects from joint distribution and estimate their probability using chain rule. DISPLAYFORM1 For example, DISPLAYFORM2 Conditional sampling or conditional likelihood estimation for one object requires |b| requests to UM to compute p θ (x i |x 1−b, b). Each request is a forward pass through the neural network. In the case of conditional sampling those requests even cannot be paralleled because the input of the next request contains the output of the previous one. We propose a slight modification of the original UM training procedure which allows learning UM efficiently for any kind of masks including those considered in this paper. The details of the modification are described in appendix B.3. Without skip-connections all information for decoder goes through the latent variables. In image inpainting we found skip-connections very useful in both terms of log-likelihood improvement and the image realism, because latent variables are responsible for the global information only while the local information passes through skip-connections. Therefore the border between image and inpainting becomes less conspicuous. The main idea of neural networks architecture is reflected in FIG4. The number of hidden layers, their widths and structure may be different. The neural networks we used for image inpainting have He-Uniform initialization of convolutional ResNet blocks, and the skip-connections are implemented using concatenation, not addition. The proposal network structure is exactly the same as the prior network except skip-connections. Also one could use much simpler fully-connected networks with one hidden layer as a proposal, prior and generative networks in VAEAC and still obtain nice inpaintings on MNIST. We split the dataset into train and test set with size ratio 3:1. Before training we drop randomly 50% of values both in train and test set. We repeat each experiment 5 times with different train-test splits and dropped features and then average and compute their standard deviation. As we show in appendix B.2, the better can be achieved when the model learns the concatenation of objects features x and targets y. So we treat y as an additional feature that is always unobserved during the testing time. To train our model we use distribution p(b i |x) in which p(b i |x i = ω) = 1 and p(b i |x) = 0.2 otherwise. Also for VAEAC trainig we normalize real-valued features, fix σ θ = 1 in the generative model of VAEAC in order to optimize RMSE, and use 25% of training data as validation set to select the best model among all epochs of training. For the test set, the classifier or regressor is applied to each of the n imputed objects and the predictions are combined. For regression problems we report R2-score of combined predictions, so we use averaging as a combination method. For classification problem we report accuracy, and therefore choose the mode. We consider the workflow where the imputed values of y are not fed to the classifier or regressor to make a fair comparison of feature imputation quality. MNIST is a dataset of 60000 train and 10000 test grayscale images of digits from 0 to 9 of size 28x28. We binarize all images in the dataset. For MNIST we consider Bernoulli log-likelihood as the reconstruction loss: DISPLAYFORM0 is an output of the generative neural network. We use 16 latent variables. In the mask for this dataset the observed pixels form a three pixels wide horizontal line which position is distributed uniformly. Omniglot is a dataset of 19280 train and 13180 test black-and-white images of different alphabets symbols of size 105x105. As in previous section, the brightness of each pixel is treated as a Bernoulli probability of it to be 1. The mask we use is a random rectangular which is described below. We use 64 latent variables. We train model for 50 epochs and choose best model according to IWAE log-likelihood estimation on the validation set after each epoch. CelebA is a dataset of 162770 train, 19867 validation and 19962 test color images of faces of celebrities of size 178x218. Before learning we normalize the channels in dataset. We use logarithm of fully-factorized Gaussian distribution as reconstruction loss. The mask we use is a random rectangular which is describe below. We use 32 latent variables. Rectangular mask is the common shape of unobserved region in image inpainting. We use such mask for Omniglot and Celeba. We sample the corner points of rectangles uniprobably on the image, but reject those rectangles which area is less than a quarter of the image area. In Li et al. FORMULA1 six different masks O1-O6 are used on the testing stage. We reconstruct the positions of masks from the illustrations in the paper and give their coordinates in table 6. The visualizations of the masks are available in FIG0.At the training stage we used a rectangle mask with uniprobable random corners. We reject masks with width or height less than 16pt. We use 64 latent variables and take the best model over 50 epochs based on the validation IWAE log-likelihood estimation. We can obtain slightly higher PSNR values than reported in table 4 if use only masks O1-O6 at the training stage. four types of masks are used. Center mask is just an unobserved 32x32 square in the center of 64x64 image. Half mask mean that one of upper, lower, left or right half of the image is unobserved. All these types of a half are equiprobable. Random mask means that we use pixelwise-independent Bernoulli distribution with probability 0.8 to form a mask of unobserved pixels. Pattern mask is proposed in. As we deduced from the code 3, the generation process is follows: firstly we generate 600x600 one-channel image with uniform distribution over pixels, then bicubically interpolate it to image of size 10000x10000, and then apply Heaviside step function H(x − 0.25) (i. e. all points with value less than 0.25 are considered as unobserved). To sample a mask we sample a random position in this 10000x10000 binary image and crop 64x64 mask. If less than 20% or more than 30% of pixel are unobserved, than the mask is rejected and the position is sampled again. In comparison with this paper in section 5.2 we use the same distribution over masks at training and testing stages. We use VAEAC with 64 latent variables and take the best model over 50 epochs based on the validation IWAE log-likelihood estimation. For missing feature imputation we reimplemented GAIN in PyTorch based on the paper and the available TensorFlow source code for image inpainting 4.For categorical features we use one-hot encoding. We observe in experiments that it works better in terms of NRMSE and PFC than processing categorical features in GAIN as continuous ones and then rounding them to the nearest category. For categorical features we also use reconstruction loss L M (x i, x i) = − 1 |Xi| |Xi| j=1 x i,j log(x i,j). |X i | is the number of categories of the i-th feature, and x i,j is the j-th component of one-hot encoding of the feature x i. Such L M enforces equal contribution of each categorical feature into the whole reconstruction loss. We use one more modification of L M (x, x) for binary and categorical features. Cross-entropy loss in L M penalizes incorrect reconstructions of categorical and binary features much more than incorrect reconstructions for continuous ones. To avoid such imbalance we mixed L2 and cross-entropy reconstruction losses for binary and categorical features with weights 0.8 and 0.2 respectively: DISPLAYFORM0 We observe in experiments that this modification also works better in terms of NRMSE and PFC than the original model. We use validation set which contains 5% of the observed features for the best model selection (hyperparameter is the number of iterations).In the original GAIN paper authors propose to use cross-validation for hyper-parameter α ∈ {0.1, 0.5, 1, 2, 10}. We observe that using α = 10 and a hint h = b • m + 0.5(1 − b) where vector b is sampled from Bernoulli distribution with p = 0.01 provides better in terms of NRMSE and PFC than the original model with every α ∈ {0.1, 0.5, 1, 2, 10}. Such hint distribution makes model theoretically inconsistent but works well in practice (see table 7). Table 7 shows that our modifications provide consistently not worse or even better imputations than the original GAIN (in terms of NRMSE and PFC, on the considered datasets). So in this paper for the missing feature imputation problem we report the of our modification of GAIN. We can imagine 2 D CVAEs learned each for the certain mask. Because neural networks are universal approximators, VAEAC networks could model the union of CVAE networks, so that VAEAC network performs transformation defined by the same network of the corresponding to the given mask CVAE. DISPLAYFORM1 So if CVAE models any distribution p(x|y), VAEAC also do. The guarantees for CVAE in the case of continuous variables are based on the point that every smooth distribution can be approximated with a large enough mixture of Gaussians, which is a special case of CVAE's generative model. These guarantees can be extended on the case of categorical-continuous variables also. Actually, there are distributions over categorical variables which CVAE with Gaussian prior and proposal distributions cannot learn. Nevertheless, this kind of limitation is not fundamental and is caused by poor proposal distribution family. Consider a dataset with D-dimensional objects x where each feature may be missing (which we denote by x i = ω) and their target values y. In this section we show that the better are achieved when our model learns the concatenation of objects features x and targets y. The example that shows the necessity of it is following. Consider a dataset where x 1 = 1, x 2 ∼ N (x 2 |y, 1), p d (y = 0) = p(y = 5) = 0.5. In this case p d (x 2 |x 1 = 1) = 0.5N (x 2 |0, 1) + 0.5N (x 2 |5, 1). We can see that generating data from p d (x 2 |x 1) may only confuse the classifier, because with probability 0.5 it generates x 2 ∼ N for y = 5 and x 2 ∼ N for y = 0. On the other hand, p d (x 2 |x 1, y) = N (x 2 |y, 1). Filling gaps using p d (x 2 |x 1, y) may only improve classifier or regressor by giving it some information from the joint distribution p d (x, y) and thus simplifying the dependence to be learned at the training time. So we treat y as an additional feature that is always unobserved during the testing time. The problem authors did not address in the original paper is the relation between the distribution of unobserved components p(b) at the testing stage and the distribution of masks in the requests to UMp(b). The distribution over masks p(b) induces the distributionp(b), and in the most cases p(b) =p(b). The distributionp(b) also depends on the permutations (i 1, i 2, . . ., i |b|) that we use to generate objects. We observed in experiments, that UM must be trained using unobserved mask distributionp(b). For example, if all masks from p(b) have a fixed number of unobserved components (e. g., 2), then UM will never see an example of mask with 1, 2,..., D 2 − 1 unobserved components, which is necessary to generate a sample conditioned on D 2 components. That leads to drastically low likelihood estimate for the test set and unrealistic samples. We developed an easy generative process forp(b) for arbitrary p(b) if the permutation of unobserved components (i 1, i 2, . . ., i |b|) is chosen randomly and equiprobably: firstly we generate DISPLAYFORM0 More complicated generative process exists for a sorted permutation where i j−1 < i j ∀j: 2 ≤ j ≤ |b|.In experiments we use uniform distribution over the permutations. Gaussian stochastic neural network and hybrid model are originally proposed in the paper on Conditional VAE . The motivation authors mention in the paper is as follows. During training the proposal distribution q φ (z|x, y) is used to generate the latent variables z, while during the testing stage the prior p ψ (z|y) is used. KL divergence tries to close the gap between two distributions but, according to authors, it is not enough. To overcome the issue authors propose to use a hybrid model FORMULA4, a weighted mixture of variational lower bound and a single-sample Monte-Carlo estimation of log-likelihood. The model corresponding to the second term is called Gaussian Stochastic Neural Network, because it is a feed-forward neural network with a single Gaussian stochastic layer in the middle. Also GSNN is a special case of CVAE where q φ (z|x, y) = p ψ (z|y). DISPLAYFORM0 L(x, y; θ, ψ, φ) = αL CV AE (x, y; θ, ψ, φ) DISPLAYFORM1 Authors report that hybrid model and GSNN outperform CVAE in terms of segmentation accuracy on the majority of datasets. We can also add that this technique seems to soften the "holes problem" . authors observe that vectors z from prior distribution may be different enough from all vectors z from the proposal distribution at the training stage, so the generator network may be confused at the testing stage. Due to this problem CVAE can have good reconstructions of y given z ∼ q φ (z|x, y), while samples of y given z ∼ p ψ (z|x) are not realistic. The same trick is applicable to our model as well: DISPLAYFORM2 In order to reflect the difference between sampling z from prior and proposal distributions, authors of CVAE use two methods of log-likelihood estimation: DISPLAYFORM3 The first estimator is called Monte-Carlo estimator and the second one is called Importance Sampling estimator (also known as IWAE). They are asymptotically equivalent, but in practice the Monte-Carlo estimator requires much more samples to obtain the same accuracy of estimation. Small S leads to underestimation of the log-likelihood for both Monte-Carlo and Importance Sampling , but for Monte-Carlo the underestimation is expressed much stronger. We perform an additional study of GSNN and hybrid model and show that they have drawbacks when the target distribution p(x|y) is has multiple different local maximums. In this section we show why GSNN cannot learn distributions with several different modes and leads to a blurry image samples. For the simplicity of the notation we consider hybrid model for a standard VAE: DISPLAYFORM0 The hybrid model for VAEAC can be obtained from by replacing x with x b and conditioning all distributions on x 1−b and b. The validity of the further equations and remains for VAEAC after this replacement. Consider now a categorical latent variable z which can take one of K values. Let x be a random variable with true distribution p d (x) to be modeled. Consider the following true data distribution: DISPLAYFORM1.., K} and some values x 1, x 2,..., x K. So the true distribution has K different equiprobable modes. Suppose the generator network N N θ which models mapping from z to some vector of parameters v z = N N θ (z). Thus, we define generative distribution as some function of these parameters: DISPLAYFORM2. Therefore, the parameters θ are just the set of v 1, v 2,..., v K.For the simplicity of the model we assume x,vj ). Using and the above formulas for q φ, p ψ and p θ we obtain the following optimization problem: It is easy to show that FORMULA1 is equivalent to the following optimization problem: DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 It is clear from that when α = 1 the log-likelihood of the initial model is optimized. On the other hand, when α = 0 the optimal point is DISPLAYFORM6 e. z doesn't influence the generative process, and for each z generator produces the same v which maximizes likelihood estimation of the generative model f (x, v) for the given dataset of x's. For Bernoulli and Gaussian generative distributions f such v is just average of all modes x 1, x 2,..., x K. That explains why further we observe blurry images when using GSNN model. The same holds for for continuous latent variables instead of categorical. Given K different modes in true data distribution, VAE uses proposal network to separate prior distribution into K components (i. e. regions in the latent space), so that each region corresponds to one mode. On the other hand, in GSNN z is sampled independently on the mode which is to be reconstructed from it, so for each z the generator have to produce parameters suitable for all modes. From this point of view, there is no difference between VAE and VAEAC. If the true conditional distribution has several different modes, then VAEAC can fit them all, while GSNN learns their average. If true conditional distribution has one mode, GSNN and VAEAC are equal, and GSNN may even learn faster because it has less parameters. Hybrid model is a trade-off between VAEAC and GSNN: the closer α to zero, the more blurry and closer to the average is the distribution of the model. The exact dependence of the model distribution on α can be derived analytically for the simple data distributions or evaluated experimentally. We perform such experimental evaluation in the next sections. In this section we show that VAEAC is capable of learning a complex multimodal distribution of synthetic data while GSNN and hybrid model are not. Let x ∈ R 2 and p(DISPLAYFORM0 is plotted in figure 6 . The dataset contains 100000 points sampled from p d (x). We use multi-layer perceptron with four ReLU layers of size 400-200-100-50, 25-dimensional Gaussian latent variables. For different mixture coefficients α we visualize samples from the learned distributions p ψ,θ (x 1, x 2), p ψ,θ (x 1 |x 2), and p ψ,θ (x 2 |x 1). The observed features for the conditional distributions are generated from the marginal distributions p(x 2) and p(x 1) respectively. We see in table 8 and in FIG5, that even with very small weight GSNN prevents model from learning distributions with several local optimas. GSNN also increases Monte-Carlo log-likelihood estimation with a few samples and decreases much more precise Importance Sampling log-likelihood estimation. When α = 0.9 the whole distribution structure is lost. We see that using α = 1 ruins multimodality of the restored distribution, so we highly recommend to use α = 1 or at least α ≈ 1. Table 9: Average negative log-likelihood of inpaintings for 1000 objects. IS-S refers to Importance Sampling log-likelihood estimation with S samples for each object. MC-S refers to Monte-Carlo log-likelihood estimation with S samples for each object. Naive Bayes is a baseline method which assumes pixels and colors independence. In FIG8 we can see that the inpaintings produced by GSNN are smooth, blurry and not diverse compared with VAEAC. Table 9 shows that VAEAC learns distribution over inpaintings better than GSNN in terms of test loglikelihood. Nevertheless, Monte-Carlo estimations with a small number of samples sometimes are better for GSNN, which means less local modes in the learned distribution and more blurriness in the samples. In FIG9 one can see that VAEAC has similar convergence speed to VAE in terms of iterations on MNIST dataset. In our experiments we observed the same behaviour for other datasets. Each iteration of VAEAC is about 1.5 times slower than VAE due to usage of three networks instead of two. We see that for some datasets MICE and MissForest outperform VAEAC, GSNN and NN. The reason is that for some datasets random forest is more natural structure than neural network. The also show that VAEAC, GSNN and NN show similar imputation performance in terms of NRMSE, PFC, post-imputation R2-score and accuracy. Given the from appendix C we can take this as a weak evidence that the distribution of imputations has only one local maximum for datasets from . doesnt use unobserved data during training, which makes it easier to apply to the missing features imputation problem. Nevertheless, it turns into a disadvantage when the fully-observed training data is available but the missingness rate at the testing stage is high. We consider the horizontal line mask for MNIST which is described in appendix A.3. We use the released GAIN code 5 with a different mask generator. The inpaintings from VAEAC which uses the unobserved pixels during training are available in figure 1. The inpaintings from GAIN which ignores unobserved pixels are provided in FIG0. As can be seen in FIG0, GAIN fails to learn conditional distribution for given mask distribution p(b).Nevertheless, we don't claim that GAIN is not suitable for image inpainting. As it was shown in the supplementary of and in the corresponding code, GAIN is able to learn conditional distributions when p(b) is pixel-wise independent Bernoulli distribution with probability 0.5. In FIG0 we provide samples of Universal Marginalizer (UM) and VAEAC for the same inputs. Consider the case when UM marginal distributions are parametrized with Gaussians. The most simple example of a distribution, which UM cannot learn but VAEAC can, is given in figure 13.

Represent the TLDR for finding the corresponding scientific article

We propose an extension of conditional variational autoencoder that allows conditioning on an arbitrary subset of the features and sampling the remaining ones.

Represent the short summary for finding the corresponding scientific article (long)

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour. Discovering high-performance neural network architectures required years of extensive research by human experts through trial and error. As far as the image classification task is concerned, state-ofthe-art convolutional neural networks are going beyond deep, chain-structured layout BID19 BID6 towards increasingly more complex, graph-structured topologies BID12 BID27 BID9. The combinatorial explosion in the design space makes handcrafted architectures not only expensive to obtain, but also likely to be suboptimal in performance. Recently, there has been a surge of interest in using algorithms to automate the manual process of architecture design. Their goal can be described as finding the optimal architecture in a given search space such that the validation accuracy is maximized on the given task. Representative architecture search algorithms can be categorized as random with weights prediction BID1, Monte Carlo Tree Search BID15, evolution BID21 BID26 BID13 BID16, and reinforcement learning BID0 BID30 BID31 BID29, among which reinforcement learning approaches have demonstrated the strongest empirical performance so far. Architecture search can be computationally very intensive as each evaluation typically requires training a neural network. Therefore, it is common to restrict the search space to reduce complexity and increase efficiency of architecture search. Various constraints that have been used include: growing a convolutional "backbone" with skip connections BID16, a linear sequence of filter banks BID1, or a directed graph where every node has exactly two predecessors BID31. In this work we constrain the search space by imposing a hierarchical network structure, while allowing flexible network topologies (directed acyclic graphs) at each level of the hierarchy. Starting from a small set of primitives such as convolutional and pooling operations at the bottom level of the hierarchy, higher-level computation graphs, or motifs, are formed by using lower-level motifs as their building blocks. The motifs at the top of the hierarchy are stacked multiple times to form the final neural network. This approach enables search algorithms to implement powerful hierarchical modules where any change in the motifs is propagated across the whole network immediately. This is analogous to the modularized design patterns used in many handcrafted architectures, e.g. VGGNet BID19, ResNet BID6, and Inception BID24 are all comprised of building blocks. In our case, a hierarchical architecture is discovered through evolutionary or random search. The evolution of neural architectures was studied as a sub-task of neuroevolution BID8 BID14 BID28 BID21 BID3, where the topology of a neural network is simultaneously evolved along with its weights and hyperparameters. The benefits of indirect encoding schemes, such as multi-scale representations, have historically been discussed in BID5; BID11 BID20 BID22. Despite these pioneer studies, evolutionary or random architecture search has not been investigated at larger scale on image classification benchmarks until recently BID16 BID13 BID26 BID1 BID15. Our work shows that the power of simple search methods can be substantially enhanced using well-designed search spaces. Our experimental setup resembles BID31, where an architecture found using reinforcement learning obtained the state-of-the-art performance on ImageNet. Our work reveals that random or evolutionary methods, which so far have been seen as less efficient, can scale and achieve competitive performance on this task if combined with a powerful architecture representation, whilst utilizing significantly less computational resources. To summarize, our main contributions are:1. We introduce hierarchical representations for describing neural network architectures. 2. We show that competitive architectures for image classification can be obtained even with simplistic random search, which demonstrates the importance of search space construction. 3. We present a scalable variant of evolutionary search which further improves the and achieves the best published 1 among evolutionary architecture search techniques. We first describe flat representations of neural architectures (Sect. 2.1), where each architecture is represented as a single directed acyclic graph of primitive operations. Then we move on to hierarchical representations (Sect. 2.2) where smaller graph motifs are used as building blocks to form larger motifs. Primitive operations are discussed in Sect. 2.3. We consider a family of neural network architectures represented by a single-source, single-sink computation graph that transforms the input at the source to the output at the sink. Each node of the graph corresponds to a feature map, and each directed edge is associated with some primitive operation (e.g. convolution, pooling, etc.) that transforms the feature map in the input node and passes it to the output node. Formally, an architecture is defined by the representation (G, o), consisting of two ingredients: DISPLAYFORM0 2. An adjacency matrix G specifying the neural network graph of operations, where G ij = k means that the k-th operation o k is to be placed between nodes i and j. The architecture is obtained by assembling operations o according to the adjacency matrix G: DISPLAYFORM1 DISPLAYFORM2 3 are assembled into a level-2 motif o1. The top row shows how level-2 motifs o in a way that the ing neural network sequentially computes the feature map x i of each node i from the feature maps x j of its predecessor nodes j following the topological ordering: DISPLAYFORM3 DISPLAYFORM4 Here, |G| is the number of nodes in a graph, and merge is an operation combining multiple feature maps into one, which in our experiments was implemented as depthwise concatenation. An alternative option of element-wise addition is less flexible as it requires the incoming feature maps to contain the same number of channels, and is strictly subsumed by concatenation if the ing x i is immediately followed by a 1 × 1 convolution. The key idea of the hierarchical architecture representation is to have several motifs at different levels of hierarchy, where lower-level motifs are used as building blocks (operations) during the construction of higher-level motifs. Consider a hierarchy of L levels where the -th level contains M motifs. The highest-level = L contains only a single motif corresponding to the full architecture, and the lowest level = 1 is the set of primitive operations. We recursively define o m, the m-th motif in level, as the composition of lower-level motifs o DISPLAYFORM0 A hierarchical architecture representation is therefore defined by {G DISPLAYFORM1 determined by network structures of motifs at all levels and the set of bottom-level primitives. The assembly process is illustrated in FIG0 . We consider the following six primitives at the bottom level of the hierarchy ( = 1, M = 6):• 1 × 1 convolution of C channels DISPLAYFORM0 If applicable, all primitives are of stride one and the convolved feature maps are padded to preserve their spatial resolution. All convolutional operations are followed by batch normalization and ReLU activation BID10; their number of channels is fixed to a constant C. We note that convolutions with larger receptive fields and more channels can be expressed as motifs of such primitives. Indeed, large receptive fields can be obtained by stacking 3 × 3 convolutions in a chain structure BID19, and wider convolutions with more channels can be obtained by merging the outputs of multiple convolutions through depthwise concatenation. We also introduce a special none op, which indicates that there is no edge between nodes i and j. It is added to the pool of operations at each level. Evolutionary search over neural network architectures can be performed by treating the representations of Sect. 2 as genotypes. We first introduce an action space for mutating hierarchical genotypes (Sect. 3.1), as well as a diversification-based scheme to obtain the initial population (Sect. 3.2). We then describe tournament selection and random search in Sect. 3.3, and our distributed implementation in Sect. 3.4. A single mutation of a hierarchical genotype consists of the following sequence of actions:1. Sample a target non-primitive level ≥ 2.2. Sample a target motif m in the target level.3. Sample a random successor node i in the target motif.4. Sample a random predecessor node j in the target motif.5. Replace the current operation o (−1) k between j and i with a randomly sampled operation o DISPLAYFORM0 In the case of flat genotypes which consist of two levels (one of which is the fixed level of primitives), the first step is omitted and is set to 2. The mutation can be summarized as: DISPLAYFORM1 2. Alter an existing edge: DISPLAYFORM2 To initialize the population of genotypes, we use the following strategy:1. Create a "trivial" genotype where each motif is set to a chain of identity mappings.2. Diversify the genotype by applying a large number (e.g. 1000) of random mutations. In contrast to several previous works where genotypes are initialized by trivial networks BID21 BID16, the above diversification-based scheme not only offers a DISPLAYFORM0 good initial coverage of the search space with non-trivial architectures, but also helps to avoid an additional bias introduced by handcrafted initialization routines. In fact, this strategy ensures initial architectures are reasonably well-performing even without any search, as suggested by our random sample in Table 1. Our evolutionary search algorithm is based on tournament selection BID4. Starting from an initial population of random genotypes, tournament selection provides a mechanism to pick promising genotypes from the population, and to place its mutated offspring back into the population. By repeating this process, the quality of the population keeps being refined over time. We always train a model from scratch for a fixed number of iterations, and we refer to the training and evaluation of a single model as an evolution step. The genotype with the highest fitness (validation accuracy) among the entire population is selected as the final output after a fixed amount of time. A tournament is formed by a random set of genotypes sampled from the current effective population, among which the individual with the highest fitness value wins the tournament. The selection pressure is controlled by the tournament size, which is set to 5% of the population size in our case. We do not remove any genotypes from the population, allowing it to grow with time, maintaining architecture diversity. Our evolution algorithm is similar to the binary tournament selection used in a recent large-scale evolutionary method BID16.We also investigated random search, a simpler strategy which has not been sufficiently explored in the literature, as an alternative to evolution. In this case, a population of genotypes is generated randomly, the fitness is computed for each genotype in the same way as done in evolution, and the genotype with the highest fitness is selected as the final output. The main advantage of this method is that it can be run in parallel over the entire population, substantially reducing the search time. Our distributed implementation is asynchronous, consisting of a single controller responsible for performing evolution over the genotypes, and a set of workers responsible for their evaluation. Both parties have access to a shared tabular memory M recording the population of genotypes and their fitness, as well as a data queue Q containing the genotypes with unknown fitness which should be evaluated. Specifically, the controller will perform tournament selection of a genotype from M whenever a worker becomes available, followed by the mutation of the selected genotype and its insertion into Q for fitness evaluation (Algorithm 1). A worker will pick up an unevaluated genotype from Q whenever there is one available, assemble it into an architecture, carry out training and validation, and then record the validation accuracy (fitness) in M (Algorithm 2). Architectures are trained from scratch for a fixed number of steps with random weight initialization. We do not rely on weight inheritance as in BID16, though incorporating it into our system is possible. Note that during architecture evolution no synchronization is required, and all workers are fully occupied. In our experiments, we use the proposed search framework to learn the architecture of a convolutional cell, rather than the entire model. The reason is that we would like to be able to quickly compute the fitness of the candidate architecture and then transfer it to a larger model, which is achieved by using less cells for fitness computation and more cells for full model evaluation. A similar approach has recently been used in BID31 BID29.Architecture search is carried out entirely on the CIFAR-10 training set, which we split into two sub-sets of 40K training and 10K validation images. Candidate models are trained on the training subset, and evaluated on the validation subset to obtain the fitness. Once the search process is over, the selected cell is plugged into a large model which is trained on the combination of training and validation sub-sets, and the accuracy is reported on the CIFAR-10 test set. We note that the test set is never used for model selection, and it is only used for final model evaluation. We also evaluate the cells, learned on CIFAR-10, in a large-scale setting on the ImageNet challenge dataset (Sect. 4.3). For CIFAR-10 experiments we use a model which consists of 3 × 3 convolution with c 0 channels, followed by 3 groups of learned convolutional cells, each group containing N cells. After each cell (with c input channels) we insert 3 × 3 separable convolution which has stride 2 and 2c channels if it is the last cell of the group, and stride 1 and c channels otherwise. The purpose of these convolutions is to control the number of channels as well as reduce the spatial resolution. The last cell is followed by global average pooling and a linear softmax layer. For fitness computation we use a smaller model with c 0 = 16 and N = 1, shown in FIG2. It is trained using SGD with 0.9 momentum for 5000 steps, starting with the learning rate 0.1, which is reduced by 10x after 4000 and 4500 steps. The batch size is 256, and the weight decay value is 3 · 10 −4. We employ standard training data augmentation where a 24 × 24 crop is randomly sampled from a 32 × 32 image, followed by random horizontal flipping. The evaluation is performed on the full size 32 × 32 image. A note on variance. We found that the variance due to optimization was non-negligible, and we believe that reporting it is important for performing a fair comparison and assessing model capabilities. When training CIFAR models, we have observed standard deviation of up to 0.2% using the exact same setup. The solution we adopted was to compute the fitness as the average accuracy over 4 training-evaluation runs. For the evaluation of the learned cell architecture on CIFAR-10, we use a larger model with c 0 = 64 and N = 2, shown in FIG2. The larger model is trained for 80K steps, starting with a learning rate 0.1, which is reduced by 10x after 40K, 60K, and 70K steps. The rest of the training settings are the same as used for fitness computation. We report mean and standard deviation computed over 5 training-evaluation runs. For the evaluation on the ILSVRC ImageNet challenge dataset BID18, we use an architecture similar to the one used for CIFAR, with the following changes. An input 299 × 299 image is passed through two convolutional layers with 32 and 64 channels and stride 2 each. It is followed by 4 groups of convolutional cells where the first group contains a single cell (and has c 0 = 64 input channels), and the remaining three groups have N = 2 cells each FIG2. We use SGD with momentum which is run for 200K steps, starting with a learning rate of 0.1, which is reduced by 10x after 100K, 150K, and 175K steps. The batch size is 1024, and weight decay is 10 −4. We did not use auxiliary losses, weight averaging, label smoothing or path dropout empirically found effective in BID31. The training augmentation is the same as in BID24, and consists in random crops, horizontal flips and brightness and contrast changes. We report the single-crop top-1 and top-5 error on the ILSVRC validation set. Figure 3: Fitness and number of parameters vs evolution step for flat and hierarchical representations. Left: fitness of a genotype generated at each evolution step. Middle: maximum fitness across all genotypes generated before each evolution step. Right: number of parameters in the small CIFAR-10 model constructed using the genotype generated at each evolution step. We run the evolution on flat and hierarchical genotypes for 7000 steps using 200 GPU workers. The initial size of the randomly initialized population is 200, which later grows as a of tournament selection and mutation (Sect. 3). For the hierarchical representation, we use three levels (L = 3), with M 1 = 6, M 2 = 6, M 3 = 1. Each of the level-2 motifs is a graph with |G | = 4 nodes, and the level-3 motif is a graph with |G | = 5 nodes. Each level-2 motif is followed by a 1 × 1 convolution with the same number of channels as on the motif input to reduce the number of parameters. For the flat representation, we used a graph with 11 nodes to achieve a comparable number of edges. The evolution process is visualized in Fig. 3. The left plot shows the fitness of the genotype generated at each step of evolution: the fitness grows fast initially, and plateaus over time. The middle plot shows the best fitness observed by each evolution step. Since the first 200 steps correspond to a random initialization and mutation starts after that, the best architecture found at step 200 corresponds to the output of random search over 200 architectures. Fig. 3 (right) shows the number of parameters in the small network (used for fitness computation), constructed using the genotype produced at each step. Notably, flat genotypes achieve higher fitness, but at the cost of larger parameter count. We thus also consider a parameter-constrained variant of the flat genotype, where only the genotypes with the number of parameters under a fixed threshold are permitted; the threshold is chosen so that the flat genotype has a similar number of parameters to the hierarchical one. In this setting hierarchical and flat genotypes achieve similar fitness. To demonstrate that improvement in fitness of the hierarchical architecture is correlated with the improvement in the accuracy of the corresponding large model trained till convergence, we plot the relative accuracy improvements in Fig. 4. Figure 4: Accuracy improvement over the course of evolution, measured with respect to the first random genotype. The small model is the model used for fitness computation during evolution (its absolute fitness value is shown with the red curve in Fig. 3 (middle) ). The large model is the model where the evolved cell architecture is deployed for training and evaluation. As far as the architecture search time is concerned, it takes 1 hour to compute the fitness of one architecture on a single P100 GPU (which involves 4 rounds of training and evaluation). Using 200 GPUs, it thus takes 1 hour to perform random search over 200 architectures and 1.5 days to do the evolutionary search with 7000 steps. This is significantly faster than 11 days using 250 GPUs reported by BID16 and 4 days using 450 GPUs reported by BID31. Table 1: Classification on the CIFAR-10 test set and ILSVRC validation set obtained using the architectures found using various representations and search methods. We now turn to the evaluation of architectures found using random and evolutionary search on CIFAR-10 and ImageNet. The are presented in Table 1.First, we note that randomly sampled architectures already perform surprisingly well, which we attribute to the representation power of our architecture spaces. Second, random search over 200 architectures achieves very competitive on both CIFAR-10 and ImageNet, which is remarkable considering it took 1 hour to carry out. This demonstrates that well-constructed architecture repre-sentations, coupled with diversified sampling and simple search form a simple but strong baseline for architecture search. Our best are achieved using evolution over hierarchical representations: 3.75% ± 0.12% classification error on the CIFAR-10 test set (using c 0 = 64 channels), which is further improved to 3.63% ± 0.10% with more channels (c 0 = 128). On the ImageNet validation set, we achieve 20.3% top-1 classification error and 5.2% top-5 error. We put these in the context of the state of the art in Tables 2 and 3. We achieve the best published on CIFAR-10 using evolutionary architecture search, and also demonstrate competitive performance compared to the best published methods on both CIFAR-10 and ImageNet. Our ImageNet model has 64M parameters, which is comparable to Inception-ResNet-v2 (55.8M) but larger than NASNet-A (22.6M). ResNet-1001 + pre-activation BID7 4.62 Wide ResNet-40-10 + dropout 3.8 DenseNet (k=24) BID9 3.74 DenseNet-BC (k=40) BID9 3.46MetaQNN BID0 6.92 NAS v3 BID30 3.65 Block-QNN-A BID29 3.60 NASNet-A BID31 3.41Evolving DNN BID13 7.3 Genetic CNN BID26 7.10 Large-scale Evolution BID16 5.4 SMASH BID1 4.03Evolutionary search, hier. repr., c0 = 64 3.75 ± 0.12 Evolutionary search, hier. repr., c0 = 128 3.63 ± 0.10 Table 2: Classification error on the CIFAR-10 test set obtained using state-of-the-art models as well as the best-performing architecture found using the proposed architecture search framework. Existing models are grouped as (from top to bottom): handcrafted architectures, architectures found using reinforcement learning, and architectures found using random or evolutionary search. Top-1 error (%) Top-5 error (%)Inception-v3 BID24 21.2 5.6 Xception BID2 21.0 5.5 Inception-ResNet-v2 BID25 19.9 4.9 NASNet-A BID31 19.2 4.7Evolutionary search, hier. repr., c0 = 64 20.3 5.2 Table 3: Classification error on the ImageNet validation set obtained using state-of-the-art models as well as the best-performing architecture found using our framework. The evolved hierarchical cell is visualized in Appendix A, which shows that architecture search have discovered a number of skip connections. For example, the cell contains a direct skip connection between input and output: nodes 1 and 5 are connected by Motif 4, which in turn contains a direct connection between input and output. The cell also contains several internal skip connections, through Motif 5 (which again comes with an input-to-output skip connection similar to Motif 4). We have presented an efficient evolutionary method that identifies high-performing neural architectures based on a novel hierarchical representation scheme, where smaller operations are used as the building blocks to form the larger ones. Notably, we show that strong can be obtained even using simplistic search algorithms, such as evolution or random search, when coupled with a well-designed architecture representation. Our best architecture yields the state-of-the-art on A ARCHITECTURE VISUALIZATION Visualization of the learned cell and motifs of our best-performing hierarchical architecture. Note that only motifs 1,3,4,5 are used to construct the cell, among which motifs 3 and 5 are dominating.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper we propose a hierarchical architecture representation in which doing random or evolutionary architecture search yields highly competitive results using fewer computational resources than the prior art.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.