Question: Represent the article for finding a single-sentence TLDR

Question: Deep generative models such as Variational AutoEncoder (VAE) and Generative Adversarial Network (GAN) play an increasingly important role in machine learning and computer vision. However, there are two fundamental issues hindering their real-world applications: the difficulty of conducting variational inference in VAE and the functional absence of encoding real-world samples in GAN. In this paper, we propose a novel algorithm named Latently Invertible Autoencoder (LIA) to address the above two issues in one framework. An invertible network and its inverse mapping are symmetrically embedded in the latent space of VAE. Thus the partial encoder first transforms the input into feature vectors and then the distribution of these feature vectors is reshaped to fit a prior by the invertible network. The decoder proceeds in the reverse order of the encoder's composite mappings. A two-stage stochasticity-free training scheme is designed to train LIA via adversarial learning, in the sense that the decoder of LIA is first trained as a standard GAN with the invertible network and then the partial encoder is learned from an autoencoder by detaching the invertible network from LIA. Experiments conducted on the FFHQ face dataset and three LSUN datasets validate the effectiveness of LIA for inference and generation. Deep generative models play a more and more important role in cracking challenges in computer vision as well as in other disciplines, such as high-quality image generation a; ), text-to-speech transformation (van den a;, information retrieval , 3D rendering , and signal-to-image acquisition . Overall, the generative models fall into four categories: autoencoder and its most important variant of Variational AutoEncoder (VAE) , auto-regressive models (van den b; a), Generative Adversarial Network (GAN) , and normalizing flows (NF) (; ;). Here we compare these models through the perspective of data dimensionality reduction and reconstruction. To be formal, let x be a data point in the d x -dimensional observable space R dx and y be its corresponding low-dimensional representation in the feature space R dy. The general formulation of dimensionality reduction is where f (·) is the mapping function and d y d x. The manifold learning aims at requiring f under various constraints on y (Tenenbaum1 et al., 2000;). However, the sparsity of data points in high-dimensional space often leads to model overfitting, thus necessitating research on opposite mapping from y to x, i.e. where g(·) is the opposite mapping function with respect to f (·), to reconstruct the data. In general, the role of g(·) is a regularizer to f (·) or a generator to produce more data. The autoencoder is of mapping x f → y g →x. A common assumption in autoencoder is that the variables in lowdimensional space are usually sampled from a prior distribution P(z; θ) such as uniform or Gaussian. To differentiate from y, we let z represent the low-dimensional vector following the prior distribution. Thus we can write g: R dz → R dx, z → x = g(z), z ∼ P(z; θ). It is crucial to establish such dual maps z = f (x) and x = g(z). In the parlance of probability, the process of x → z = f (x) is called inference, and the other procedure of z → x = g(z) is called sampling or generation. VAE is capable of carrying out inference and generation in one framework by two collaborative functional modules. However, it is known that in many cases VAEs are only able to generate blurry images due to the imprecise variational inference. To see this, we write the approximation of the marginal log-likelihood where KL[q(z|x)||p(z)] is the Kullback-Leibler divergence with respect to posterior probability q(z|x) and prior p(z). This lower-bound log-likelihood usually produces imprecise inference. Furthermore, the posterior collapse frequently occurs when using more sophisticated decoder models . These two issues greatly limit the generation capability of the VAE. On the other hand, GAN is able to achieve photo-realistic generation (a; . However, its critical limitation is the absence of the encoder f (x) for carrying inference on real images. Effort has been made on learning an encoder for GAN under the framework of VAE, however the previous two issues of learning VAE still exist. Normalizing flows can perform the exact inference and generation with one architecture by virtue of invertible networks . But it requires the dimension d x of the data space to be identical to the dimension d z of the latent space, thus posing computational issues due to high complexity of learning deep flows and computing the Jacobian matrices. Inspired by recent success of GANs (a; and normalizing flows , we develop a new model called Latently Invertible Autoencoder (LIA). LIA utilizes an invertible network to bridge the encoder and the decoder of VAE in a symmetric manner. We summarize its key advantages as follows: • The symmetric design of the invertible network brings two benefits. The prior distribution can be exactly fitted from an unfolded feature space, thus significantly easing the inference problem. Besides, since the latent space is detached, the autoencoder can be trained without variational optimization thus there is no approximation here. • The two-stage adversarial learning decomposes the LIA framework into a Wasserstein GAN (only a prior needed) and a standard autoencoder without stochastic variables. Therefore the training is deterministic 2, implying that the model will be not affected by the posterior collapse when the decoder is more complex or followed by additional losses such as the adversarial loss and the perceptual loss. • We compare LIA with state-of-the-art generative models on inference and generation/reconstruction. The experimental on FFHQ and LSUN datasets show the LIA achieves superior performance on inference and generation. The neural architecture of LIA is designed such that the data distribution can be progressively unrolled from a complex or manifold-valued one to a given simple prior. We now describe the detail as follows. Framework of LIA is based on the classic VAE and the realization of normalizing flow. As shown in Figure 1f, we symmetrically embed an invertible neural network in the latent space of VAE, following the diagram of mapping process as, an invertible network φ to reshape feature embeddings to match the prior distribution z = φ(x) and φ −1 to map latent variables to feature vectors y = ψ −1 (z), a decoder to produce outputx = g(ỹ), a feature extractor to perform reconstruction measure, and a discriminator c to distinguish real/fake distributions. where φ = φ 1 • · · · • φ k denotes the deep composite mapping of normalizing flow with depth k. LIA first performs nonlinear dimensionality reduction on the input data x and transform them into the low-dimensional feature space R dy. The role of f (x) for LIA can be regarded to unfold the underlying data manifold, as illustrated in Figures 1a and 1b. Therefore, the Euclidean operations such as linear interpolation and vector arithmetic are reliable in this feature space. Then we establish an invertible mapping φ(y) from the feature y to the latent variable z, as opposed to the VAEs that directly map original data to latent variables. The feature y can be exactly recovered via the invertibility of φ from z, which is the advantage of using invertible networks. The recovered feature y is then fed into a partial decoder g(y) to generate the corresponding datax. If the maps φ and φ In general, any invertible networks are applicable in the LIA framework. We find in practice that a simple invertible network is sufficiently capable of constructing the mapping from the feature space R dy to the latent space R dz. Let x = [x t ; x b] and z = [z t ; z b] be the forms of the top and bottom fractions of x and z, respectively. Then the invertible network can be built as follows, where τ is the transformation that can be an arbitrary differentiable function. Alternatively, one can attempt to exploit the complex invertible network with affine coupling mappings for more challenging tasks . As conducted in , we set τ as a multi-layer perceptron with the leaky ReLU activation. To guarantee the precise reconstructionx, the conventional way by (variational) autoencoders is to use the distance x −x or the cross entropy directly between x andx. Here, we utilize the perceptual loss that is proven to be more robust to variations of image details . Let denote the feature extractor, e.g. VGG . Then we can write the loss It suffices to emphasize that the functionality of here is in essence to produce the representations of the input x and the outputx. The acquisition of is fairly flexible. It can be attained by supervised or (b) Encoder training. unsupervised learning, meaning that can be trained with class labels or without class labels (van den). The norm-based reconstruction constraints usually incur the blurry image generation in autoencoderlike architectures. This problem can be handled via the adversarial learning . To do so, a discriminator c is employed to balance the loss of the comparison between x andx. Using the ), we can write the optimization objective as where P x and Px denote the probability distributions of the real data and the generated data, respectively. γ is the hyper-parameter of the regularization. The R 1 regularizer is formulated in , which is proven more stable for training convergence. Training deep models usually follows an end-to-end fashion for the whole architecture. To backpropagate gradients through random variables for VAE, the reparameterization trick is harnessed , i.e. z = µ + σ * N, where µ is the mean and σ the standard deviation. The regularization of coupling the prior and the posterior is the KL divergence used to optimize the parameters of the encoder by backpropagation. For our framework, however, we find that this endto-end learning strategy cannot lead the algorithm to converge to a satisfactory optima. To proceed, we propose a scheme of two-stage stochasticity-free training, which decomposes the framework into two parts that can be well trained end-to-end respectively, as shown in Figure 2. At the first step, the decoder of LIA is trained using adversarial learning with invertible network to acquire the ability of high-quality generation. At the second step, the invertible network that connects feature space and latent space is detached from the architecture, reducing the framework to a standard autoencoder without variational inference. Thus this two-stage design prevents the posterior collapse issue while facilitates the decoder with more complex structure as well as adversarial loss for high-resolution image generation. ProGAN (a), StyleGAN (b), and BigGAN are capable of generating photo-realistic images from random noise sampled from some prior distribution. Then it is naturally supposed that such GAN models are applicable to recover a precisex if we can find the latent variable z for the given x. Namely, we may train the associated GAN model separately in the LIA framework. To conduct this, we single out a standard GAN model for the first-stage training, as displayed in Figure 2a. Here z is directly sampled from a prior distribution. According to the principle of Wasserstein GAN, the optimization objective can be written as where the superscript * denotes that the parameters of corresponding mappings have already been learned. It is worth noting that the role of the invertible network here is just its transformation invertibility. We do not pose any constraints on the probabilities of z and φ(y) in contrast to normalizing flows. Our strategy of attaching an invertible network in front of the generator can be potentially applied to any GAN models. In the LIA architecture, the invertible network is embedded in the latent space in a symmetric fashion, in the sense that f (x) = y = φ −1 (z). The unique characteristic of the invertible network allows us to detach the invertible network φ from the LIA framework. Thus we attain a conventional autoencoder without stochastic variables, as shown in Figure 2b. In practice, the feature extractor in perceptual loss is the VGG weight up to conv4 pretrained on the ImageNet dataset. After the first-stage encoder training, the parameter of f is needed to be learned as where β is the hyper-parameter. The above optimization serving to the architecture in Figure 2b is widely applied in computer vision. It is the backbone framework of various GANs for diverse image processing tasks. For LIA, however, it is much simpler because we only need to learn the partial encoder f. This simplicity brought by the two-stage training is able to enforce the encoder to converge with more precise inference. Our LIA model is relevant to the works that solve the inference problem for VAEs with adversarial learning as well as the works that design encoders for GANs. The integration of GAN with VAE can be traced back to the work of VAE/GAN and implicit autoencoders . These methods encounter the difficulty of end-to-end training, because the gradients are prone to becoming unstable after going through the latent space in deep complex architectures . Besides, there is an intriguing attempt of training VAE in the adversarial manner . These approaches confront the trade-off between the roles of the encoder that performs inference and compares the real/fake distributions. This is difficult to tune. So we prefer the complete GAN with an indispensable discriminator. The closely related works to LIA are the models of combining VAE and the inverse autoregressive flow and the latent-flow-based VAE approach that are VAEs with latent variables conditioned by normalizing flows . These three models all need to optimize the posterior probability of normalizing flows, which is essentially different from our deterministic optimization in LIA. The stochasticity-free training is directly derived from the symmetric design of the invertible network in the latent space, which is different from and . There are alternative attempts of specifying the generator of GAN with normalizing flow or mapping images into feature space with partially invertible network . These approach suffers from high complexity computation for high dimensions. The approach of two-stage training in is incapable of solving the posterior estimation. It worth noticing that the reconstruction task we focus here is different to the recent work of representation learning which learns features for recognition using adversarial inference; ). Our primary goal is to faithfully reconstruct real images from the latent code. For experimental setup, we instantiate the decoder of LIA with the generator of StyleGAN (b). The difference is that we replace the mapping network (MLP) in StyleGAN with the invertible network. The layer number of the invertible network is 8. The hyper-parameters for the discriminator are γ = 10 (equation) and β = 0.001 (equation). For perceptual loss in equation, we take = conv4_3 from the VGG weight. The generative models we compare are the MSE-based optimization methods (; ;) 3, the adversarially learned inference (ALI), and the adversarial generator-encoder (AGE) network (the necessity of the invertible network, we also train an encoder and a StyleGAN with its original multi-layer perceptron, which is the last column in Figure 3 . The two-stage training scheme is used as LIA does. The generator and discriminator of the StyleGAN is exactly same to that of StyleGAN. For quantitative evaluation metrics, we use Fréchet inception distance (FID), sliced Wasserstein distance (SWD), and mean square error (MSE). These three metrics are commonly used to measure the numerical accuracy of generative algorithms (; a; ; b). We directly use the code released by the authors of ProGAN (a). The prior for z is Gaussian and the dimension is 512. Figure 4: Manipulating reconstructed faces. The algorithm in is applied to manipulate the image reconstructed from the latent code w given by LIA. Each row shows the original image, the reconstruction, glass, pose, gender, smile, and age. All models are first tested on the Flickr-Faces-HQ (FFHQ) database 4 created by the authors of StyleGAN as the benchmark. FFHQ contains 70,000 high-quality face images. We take the first 65,000 faces as the training set and the remaining 5,000 faces as the reconstruction test according to the exact order of the dataset. We do not split the dataset by random sampling for interested readers can precisely reproduce all the reported with our experimental protocol. The exemplar real images of objects and scenes from LSUN database and their reconstructed images by LIA. Three categories are tested, i.e. cat, bedroom, and car. For each group, the first row is the original images, the second row shows the reconstructed images. Figure 3 shows the reconstructed faces of all models. It is clear that LIA significantly outperforms others. The reconstructed faces by ALI and AGE look correct, but the quality is mediocre. The ideas of ALI and AGE are elegant. Their performance may be improved with the new techniques such as progressive growing of neural architecture or style-based one. The method of the MSE-based optimization produces facial parts of comparable quality with LIA when the faces are normal. But this approach fails when the variations of faces become large. For example, the failure comes from the long fair, hats, beard, and large pose. The interesting phenomenon is that the StyleGAN with encoder only does not succeed in recovering the target faces using the same training strategy as LIA, even though it is capable of generating photo-realistic faces in high quality due to the StyleGAN generator. This indicates that the invertible network plays the crucial role to make the LIA work. The quantitative in Table 1 shows the consistent superiority of LIA. More reconstruction are included in Appendix A.5. The reconstruction from LIA facilitates semantic photo editing. Figure 4 shows the manipulation on reconstructed faces. More can be found in Appendix A.7. The interpolation and style mixing are also displayed for reference in Appendix A.2. To further evaluate LIA on the data with large variations, we use the three categories from the large-scale LSUN database , i.e. cat, car, and bedroom. For each category, the 0.1 million images are selected by ranking algorithm from the first 0.5 million images in the dataset. Each cat and bedroom images are resized to be 128 × 128 and the size of the car image is 128 × 96 for training. We take subsets because it does not take too long for training to converge while still maintains the data complexity. These subsets will be made available for evaluation. Figure 5 shows that the reconstructed objects by LIA faithfully maintain the semantics as well as the appearance of the original ones. For example, the cats' whiskers are recovered, indicating that LIA is able to recover very detailed information. More are attached in Appendix A.6. We can see that LIA significantly improves the reconstruction quality. The improvement mainly comes from the two-stage training of LIA. The decoder trained with adversarial learning guarantees that the generated images are photo-realistic. The encoder deterministically trained with perceptual and adversarial losses ensures that latent feature vectors can be obtained more precisely. This two-stage training is enabled by the design that the invertible network detachs the encoder and decoder, thus avoiding the optimization of the posterior probability when learning the encoder. Figure 6 shows the Figure 7 clearly illustrates the difference of gradients between these two cases. The gradient volatility for variational inference is high and the associated loss is not effectively reduced, meaning that the gradients during training are noisy and not always informative. This may indicate that the stochasticity in latent space causes problems for training encoder via variational inference. Instead, the encoder's gradients for LIA are rather stable across different layers and the loss decreases monotonically, showing the importance of the stochasticity-free training and the invertible network. A new generative model, named Latently Invertible Autoencoder (LIA), has been proposed for generating image sample from a probability prior and simultaneously inferring accurate latent code for a given sample. The core idea of LIA is to symmetrically embed an invertible network in an autoencoder. Then the neural architecture is trained with adversarial learning as two decomposed modules. With the design of two-stage training, the decoder can be replaced with any GAN generator for high-resolution image generation. The role of the invertible network is to remove any probability optimization and bridge the prior with unfolded feature vectors. The effectiveness of LIA is validated with experiments of reconstruction (inference and generation) on FFHQ and LSUN datasets. It is still challenging to faithfully recover all the image content especially when the objects or scenes have unusual parts. For example, LIA fails to recover the hand appeared at the top of the little girl (the second row in Figure 3). Besides, the Bombay cat's necklace (the second row in Figure 5) is missed in the reconstructed image. These features belong to multiple unique parts of the objects or scenes, which are difficult for the generative model to capture. One possible solution is to raise the dimension of latent variables (e.g. using multiple latent vectors) or employ the attention mechanism to highlight such unusual structures in the decoder, which is left for future work. Examining the interpolation in the latent feature space is an effective way of visualizing the capability of generative models as well as measuring how well it fits the underlying data distribution. Here we compare the three algorithms. As shown in Figure 9, LIA achieves smoother interpolation while well preserve the facial properties. The interpolation quality of the MSE-based optimization is actually based on the reconstruction performance because it has a good generator (StyleGAN). The intermediate interpolation from Glow deviates from real faces. Figure 9: Interpolation by three generative models. The first row shows the of LIA (ours), the second row that of the MSE-based optimization, and the last row that of Glow. The first and last faces in the Glow are the real face images of FFHQ. We further perform style mixing using a small set of reconstructed faces. Style mixing is conducted using the same approach presented in (b). The difference is that LIA uses the real faces thanks to its encoding capability. Figure 10 shows that our algorithm can infer accurate latent codes and generate high-quality mixed faces. Figure 13: Generated faces along the optimization path with the latent code z and the feature w. The path is formed by the optimization of the MSE loss with respect to z and w, respectively. The faces in the first column are generated by the initial values of z and w. Then the following faces are sequentially generated with the optimization process until the optimization converges. For each group, the first row shows the with z and the second row shows the with w. The disentanglement effect of w is clearly shown. Instead, the latent code z suffers from the entanglement of features.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A new model Latently Invertible Autoencoder is proposed to solve the problem of variational inference in VAE using the invertible network and two-stage adversarial training.

Represent the TLDR for finding the corresponding scientific article

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism. Despite the potential to increase the interpretability and the compositionality of the behavior of artificial agents, it remains difficult to learn from demonstrations neural networks that represent computer programs. The main challenges that set algorithmic domains apart from other imitation learning domains are the need for high accuracy, the involvement of specific structures of data, and the extremely limited observability. To address these challenges, we propose to model programs as Parametrized Hierarchical Procedures (PHPs). A PHP is a sequence of conditional operations, using a program counter along with the observation to select between taking an elementary action, invoking another PHP as a sub-procedure, and returning to the caller. We develop an algorithm for training PHPs from a set of supervisor demonstrations, only some of which are annotated with the internal call structure, and apply it to efficient level-wise training of multi-level PHPs. We show in two benchmarks, NanoCraft and long-hand addition, that PHPs can learn neural programs more accurately from smaller amounts of both annotated and unannotated demonstrations. Representing the logic of a computer program with a parametrized model, such as a neural network, is a central challenge in AI with applications including reinforcement learning, robotics, natural language processing, and programming by example. A salient feature of recently-proposed approaches for learning programs BID32 BID6 is their ability to leverage the hierarchical structure of procedure invocations present in well-designed programs. Explicitly exposing this hierarchical structure enables learning neural programs with empirically superior generalization, compared to baseline methods that learn only from elementary computer operations, but requires training data that does not consists only of low-level computer operations but is annotated with the higher-level procedure calls BID32 BID6. tackled the problem of learning hierarchical neural programs from a mixture of annotated training data (hereafter called strong supervision) and unannotated training data where only the elementary operations are given without their call-stack annotations (called weak supervision). In this paper, we propose to learn hierarchical neural programs from a mixture of strongly supervised and weakly supervised data via the Expectation-Gradient method and an explicit program counter, in lieu of a high-dimensional real-valued state of a recurrent neural network. Our approach is inspired by recent work in robot learning and control. In Imitation Learning (IL), an agent learns to behave in its environment using supervisor demonstrations of the intended behavior. However, existing approaches to IL are largely insufficient for addressing algorithmic domains, in which the target policy is program-like in its accurate and structured manipulation of inputs and data structures. An example of such a domain is long-hand addition, where the computer loops over the digits to be added, from least to most significant, calculating the sum and carry. In more complicated examples, the agent must correctly manipulate data structures to compute the right output. Three main challenges set algorithmic domains apart from other IL domains. First, the agent's policy must be highly accurate. Algorithmic behavior is characterized by a hard constraint of output correctness, where any suboptimal actions are simply wrong and considered failures. In contrast, many tasks in physical and simulated domains tolerate errors in the agent's actions, as long as some goal region in state-space is eventually reached, or some safety constraints are satisfied. A second challenge is that algorithms often use specific data structures, which may require the algorithmic policies to have a particular structure. A third challenge is that the environment in algorithmic domains, which consists of the program input and the data structures, is almost completely unobservable directly by the agent. They can only be scanned using some limited reading apparatus, such as the read/write heads in a Turing Machine or the registers in a register machine. Recently proposed methods can infer from demonstration data hierarchical control policies, where high-level behaviors are composed of low-level manipulation primitives BID8. In this paper, we take a similar approach to address the challenges of algorithmic domains, by introducing Parametrized Hierarchical Procedures (PHPs), a structured model of algorithmic policies inspired by the options framework BID38, as well as the procedural programming paradigm. A PHP is a sequence of statements, such that each statement branches conditionally on the observation, to either perform an elementary operation, invoke another PHP as a sub-procedure, or terminate and return control to the caller PHP. The index of each statement in the sequence serves as a program counter to accurately remember which statement was last executed and which one is next. The conditional branching in each statement is implemented by a neural network mapping the program counter and the agent's observation into the elementary operation, sub-procedure, or termination to be executed. The PHP model is detailed in Section 4.1.PHPs have the potential to address the challenges of algorithmic domains by strictly maintaining two internal structures: a call stack containing the current branch of caller PHPs, and the current program counter of each PHP in the stack. When a statement invokes a PHP as a sub-procedure, this PHP is pushed into the call stack. When a statement terminates the current PHP, it is popped from the stack, returning control to the calling PHP to execute its next statement (or, in the case of the root PHP, ending the entire episode). The stack also keeps the program counter of each PHP, which starts at 0, and is incremented each time a non-terminating statement is executed. PHPs impose a constraining structure on the learned policies. The call stack arranges the policy into a hierarchical structure, where a higher-level PHP can solve a task by invoking lower-level PHPs that solve sub-tasks. Since call stacks and program counters are widely useful in computer programs, they provide a strong inductive bias towards policy correctness in domains that conform to these constraints, while also being computationally tractable to learn. To support a larger variety of algorithmic domains, PHPs should be extended in future work to more expressive structures, for example allowing procedures to take arguments. We experiment with PHPs in two benchmarks, the NanoCraft domain introduced in, and long-hand addition. We find that our algorithm is able to learn PHPs from a mixture of strongly and weakly supervised demonstrations with better sample complexity than previous algorithms: it achieves better test performance with fewer demonstrations. In this paper we make three main contributions:• We introduce the PHP model and show that it is easier to learn than the NPI model BID32 ).• We propose an Expectation-Gradient algorithm for efficiently training PHPs from a mixture of annotated and unannotated demonstrations (strong and weak supervision).• We demonstrate efficient training of multi-level PHPs on NanoCraft and long-hand addition BID32, and achieve improved success rate.2 RELATED WORK BID32 Recursive NPI BID6 (recursive) NPL Mixed PHP (this work) Mixed BID18, the Neural GPU BID19, and End-to-End Memory Networks BID37, have been proposed for learning neural programs from input-output examples, with components such as variable-sized memory and novel addressing mechanisms facilitating the training process. In contrast, our work considers the setting where, along with the input-output examples, execution traces are available which describe the steps necessary to solve a given problem. The Neural Programmer-Interpreter (NPI, BID32) learns hierarchical policies from execution traces which not only indicate the low-level actions to perform, but also a structure over them specified by higher-level abstractions. BID6 showed that learning from an execution trace with recursive structure enables perfect generalization. Neural Program Lattices work within the same setting as the NPI, but can learn from a dataset of execution traces where only a small fraction contains information about the higher-level hierarchy. In demonstrations where the hierarchical structure along the trace is missing, this latent space grows exponentially in the trace length. address this challenge via an approximation method that selectively averages latent variables on different computation paths to reduce the complexity of enumerating all paths. In contrast, we compute exact gradients using dynamic programming, by considering a hierarchical structure that has small discrete latent variables in each time step. Other works use neural networks as a tool for outputting programs written in a discrete programming language, rather than having the neural network itself represent a program. BID3 learned to generate programs for solving competition-style problems. BID9 and BID31 generate programs in a domain-specific language for manipulating strings in spreadsheets. Automatic discovery of hierarchical structure has been well-studied, and successful approaches include action-sequence compression BID39, identifying important transitional states BID28 BID29 Şimşek &; BID36 BID25, learning from demonstrations BID5 BID22 BID7 BID23, considering the set of initial states from which the MDP can be solved BID20 BID21, policy gradients BID26, information-theoretic considerations BID13 BID11 BID17 BID10, active learning BID15, and recently value-function approximation BID2 BID16 BID34.Our approach is inspired by the Discovery of Deep Options (DDO) algorithm of. Following the work of BID8, who use Expectation-Maximization (EM) to train an Abstract Hidden Markov Model BID5, DDO parametrizes the model with neural networks where complete maximization in the M-step is infeasible. Instead, DDO uses Expectation-Gradient (EG) to take a single gradient step using the same forward-backward E-step as in the EM algorithm. A variant of DDO for continuous action spaces (DDCO) has shown success in simulated and physical robot control. This paper extends DDO by proposing an E-step that can infer a call-stack of procedures and their program counters. Computation can be modeled as a deterministic dynamical system, where the computer is an agent interacting with its environment, which consists of the program input and its data structures. Mathematically, the environment is a Deterministic Partially Observable Markov Decision Process (DET-POMDP BID4), which consists of a state space S, an observation space O, an action space A, the state-dependent observation o t ps t q, and the state transition s t`1 " f ps t, a t q. The initial state s 0 includes the program input, and is generated by some distribution p 0 ps 0 q. This notation is general enough to model various computation processes. In a Turing Machine, for example, s t is the machine's configuration, o t is the vector of tape symbols under the read/write heads, and a t contains writing and moving instructions for the heads. In partially observable environments, the agent often benefits from maintaining memory m t of past observations, which reveals temporarily hidden aspects of the current state. The agent has a parametrized stochastic policy π θ, in some parametric family θ P Θ, where π θ pm t, a t |m t´1, o t q is the probability of updating the memory state from m t´1 to m t and taking action a t, when the observation is o t. The policy can be rolled out to induce the stochastic process ps 0:T, o 0:T, m 0:T´1, a 0:T´1 q, such that upon observing o T the agent chooses to terminate the process. In a computation device, the memory m t stands for its internal state, such as the Finite State Machine of a Turing Machine. We can scale computer programs by adding data structures to their internal state, such as a call stack, which we model in the next section as Parametrized Hierarchical Procedures. In Imitation Learning (IL), the learner is provided with direct supervision of the correct actions to take. The setting we use is Behavior Cloning (BC), where the supervisor rolls out its policy to generate a batch of demonstrations before learning begins, and the agent's policy is trained to minimize a loss on its own selection of actions in demonstrated states, with respect to the demonstrated actions. In strong supervision, a demonstration contains not only the sequence of observable variables ξ " po 0:T, a 0:T´1 q, where a 0:T´1 is the sequence of supervisor actions during the demonstration, but also the sequence of the supervisor's memory states ζ " m 0:T´1, which are ordinarily latent. This allows the agent to directly imitate not just the actions, but also the memory updates of the supervisor, for example by maximizing the log-likelihood of the policy given the demonstrations arg max DISPLAYFORM0 the latter being the negative cross-entropy loss with respect to the demonstrations. In weak supervision, on the other hand, only the observable trajectories ξ are given as demonstrations. This makes it difficult to maximize the likelihood Ppξ|θq " ř ζ Ppζ, ξ|θq, due to the large space of possible memory trajectories ζ. We address this difficulty via the Expectation-Gradient algorithm described in Section 4. The semantics of this definition are given by the following control policy. The agent's memory maintains a stack rph 1, τ 1 q,..., ph n, τ n qs of the active procedures and their program counters. Initially, this stack contains only the root procedure and the counter is 0. Upon observing o t, the agent checks whether the top procedure should terminate, i.e. ψ τn hn po t q " 1. If the procedure h n terminates, it is popped from the stack, the next termination condition ψ τn´1 hn´1 po t q is consulted, and so on. For the first procedure h i that does not terminate, we select the operation η τi`1 hi po t q, after incrementing the program counter τ i. If this operation is an invocation of procedure h 1 i`1, we push ph 1 i`1, 0q onto the stack, consult its operation statement η 0 h 1 i`1 po t q, and so on. Upon the first procedure h 1 n 1 to select an elementary action a t, we save the new memory state m t " rph 1, τ 1 q,..., ph i´1, τ i´1 q, ph i, τ i`1 q, ph 1 i`1, 0q,..., ph 1 n 1, 0qs, and take the action a t in the environment. DISPLAYFORM1 The call stack and program counters act as memory for the agent, so that it can remember certain hidden aspects of the state that were observed before. In principle, any finite memory structure can be implemented with sufficiently many PHPs, by having a distinct procedure for each memory state. However, PHPs leverage the call stack and program counters to allow exponentially many memory states to be expressed with a relatively small set of PHPs. We impose two practical limitations on the general definition of PHPs. Our training algorithm in Section 4.2 does not support recursive procedures, i.e. cycles in the invocation graph. In addition, for simplicity, we allow each procedure to either invoke other procedures or execute elementary actions, not both. These two limitations are achieved by layering the procedures in levels, such that only the lowest-level procedures can execute elementary actions, and each higher-level procedure can only invoke procedures in the level directly below it. This does not lose generality, since instead of invoking a procedure or action at a certain level, we can wrap it in a one-level-higher surrogate procedure that invokes it and terminates. A Parametrized Hierarchical Procedure (PHP) is a representation of a hierarchical procedure by differentiable parametrization. In this paper, we represent each PHP by two multi-layer perceptrons (MLPs) with ReLU activations, one for the PHP's operation statement and one for its termination statement. The input is a concatenation of the observation o and the program counter τ, where τ is provided to the MLPs as a real number. During training, we apply the soft-argmax activation function to the output of each MLP to obtain stochastic statements η τ h p¨|o t q and ψ τ h p¨|o t q. During testing, we replace the soft-argmax with argmax, to obtain deterministic statements as above. In weak supervision, only the observable trajectory ξ " po 0:T, a 0:T´1 q is available in a demonstration, and the sequence of memory states ζ " m 0:T´1 is latent. This poses a challenge, since the space of possible memory trajectories ζ grows exponentially in the length of the demonstration, which at first seems to prohibit the computation of the log-likelihood gradient ∇ θ log Ppξ|π θ q, needed to maximize the log-likelihood via gradient ascent. We use the Expectation-Gradient (EG) method to overcome this challenge BID33. This method has been previously used in dynamical settings to play Atari games and to control simulated and physical robots. The EG trick expresses the gradient of the observable log-likelihood as the expected gradient of the full log-likelihood: DISPLAYFORM0 where the first and third equations follow from two applications of the identity ∇ θ x " x∇ θ log x. In the E-step of the EG algorithm, we find the posterior distribution of ζ given the observed ξ and the current parameter θ. In the G-step, we use this posterior to calculate and apply the exact gradient of the observable log-likelihood. We start by assuming a shallow hierarchy, where the root PHP calls level-one PHPs that only perform elementary operations. At any time t, the stack contains two PHPs, the root PHP and the PHP it invoked to select the elementary action. The stack also contains the program counters of these two PHPs, however we ignore the root counter to reduce complexity, and bring it back when we discuss multi-level hierarchies in the next section. Let us denote by η τ h pa t |o t q and ψ τ h pb t |o t q, respectively, the stochastic operation and termination statements of procedure h P H Y tKu, where K is the root PHP. Let ph t, τ t q be the top stack frame when action a t is selected. Then the full likelihood Ppζ, ξ|θq of the policy given an annotated demonstration is a product of the terms that generate the demonstration, including η τt ht pa t |o t q for the generation of each a t, as well as ψ τt´1 ht´1 p1|o t qη K ph t |o t q whenever h t´1 terminates and h t is pushed with τ t " 0, and ψ τt´1 ht´1 p0|o t q whenever h t´1 does not terminate (i.e. h t " h t´1 and τ t " τ t´1`1). Crucially, the form of Ppζ, ξ|θq as a product implies that ∇ θ log Ppζ, ξ|θq decomposes into a sum of policy-gradient terms such as ∇ θ log η τt ht pa t |o t q, and computing its expectation over Ppζ|ξ, θq only requires the marginal posterior distributions over single-step latent variables v t ph, τ q " Pph t "h, τ t "τ |ξ, θq w t ph, τ q " Pph t "h, τ t "τ, τ t`1 "τ`1|ξ, θq. The marginal posteriors v t and w t can be found via a forward-backward algorithm, as described in Appendix A, and used to compute the exact gradient DISPLAYFORM0 t ph, τ q∇ θ log η τ h pa t |o t q w t ph, τ q∇ θ log ψ τ h p0|o t`1 qq pv t ph, τ q´w t ph, τ qq∇ θ log ψ τ h p1|o t`1 q¸¸. A naive attempt to generalize the same approach to multi-level PHPs would in an exponential blow-up of the forward-backward state, which would need to include the entire stack. Instead, we train each level separately, iterating over the PHP hierarchy from the lowest level to the highest. Let us denote by d the number of levels in the hierarchy, with 0 being the root and d´1 the lowest level, then we train level i in the hierarchy after we have trained levels i`1,..., d´1.Two components are required to allow this separation. First, we need to use our trained levels i`1,..., d´1 to abstract away from the elementary actions, and generate demonstrations where the level-pi`1q PHPs are treated as the new elementary operations. In this way, we can view level-i PHPs as the new lowest-level PHPs, whose operations are elementary in the demonstrations. This is easy to do in strongly supervised demonstrations, since we have the complete stack, and we only need to truncate the lowest d´i´1 levels. In weakly supervised demonstrations, on the other hand, we need an algorithm for decoding the observable trajectories, and replacing the elementary actions with higher-level operations. We present such an algorithm below. The second component needed for level-wise training is approximate separation from higher levels that have not been trained yet. When we train level i ą 1 via the EG algorithm in the previous section, the "root PHP" would be at level i´1, had it corresponded to any real PHP. In all but the simplest domains, we cannot expect a single PHP to perfectly match the behavior of the i-levels PHP hierarchy (levels 0, . . ., i´1) that actually selected the level-i PHPs that generated the demonstrations. To facilitate better separation from higher levels, we augment the "root PHP" used for training with an LSTM that approximates the i-levels stack memory as η LST M K ph t |o 0,..., o t q. As mentioned above, abstraction from lower levels is achieved by rewriting weakly supervised demonstrations to show level-pi`1q operations as elementary. After level i`1 is trained, the levelpi`1q PHPs that generated the demonstrations are decoded using the trained parameters. We considered three different decoding algorithms: finding the most likely level-pi`1q PHP at each time step, by taking the maximum over v t; using a Viterbi-like algorithm to find the most likely latent trajectory of level-pi`1q PHPs; sampling from the posterior distribution Ppζ|ξ, θq over latent trajectories. In our current experiments we used latent trajectories sampled from the posterior distribution, given by DISPLAYFORM0 where the denominators can be computed via the same forward-backward algorithm used in the previous section to compute v t and w t, as detailed in Appendix A. We evaluate our proposed method on the two settings studied by: NanoCraft, which involves an agent interacting in a grid world, and long-hand addition, which was also considered by BID32 and BID6. Task description. The NanoCraft domain, introduced by, involves placing blocks in a two-dimensional grid world. The goal of the task is to control an agent to build a rectangular building of a particular height and width, at a specified location within the grid, by moving around the grid and placing blocks in appropriate cells. The state contains a 6ˆ6 grid. In our version, each grid cell can either be empty or contain a block. The state also includes the current location of the agent, as well as the building's desired height, width, and location, expressed as the offset from the agent's initial location at the top left corner. Initially, some of the blocks are already in place and must not be placed again. The state-dependent observation o t ps t q reveals whether the grid cell at which the agent is located contains a block or not, and four numbers for the building's specifications. We provide each observation to the MLPs as a 5-dimensional real-valued feature vector. PHPs and elementary actions. The top-level PHP nanocraft executes (moves_r, moves_d, builds_r, builds_d, builds_l, builds_u, return). moves_r calls move_r a number of times equal to the building's horizontal location, and similarly for moves_d w.r.t. move_d and the vertical location; builds_r w.r.t. build_r and the building's width; and so on for builds_d, builds_l, and builds_u. At the lowest level, move_r takes the elementary action MOVE_RIGHT and terminates, and similarly for move_d taking MOVE_DOWN. build_r executes (MOVE_RIGHT, if cell full: return, else: PLACE_BLOCK, return), and similarly for build_d, build_l, and build_u w.r.t. MOVE_DOWN, MOVE_LEFT, and MOVE_UP.Experiment setup. We trained our model on datasets of 16, 32, and 64 demonstrations, of which some are strongly supervised and the rest weakly supervised. We trained each level for 2000 iterations, iteratively from the lowest level to the highest. The are averaged over 5 trials with independent datasets. Results. Our summarized in FIG2 show that 32 strongly supervised demonstrations are sufficient for achieving perfect performance at the task, and that 16 such demonstrations approach the same success rate when used along with weakly supervised demonstrations, for a total of 16, 32, or 64 demonstrations. An interesting question is whether these performance gains are due to the simplicity of the PHP model itself, the use of exact gradients in its optimization via the EG algorithm, or both. The PHP and NPL/NPI experiments with 64 strongly supervised demonstrations FIG2, blue curves at the 64 mark) directly compare the PHP model with the NPI model, since both algorithms use exact gradients in this case. 1 The accuracy is 1.0 for PHP; 0.724 for NPL/NPI, suggesting that the gains of PHP are at least in part due to the PHP model inducing an optimization landscape in which a good solution is easier to find. In the experiments with 16 strongly supervised demonstrations of a total 64 (blue curves at the 16 mark), the success rate is 0.969 for PHP; 0.502 for NPL. This 70% increase in the gain of PHP over NPL may be evidence that exact gradients are better at training the model than the approximate gradients of NPL, although the choice of an optimization method is conflated here with the choice of a model. Task description. The long-hand addition task was also considered by BID32,, and BID6. In this task, our goal is to add two numbers represented in decimal, by starting at the rightmost column (least significant digit) and repeatedly summing each column to write the ing digit and a carry if needed. The state consists of 4 tapes, as in a Turing Machine, corresponding to the first number, the second number, the carries, and the output. The state also includes the positions of 4 read/write heads, one for each tape. Initially, each of the first two tapes contains the K digits of a number to be added, all other cells contain the empty symbol, and the heads point to the least significant digits. The state-dependent observation o t ps t q reveals the value of the digits (or empty symbols) pointed to by the pointers. The four values are provided to the MLPs in one-hot encoding, i.e., the input vector has 11ˆ4 dimensions with exactly one 1-valued entry in each of the four group. PHPs and elementary actions. The top-level PHP add repeatedly calls add1 to add each column of digits. add1 calls write, carry, and lshift in order to compute the sum of the column, write the carry in the next column, and move the pointers to the next column. If the sum for a column is less than 10, then add1 does not call carry. There are two kinds of elementary actions: one which moves a specified pointer in a specified direction (e.g. MOVE CARRY LEFT), and one which writes a specified digit to a specified tape (e.g. WRITE OUT 2). η write, η carry, and η lshift output the probability distribution over possible action and argument combinations as the product of 3 multinomial distributions, each with 2, 4, and 10 possibilities respectively. Experiment setup. Following, we trained our model on execution traces for inputs of each length 1 to 10. We used 16 traces for each input length, for a total of 160 traces. 2 We experimented with providing 1, 2, 3, 5, and 10 strongly supervised traces, with the remainder containing only the elementary actions. For training our model, we performed a search over two hyperparameters:• Weight on loss from strongly supervised traces: When the number of weakly supervised demonstrations overwhelms the number of strongly supervised traces, the model can learn a hierarchy which does not match the supervisor. By appropriately scaling up the loss contribution from the strongly supervised traces, we can ensure that the model learns to follow the hierarchy specified in them.• Use of τ in ψ: The termination condition ψ τ h pb t |o t q contains a dependence on τ, the number of steps that the current procedure h has executed. However, sometimes the underlying definition for ψ does not contain any dependence on τ: ψ 1 h pb|oq " ψ 2 h pb|oq "¨¨¨. In such a case, the MLP for ψ h may learn a spurious dependency on τ, and generalize poorly to values of τ seen during test time. Therefore, we searched over whether to use τ for ψ at each level of the hierarchy. Results. Our are summarized in Table 2. Previous work by learns a model which can generalize perfectly to input lengths of 500 but not 1000. In our experiments with the same Accuracy for input length Strongly supervised / total traces 500 1000NPI BID32 3 160 / 160 <100% <100% NPL 3 10 / 160 100% <100% PHP 3 / 160 100% 100% Table 2: Empirical for the long-hand addition task. All models were trained with 16 traces per input length between 1 and 10, for a total of 160 traces, some of which strongly supervised.sample complexity, EG can train PHPs which generalize to length-1000 inputs with 100% empirical test accuracy. Moreover, we successfully learn models with as few as 3 strongly supervised demonstrations, compared to the 10 used by. However, we found that when the number of strongly supervised demonstrations was smaller than 10, early stopping of the training of the top-level policy was needed to learn a correct model. To obtain our reported , we evaluated different snapshots of the model generated dur reporteding training. In this paper we introduced the Parametrized Hierarchical Procedures (PHP) model for hierarchical representation of neural programs. We proposed an Expectation-Gradient algorithm for training PHPs from a mixture of strongly and weakly supervised demonstrations of an algorithmic behavior, showed how to perform level-wise training of multi-level PHPs, and demonstrated the benefits of our approach on two benchmarks. PHPs alleviate the sample complexity required to train policies with unstructured memory architectures, such as LSTMs, by imposing the structure of a call stack augmented with program counters. This structure may be limiting in that it requires the agent to also rely on observable information that could otherwise be memorized, such as the building specifications in the NanoCraft domain. The benchmarks used so far in the field of neural programming are simple enough and observable enough to be solvable by PHPs, however we note that more complicated and less observable domains may require more expressive memory structures, such as passing arguments to sub-procedures. Future work will explore such structures, as well as new benchmarks to further challenge the community. Our suggest that adding weakly supervised demonstrations to the training set can improve performance at the task, but only when the strongly supervised demonstrations already get decent performance. Weak supervision could attract the optimization process to a different hierarchical structure than intended by the supervisor, and in such cases we found it necessary to limit the number of weakly supervised demonstrations, or weight them less than demonstrations annotated with the intended hierarchy. An open question is whether the attractors strengthened by weak supervision are alternative but usable hierarchical structures, that are as accurate and interpretable as the supervisor's. Future work will explore the quality of solutions obtained by training from only weakly supervised demonstrations. In weak supervision, only the observable trajectory ξ " po 0:T, a 0:T´1 q is available in a demonstration, and the sequence of memory states ζ " m 0:T´1 is latent. This poses a challenge, since the space of possible memory trajectories ζ grows exponentially in the length of the demonstration, which at first seems to prohibit the computation of the log-likelihood gradient ∇ θ log Ppξ|π θ q, needed to maximize the log-likelihood via gradient ascent. Our key insight is that the log-likelihood gradient can be computed precisely and efficiently using an instance of the Expectation-Gradient (EG) method BID33, which we detail below: DISPLAYFORM0 where the first and third equations follow from the identity ∇ θ x " x∇ θ log x. We start by assuming two-level PHPs, so that at any time t the stack contains the root PHP and the PHP it invoked to select the elementary action. The stack also contains the program counters of these two PHPs, however we ignore the root counter to reduce complexity, and bring it back when we discuss multi-level hierarchies in Section 4.2.3 (and below).Let us denote by η τ h pa t |o t q and ψ τ h pb t |o t q, respectively, the stochastic operation and termination statements of procedure h P H Y tKu, where K is the root PHP. Let ph t, τ t q be the top stack frame when action a t is selected. Then the full likelihood Ppζ, ξ|θq of the policy given an annotated demonstration is Ppζ, ξ|θq 9 η K ph 0 |o 0 qδ τ0"0 DISPLAYFORM1 where from the right-hand side we omitted the constant causal dynamics factor DISPLAYFORM2 Ppo t |o 0:t´1, a 0:t´1 q, and with Pph t, τ t |h t´1, τ t´1, o t q " # ψ τt´1 ht´1 p1|o t qη K ph t |o t q if τ t " 0 ψ τt´1 ht´1 p0|o t qδ ht"ht´1 if τ t " τ t´1`1.This formulation of the likelihood has the extremely useful property that ∇ θ log Ppζ, ξ|θq decomposes into a sum of gradients. To find the expected gradient, as in, we do not need to represent the entire posterior distribution Ppζ|ξ, θq, which would be intractable. Instead, we only need the marginal posteriors that correspond to the various terms, namely v t ph, τ q " Pph t "h, τ t "τ |ξ, θq w t ph, τ q " Pph t "h, τ t "τ, τ t`1 "τ`1|ξ, θq. With these, the EG trick gives us the gradient of the observable demonstration DISPLAYFORM3 t ph, 0q∇ θ log η K ph|o t q t ÿ τ "0˜v t ph, τ q∇ θ log η τ h pa t |o t q w t ph, τ q∇ θ log ψ τ h p0|o t`1 qq pv t ph, τ q´w t ph, τ qq∇ θ log ψ τ h p1|o t`1 q¸¸.To allow the G-step, we take an E-step that calculates the marginal posteriors v and w with a forward-backward pass. We first compute the likelihood of a trajectory prefix φ t ph, τ q 9 Ppo 0:t, a 0:t, h t "h, τ t "τ q, up to the causal dynamics factor, via the forward recursion given by φ 0 ph, 0q " η K ph|o 0 q, and for 0 ď t ă T´1 φ t`1 ph 1, 0q "˜ÿ hPH,0ďτ ďt φ t ph, τ qη τ h pa t |o t qψ τ h p1|o t`1 q¸η K ph 1 |o t`1 q φ t`1 ph, τ`1q " φ t ph, τ qη τ h pa t |o t qψ τ h p0|o t`1 qq. We similarly compute the likelihood of a trajectory suffix ω t ph, τ q 9 Ppa t:T´1, o t`1:T |o 0:t, h t "h, τ t "τ q, via the backward recursion given by ω T´1 ph, τ q " η τ h pa T´1 |o T´1 qψ τ h p1|o T q, and for 0 ď t ă T´1 ω t ph, τ q " η τ h pa t |o t q˜ψ τ h p1|o t`1 q ÿ h 1 PH η K ph 1 |o t`1 qω t`1 ph 1, 0q`ψ τ h p0|o t`1 qqω t`1 ph, τ`1q¸.For efficiency considerations, note that this forward-backward graph has pt`1qk nodes in layer t, where k " |H|, but only pt`1qkpk`1q edges to the next layer, rather than the naive pt`1qpt`2qk 2.We can calculate our target likelihood using any 0 ď t ă T, by takingPpξ|θq " ÿ hPH,0ďτ ďt Ppξ, h t "h, τ t "τ q 9 ÿ hPH,0ďτ ďt φ t ph, τ qω t ph, τ q, so most efficient is to use t " 0 Ppξ|θq " DISPLAYFORM4 Ppξ, h 0 "h, τ 0 "0q 9 ÿ hPH φ 0 ph, 0qω 0 ph, 0q. Finally, the marginal posteriors are given by v t ph, τ q " 1 Ppξ|θq φ t ph, τ qω t pτ, hq w T´1 ph, τ q " 0, and for 0 ď t ă T´1 w t ph, τ q " 1 Ppξ|θq φ t ph, τ qη τ h pa t |o t qψ τ h p0|o t`1 qqω t`1 ph, τ`1q. As mentioned in Section 4.2.3, level-wise training of multi-level PHPs requires abstraction from lower levels and separation from higher levels. The former is achieved by rewriting weakly supervised demonstrations to show level-i operations as elementary, for the purpose of training the next-higher level i´1.After level i is trained, the level-i PHPs that generated the demonstrations are decoded using the trained parameters. In our current experiments we used latent trajectories sampled from the posterior distribution, given by Ppζ|ξ, θq " v 0 ph 0, τ 0 q T´2 ź t"0 Pph t, τ t, h t`1, τ t`1 |ξ, θq v t ph t, τ t q,where for each step 0 ď t ă T´1Pph t, τ t, h t`1, 0|ξ, θq " 1 Ppξ|θq φ t ph t, τ t qη τt ht pa t |o t qψ τt ht po t`1 qη K ph t`1 |o t`1 qω t`1 ph t`1, 0qPph t, τ t, h t`1, τ t`1 |ξ, θq " δ ht`1"ht w t ph t, τ t q.

represent the tldr for finding the corresponding scientific article. The provided query could be "This paper proposes a new approach to incorporating desired invariance to representations learning, based on the observations that the current state-of-the-art AFL has practical issues." and the positive "An adversarial feature learning (AFL) is a powerful framework to learn representations invariant to a nuisance attribute, which uses an adversarial game between a feature extractor and a categorical attribute classifier. It theoretically sounds in term of it maximize conditional entropy between attribute and representation. However, as shown in this paper, the AFL often causes unstable behavior that slows down the convergence. We propose an {\em attribute perception matching} as an alternative approach, based on the reformulation of conditional entropy maximization as {\em pair-wise distribution matching}. Although the naive approach for realizing the pair-wise distribution matching requires the significantly large number of parameters, the proposed method requires the same number of parameters with AFL but has a better convergence property. Experiments on both toy and real-world dataset prove that our proposed method converges to better invariant representation significantly faster than AFL. How to learn representations invariant to nuisance attribute a is technical challenges raised in domain generalizaton BID1 BID9 BID4 BID8, fair classification, privacy-protection BID2 BID5, and many other area. Assume that we are given a training dataset made of pairs S = (x i, y i, a i) DISPLAYFORM0, where x is an observation, y is a target of x, and a is a corresponding intrinsic attribute of K-way categorical variable A. The goal of invariant representation learning is to obtain an encoder E that reduces information about attribute a while maintaining information about y. An adversarial game between a feature extractor and an attribute classifier, called adversarial feature learning BID11, is a powerful framework for this purpose. The key of AFL is to measure the invariance by leveraging the discriminative power of neural network beyond the pre-defined metric such as l 2 distance or maximum mean discrepancy. That is, if the external network (also referred to as a discriminator) can predict a from z = E(x), AFL regards z to have considerable information about a. Formally, the AFL solves the following optimization problem: DISPLAYFORM1 where q M and q D is the conditional probability that M and D gives a correct estimation of y and a respectively. As BID11 explained, this alternating procedure can be regarded as a way to maximize the conditional entropy H(A|Z) = a∈A,z∈Z −p(a, z) log p(a|z), where A and Z is a support set of a and z. BID11 also showed that the min-max game has an equilibrium, in which E maximize the conditional entropy H(A|Z). It has been show superior performance in fairclassification, privacy-protection, and domain generalization tasks BID3 BID2 BID11 BID5, compared to the predifined metric approaches BID12 BID7 BID8.Despite the theoretical justifications, the above min-max formulation is suspicious for several practical issues. Namely, the gradient from the discriminator vanishes if the discriminator sufficiently trained since E[log q D (a|z=E(x))] is small then. Besides, in mathematical level, it only keeps away representations from the non-desired point where we can predict a label correctly, but not ensure that it approaches the desired invariant point. Please also refer FIG1 for visualization of the instability. Note that, Generative Adversarial Networks community, which utilize similar formulation to generate realistic images, evade similar issues by the incorporating alternative objectives, such as the Wasserstein distance BID0. However, the Wasserstein distance is defined over two distributions and applying to our setting (consisting of multiple distributions) is not trivial. This paper holds the following contributions to the invariant feature learning problem. First, we empirically show that AFL is suffered from practical issues that significantly slow down the convergence. We then reformulate the optimization problem of AFL as pair-wise distribution matching and derive parameter practical realization of pairwise distribution matching while inheriting the merit of AFL that leveraging the discriminative power to measure the invariance. It is worth mentioning that the reformulation enable us to use Wasserstein metric in theory, however, it is still computationally infeasible in practice because a naive way to calculate the Wasserstein distance between all the pair of the distributions requires O(K 2) discriminators, where K = |A|, which raise computational issues both in terms of parameter size and forward/backward time. Finally, we empirically validate the superior performance of our proposed method on both artificial dataset and real-world datasets.2 CONVERGENCE ISSUES OF AFL FIG1 -(a-e) visualize a behavior of AFL optimization on synthesized data. Each figure corresponds to the different timestep of the alternating optimization. The dataset consists of samples from three Gaussian distributions with different means ([sin( DISPLAYFORM2, for i ∈ 1, 2, 3, respectively) and the same variance, assuming that each distribution corresponds to different attributes. In each figure, dots represent the data point, color represents the attribute (domain id), and the contour plot represents the discriminator's decision boundary. A float value on the top of figures is the negative log-likelihood (NLL) of the dataset measured by the discriminator D (the multi-layer perceptron with 100 hidden units followed by a ReLU activation). Similarly, a float value in parenthesis on the top of figures is an NLL of a post-hoc classifier D eval that have the same architecture as D. To be more specific, we first train the discriminator 100 times with 128 batch size and train D and E iteratively with stochastic gradient descent with learning rate=0.1. FIG1 -(f,g) shows the gradient vector fields of different time steps for a = blue, where the arrow represents the direction of the gradient, and the norm represents its magnitude. For simplicity, we only show the vector fields of a = blue, but the observations are quite similar for the other a. The figure reveals two practical issues in AFL optimization. The distribution alignment is initially quite slow (compare with the behavior of the proposed method shown in Figure 2). This is because the gradient is small when the discriminator correctly distinguishes a a. AFL behavior is unstable. The distributions somewhat align after 40 steps (given 0.683 NLL with the post-hoc classifier), but it is catastrophically failed five steps later because the discriminator did not capture the true conditional entropy (implied by the mostly similar counterplot of D) and therefore gave a false gradient as shown in (f) and (g). The intuitive reason for this phenomenon is that AFLs loss essentially tries to pull a distribution apart from the non-desired point, i.e., the point where we can correctly predict the label. The problem of AFL is that it only keeps away a distribution from the non-desired point, but not ensure it approaches the desired invariant point. After several steps, D starts to follow the change of the distribution (as shown in FIG1 . The instability of the AFL also appears in the significant gap between the NLL of the D and D eval . Note that the second issue may be alleviated if D has a sufficiently large capacity and is trained many times at each iteration. However, this is not a realistic assumption since it is fair to say that real datasets are more complicated than this toy situations, making it more challenging to find the supremum. The next question we must answer is how to maximize conditional entropy while avoiding the issues mentioned above. Assume that A is a categorical random variable drawn from a uniform discrete distribution, and denote that support set of a and z as A and Z; then, the following theorem holds: Theorem 1. The maximum conditional entropy H(A|Z) is − log 1 K, and H(A|Z) is maximized if and only if p(z|a=i) = p(z|a=j) for all i = j ∈ 1, · · ·, K and z ∈ Z.The proof is followed in the appendix. One implication of the theorem is, the empirical measurements of conditional entropy should be boudned. It suggests another problematic point of AFL objective since it is not lower bounded. Another implication is that this theorem permits us to rethink the problem of the conditional entropy maximization as a problem of aligning all pairs of distributions p(z|a = i) and p(z|a = j). This reformulation is significant since we can now use any conventional measurement of two distributions. However, the naive approach requires to match all pairs of the distributions (= K C 2), which would be cumbersome to compute. We, therefore, propose a computationally efficient way to realize pairwise distribution matching, moving attribute perception matching. As with AFL, APM is based on the alternating training of attribute classifier D and feature extractor E, but APM deceives D differently. Formally, we propose the following APM matching objective: DISPLAYFORM0 where k D is some distance function defined over either a hidden representation of the discriminator D or output probability q D (a|E(x)) itself and C aj is the moving average of the centroid for attribute a j in the attribute perception. Although there are many valid choices for k D (., .), including the simple l 2 norm, we primarily use the well-known Kullback-Leibler (KL) divergence. We initialized C 0 aj via computing the centroids using all training data points. The key of the proposal is that it only requires the same number of parameters as AFL, but it ensures that representations of an attribute approach to the representations of the other attributes. Also, it inherits the merit of AFL that leveraging the discriminative power to measure the invariance. Figure 2-(a-c) shows the behavior of the APM under the exact same experimental settings shown in FIG1. The proposed method maximizes the conditional entropy significantly faster than AFL: 1.061 after only five iterations, but AFL gives 0.683 after 40 iterations. Note that the value match the theoretical maximum value of the conditional entropy (log Another merit of the proposed method is that we can enforce semantic alignment with a simple modification. Individually, semantic alignment can be carried out by merely computing the centroids for each (attribute, label) tuple and aligning the perceptions of {x, y, a} between only centroids of the same label y = y but different attributes a = a. Although this modification does not help to maximize the conditional entropy, it prevents performance degradation of predicting y. Since most applications of invariant feature learning require to keep a information about y we use this modification for all the later-described experiments. We use three domain generalization tasks and two user-anonymization tasks. MNISTR BID4 and PACS BID6 ) is a well known image-based datasets designed for the purpose of domain generalization. We also test the methods of noise-robust speech recognition scenarios using Google Speech Command Dataset (Speech). Regarding user-anonymization, we use two user-anonymization tasks on the data of wearables, OppG and USC BID5. The neural networks require to learn representations that help activity classification and at the same time, prevent to access the information about users (userID). As baselines, we use A CNN trained on the aggregation of data from all source domains. AFL BID11, which was explained in Section 3.1. FORMULA4 RevGrad (BID3) is a slightly modified version of AFL, which uses the gradient reversal layer to train all the networks. CrossGrad BID10 ) is regarded as a state-of-the-art method in domain generalization tasks. Activation Matching (AM), which trains the encoder with the regularization of the l 2 distance on a feature space. APM is our proposal. For all datasets and methods, we used RMSprop for each optimization. For all datasets except PACS, we set the learning rate to 0.001 and the batch size to 128. For PACS, we set the learning rate to 5e − 5 and the batch size to 64. For a fair comparison, hyperparameters were tuned on a validation set for each baseline. For the adversarial-training-based method, we optimized weighting parameter λ from {0.001, 0.01, 0.1, 1.0}, except for MNISTR, for which it was optimized from {0.01, 0.1, 1.0, 2.0}. The value of α for CrossGrad was selected from {0.1, 0.25, 0.5, 0.75, 0.9}. We set the decay rate γ to 0.7 for all experiments. In all the experiments, we selected the data of one or several domains for the test set and used the data of a disjoint domain as the training/validation data. Specifically, we split the data of the disjoint domain into groupings of 80% and 20%. We denote the test domain by a suffix (e.g., MNISTR-M0). We measured the label classification accuracy and the level of invariance. We measured the level of invariance by training a post-hoc classifier f eva following previous studies BID11; BID5. FIG2 compares the performance the invariance of representations. For each method, we used the largest weighting parameter λ on the condition that label classification accuracy did not significantly decrease. The show that the proposed method stably achieves better invariant representations except for speech dataset. For example, APM achieved 20% of the A-Acc in MNISTR, which is nearly perfect invariance as there are five domains in the validation data, while RevGrad and AFL achieved 30% at best. On speech dataset, the performance of AFL and APM is mostly similar. These confirm that the proposed method stably achieves more invariant representation compared with AFL. TAB0 summarizes the methods' classification performance on three different datasets: MNISTR, Speech, and PACS. The leftmost column of each table represents the test domain. We report the mean accuracy. We can make the following observations. APM demonstrates the best or comparable performance on all three datasets, including CrossGrad, which is regarded as the state-of-the-art method in domain generalization tasks. RevGrad and AFL often fail to improve performance even when compared with a standard CNN. These suggest that the previous adversarial-training-based method suffered from the lack of semantic alignment when applied to domain generalization. FORMULA4 The Wilcoxon rank sum test shows that APM is statistically better than CNN, RevGrad, and AFL with p < 0.01, and than CrossGrad with p < 0.05. This paper proposes a new approach to incorporating desired invariance to representations learning, based on the observations that the current state-of-the-art AFL has practical issues. Empirical on both toy and real-world datasets support the stable performance of the proposed method to learn invariant features and superior performance on domain generalization tasks. A PROOF OF THE THEOREM 1Proof. Using the Lagrange multiplier method, the derivative of DISPLAYFORM0 is equal to zero for the maximum entropy H(A|Z). Solving the simultaneous equations, we can say p(a=1|z) = p(a=2|z) = · · · = p(a=K|z) = 1 K for all z ∈ Z when the conditional entropy is maximized, and based on the definition, the conditional entropy become − log DISPLAYFORM1 holds ∀i = j ∈ A and z ∈ Z."

We introduce the PHP model for hierarchical representation of neural programs, and an algorithm for learning PHPs from a mixture of strong and weak supervision.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.