Question: Represent the article for finding a single-sentence TLDR

Question: We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure? This problem setting occurs frequently in real-world reinforcement learning scenarios such as a vehicle adapting to drive in a new city, or a robotic drone adapting a policy trained only in simulation. While learning without catastrophic failures is exceptionally difficult, prior experience can allow us to learn models that make this much easier. These models might not directly transfer to new settings, but can enable cautious adaptation that is substantially safer than na\"{i}ve adaptation as well as learning from scratch. Building on this intuition, we propose risk-averse domain adaptation (RADA). RADA works in two steps: it first trains probabilistic model-based RL agents in a population of source domains to gain experience and capture epistemic uncertainty about the environment dynamics. Then, when dropped into a new environment, it employs a pessimistic exploration policy, selecting actions that have the best worst-case performance as forecasted by the probabilistic model. We show that this simple maximin policy accelerates domain adaptation in a safety-critical driving environment with varying vehicle sizes. We compare our approach against other approaches for adapting to new environments, including meta-reinforcement learning. An experienced human driving a rental car for the first time is initially very aware of her lack of familiarity with the car. How sensitive is it to acceleration and braking? How does it respond to steering? How wide is the vehicle and what is its turning radius? She drives mindfully, at low speeds, braking far ahead of desired stops, and making wide turns, all the while observing the car's responses and adapting to it. Within minutes, once she is familiar with the car, she begins to drive more fluently and efficiently. Humans draw upon their prior experiences to perform this kind of safe, quick adaptation to unfamiliar situations all the time, such as when playing with a new tennis racquet, or walking on a new slippery surface. Such problems are critical to address in autonomous systems: such as when a self-driving car must learn to drive in a new country, or when a planetary rover might have to learn to explore a harsh new environment. Missteps in real-world situations can cause real damage to robots and their environments. An important bottleneck in applying today's standard machine learning approaches to control in these real-world situations is that they are trained without any notion of safe behavior under uncertainty. Recent works have attempted to address this by proposing methods for safe exploration during reinforcement learning -in other words, how might an agent avoid risky actions during training time? This still requires that the robot acquire its notions of uncertainty and risks at the same time as it is learning to perform tasks in the new environment, which is difficult and precarious. Could we instead rely on transferring notions of uncertainty and risk acquired from prior experience in other related domains, such as in simulated environments, where safety may not be as much of a concern? In other words, could we make the safe learning problem easier through knowledge transfer, relaxing the problem to safe adaptation, like the human driver? How might the planetary rover draw on its experience in many varied terrains on Earth to perform meaningfully cautious actions during learning on the unknown terrain of a new planet? Motivated by these questions, we propose a model-based reinforcement learning approach called risk averse domain adaptation (RADA). RADA works by first pretraining a probabilistic dynamics model on a population of training domains with varied, unknown dynamics. Through this experience over many environments, the model learns to estimate the epistemic uncertainty (model uncertainty) of unknown environment dynamics, thus permitting estimation of a distribution of outcomes for any action executed by the agent. When introduced into a new target environment, RADA uses this estimated distribution of outcomes to select cautious actions that obey the following maximin notion of risk-aversion: among various candidate action sequences, it executes those that lead to the best worst-case performance, as predicted by the model. Much like the human driver in the example above, all the information collected during this cautious phase of exploration is fed back into the model to finetune it to the new domain, leading to increasingly confident predictions. Over time, RADA steadily estimates lower risks and approaches optimality in the target environment. As we demonstrate in experiments in a driving domain, the experience acquired during RADA's pretraining phase enables fast yet safe adaptation within only a handful of episodes. Cautious or risk-averse learning has close connections to learning robust control policies, as well as the uncertainty estimation derived from Bayesian reinforcement learning . Rather than conventionally maximizing a reward function, accounting for risk usually involves allocating more attention to'worst-case' outcomes in an environment. Such outcomes become particularly important in out-of-domain settings, where purely optimizing in the training domain does not guarantee good performance in the test domain, the problem setting that we consider in this work. Safety in reinforcement learning. Incorporating safety requires properly managing risks and reducing the impact of unforeseen negative outcomes. Risk management is extensively studied in quantitative finance. In portfolio optimization, a commonly used quantity that measures the expected return considering the worse α-% of cases is Conditional Value at Risk (CVaR) . With probability α, the reward is greater than the CVaR measure. CVaR is formulated as E[R|R ≤ υ α]. Rather than optimizing the expected reward, risk averse policies optimize the lower α-quartile of the distribution of rewards. While meta-learning approaches like RL 2 can potentially learn safety by adapting across learning episodes, we found this was not possible in the environments we tested. To address safety more expicitly, the reinforcement learning community is adopting measures like CVaR as quantities that can be optimized (; ; ;) to create policies which are robust to shifts from source to target domains. propose learning robust policies by sampling from the α-quartile of an ensemble of models. While the model ensemble is trained on a given source distribution, the policy is only trained on the lower α-quartile rewards from trajectories sampled from this ensemble. This leads to policies which are more conservative and therefore more robust to shift when deployed from source to target domains. Epistemic uncertainty in reinforcement learning. While learning a robust model is beneficial for transferring to different domains, model-based reinforcement learning offers an additional unsupervised learning signal that can be exploited at test time. In particular, prior work has shown that a model can be quickly adapted during test time by meta-learning for fast adapting parameters during training (a;). These fast adapting parameters offers greater flexibility in adapting to unforeseen circumstances which an agent may encounter at test time. Nagabandi et al. (2018a) show that real robots can quickly adapt to broken or miscalibrations during evaluation through this fast adaptation acquired through meta-learning. Such approaches are complementary to our approach, as they provide a means to explicitly train for fast adaptation to disturbances in the environment, while they do not account for any notion of safety. propose using the uncertainty of a model to regularize policy learning. The policy is encouraged to create trajectories which are distributionally close to trajectories observed in training data. After training, the observations of the model and actions of the policy generate state trajectories which near the domain it has been trained on. In other words, the policy has a preference to keep the trajectories within its training domain. In our work, our policy is encouraged to behave cautiously in unfamiliar environments rather than remain in familiar ones. train a collision prediction model to favor'safe' (low velocity) collisions. Using uncertainty estimates, this collision model will initially behave cautiously in a new environment. Similar to our method, the model becomes less conservative as it adapts to the new environment and lowers its uncertainty. Domain randomization. Domain randomization (; ;) attempts to train policies that are able to transfer to some target environment by training an agent in deliberately randomized simulated environments to allow learning a policy that is invariant to the randomized parameters, and thus performs robustly in the target environment. RADA also pretrains on a set of environments with varied dynamics, but different from these prior works, we operate in a safety-critical setting, focusing on safe adaptation to the target environment -to accomplish this, we follow an explicitly cautious action policy at adaptation time, different from the policy used in the pretraining environments. Before discussing RADA, we first lay out some preliminaries. We build upon PETS , a recently proposed approach for model-based reinforcement learning. We describe the main features of the PETS framework below: Probabilistic dynamics model. PETS trains an ensemble of probabilistic dynamics models within its environment. Each model in the ensemble is a probabilistic neural network that outputs a distribution over the next state s conditioned on the current state s and action a. The data for training these models comes from trajectories executed by following the same scheme for action selection that will be eventually used at test time. Action selection. This action selection scheme is sampling-based model-predictive control (MPC): an evolutionary search method finds action sequences with the highest predicted reward. The reward of an action sequence in turn is computed by propagating action outcomes autoregressively through the learned probabilistic models. Reward computation. Specifically, starting from a state s 0, for each sampled action sequence A = [a 1, ..., a H], where H is the planning horizon, the dynamics model first predicts a distribution over s 1 after executing a 0. A particle propagation method samples Monte Carlo samples from this distribution. For each sample, the dynamics model then predicts the state distribution for s 2, conditioned on executing a 1, and the process repeats. This recursive particle propagation in a large number N of particles {ŝ after H steps. These N particles represent samples from the distribution of possible states after executing A. Each such particle i ∈ [1, N] is now assigned a predicted reward r i, which is a function of its full state trajectory starting from s 0. Finally, the mean of those predicted rewards is considered the score of the action sequence: We call this the action score. Then, the winning action sequence A * = arg max A R(A) with the highest action score is selected, the first action in that sequence is executed, and the whole process repeats starting from the ing new state s 1. Now we present our approach, Risk-Averse Domain Adaptation (RADA). As motivated in Sec 1, RADA approaches safe learning as an adaptation problem, where an agent may draw upon its experience in a variety of environments to guide cautious behavior in a new safety-critical target environment while minimizing the risk of catastrophic failure. Consider a set of environments, each defined by the value of an unknown domain ID variable z, which controls the dynamics in each domain. RADA assumes a setting where we train on some of these domains, and must then transfer to new domains with unknown, potentially unseen, and even out-of-distribution values of z. As a running example, consider learning to drive cars, where each car is represented by its own value of the domain ID variable z. This domain ID might include a potentially large set of unknown and hard-to-measure properties, which together determine the way that the car drives. We propose a solution, RADA, that builds upon the PETS framework (Sec 3). PETS has been demonstrated to work well across a variety of environments. Further, compared to alternative approaches, this framework has two natural advantages for our risk-averse domain adaptation setting, both of which are critical to our approach: (i) the probabilistic models of PETS can be adapted to capture the "epistemic uncertainty" about the dynamics of a new domain, and (ii) model-based RL agents contain dynamics models that can be trained in the absence of any rewards or supervision, providing a route for adaptation to a new environment. We now discuss how RADA builds upon PETS. RADA first builds a probabilistic ensemble of dynamics models on the training domains that capture the epistemic uncertainty in dynamics due to unknown z. We call this the pretraining phase. When an agent with this pretrained model encounters a new domain, we use pessimistic predictions from the model to drive cautious exploration to finetune the model to the new domain, leading to safe and fast adaptation. We call this the adaptation/finetuning phase. Algorithm 1 provides pseudocode for RADA, and the rest of this section explains RADA in detail. While the PETS probabilistic ensemble is trained to represent uncertainty within a single environment in , we would like our model to capture the uncertainty associated with being dropped into a new environment, with unknown domain ID z. To do this, we propose a "pretraining" phase, where a single PETS ensemble is trained across all the training environments, with varying, unknown values of z. Specifically, at the beginning of each training episode, we randomly sample one of the training z's from a uniform distribution. Since z determines the environment dynamics and is unavailable to the learned dynamics model, the ensemble has incentive to learn to model this as epistemic uncertainty during the pretraining phase. See the pretraining procedure in Algorithm 1. After this pretraining, how might the uncertainty captured in the ensemble inform cautious exploration during adaptation in the target environment? To do this, we adapt the PETS action selection and reward computation scheme using a maximin notion of cautious behavior, in line with notions of risk used in prior work across disciplines . Specifically, we replace the action score of equation 1 with a newly defined "generalized action score" R γ (A), in which the "caution parameter" γ ∈ controls the degree of caution exercised in evaluating action sequences in the new environment. R γ (A) is defined as: where υ k (r) denotes the value of the k th percentile of predicted rewards {r j} N j=1 among the N particles after particle propagation. Unpacking this definition, R γ measures the mean score of the bottom 100 − γ percentile of the predicted outcomes from the PETS model. When γ = 50, for instance, it measures the mean of the worst 50 percentile of predicted rewards. This is a pessimistic evaluation of the prospects of the action sequence A -it only pays attention to the worst performing particles in the distribution. At caution γ = 0, R γ exactly matches the definition of R in equation 1: it measures the mean predicted reward of all the particles produced after particle propagation through the model. In our experiments, we heuristically set γ = 50. Now, we define a "γ-cautious action policy" as one that selects actions based on the generalized action score R γ. In other words, A * γ = arg max A R γ (A). We propose to deploy such γ-cautious action policies at adaptation time. The intuition is straightforward: in an unknown environment, the agent deliberately performs cautious actions to perform safe adaptation. Even though it eventually seeks to achieve high mean performance in the target environment, it does not select actions that achieve the highest expected reward under its model. Instead, it chooses to be conservative, not trusting the model's expectations fully. Algorithm 1 RADA 1: procedure PRETRAINING 2: Initialize the probabilistic ensemble dynamics model f Initialize data D using a random controller in a random training domain for one trial. for domain ID z ∼ training domains do Train the probabilistic ensemble dynamics model f on D for t = 0 to task horizon do for evolutionary search stage=1,2,... do for sampled action sequence A do Run state propagation to produce N particles 10: end for Refine search to find A * = arg max R(A) end for Execute first action of A * Record outcome in D for target domain adaptation episode=1,2,... do for t = 0 to task horizon do for evolutionary search stage=1,2,... do for sampled action sequence A do Run state propagation 25: Evaluate A with generalized score Rγ (A) end for Refine search to find A * = arg max Rγ (A) end for Execute first action of A * Record outcome in D Finetune the probabilistic ensemble model f on D end for end for As it gathers experience in the target environment using the γ-cautious policy, RADA also improves its dynamics model over time to finetune it to the new environment. Since dynamics models do not need any manually specified reward function during training, the ensemble model can continue to be trained in the same way as during the pretraining phase. We propose to stabilize adaptation by keeping the model close to the original model. To do this, we maintain a replay buffer of data from the pretraining episodes, conducted outside the target domain. During adaptation, we compute all model updates on this full dataset, including the replay buffers. We use probabilistic neural network ensembles for the dynamics model , and training proceeds through stochastic gradient descent. As the model improves over time, the distribution of predicted outcomes becomes more and more narrow over time. For a deterministic environment, the model eventually converges to deterministic predictions, so that R γ is the same for all γ. In other words, once the model is welltrained, the γ-cautious action policy is identical to the standard action policy. The adaptation procedure in Algorithm 1 sums up cautious action selection and model finetuning. We now evaluate various ablations of RADA in a driving environment, evaluating the importance of three components of our technique for generalization to unseen out-of-domain environments: (i) pretraining on multiple domains (i.e. 'domain randomization'), (ii) replaying pretraining to stabilize finetuning in the target environment, (iii) γ-cautious action policy (γ > 0) at adaptation time -we heuristically set γ = 50 for our experiments. RADA is the version that includes all three techniques. For all methods, we use a PETS ensemble of 5 fully connected models, with each one having 4 layers. We hypothesize that being cautious would not only allow RADA to adapt quicker by avoiding risk of catastrophic failure, but also that as uncertainty about the environment is resolved during adaptation, a cautious policy will become similar to the optimal policy, leading to higher final reward. Our first baseline is RADA without cautious adaptation in the test environment: RADA\caution. Next we separately ablate out multi-domain pretraining and pretraining replay from this approach to create two more baselines: RADA\{caution,DR} and RADA\{caution,replay}. Through these baselines, we systematically compare the contributions of various components of RADA. We also compare RADA against PETS trained directly in the target environment, train-on-target. As an external baseline, we compare against robust adversarial reinforcement learning (RARL). RARL works by training a model-free RL agent jointly with an adversary that perturbs the actions emitted by the agent. We train the adversary to perturb the motor torques in Duckietown. Finally, we also implemented three meta-learning approaches as baselines: GrBal (a), RL 2 , and MOLe (b). However, in our experiments, all three metalearning approaches experienced training failures in our environment, failing to consistently reach the goal even during the pretraining phase. Consequently, we do not consider these methods in the more detailed reported in this section. Car driving environment. Our driving environment is based on Duckietown , a physically accurate driving environment designed for sim-to-real transfer. The task, illustrated in Fig 1, is to make a right turn around a corner to reach a fixed goal tile. Each tile is fixed to a size of 0.585. We modify the environment such that when the car attempts to take an action that would drive it off of the road tiles, it is held in place. If the car clips the corner, it gets stuck unless the agent has learned to reverse out from the corner and try the turn again. The task rewards faster, more efficient turns, so that there is incentive to go near the corner. At the same time, since there is a big price to pay for it, a cautious agent must avoid hitting the corner at all costs. The agent can observe its current x, y coordinates, current velocity, and steering angle. At each time step, it is given a reward equal to negative Manhattan distance from the goal, with a completion bonus of 100 if it successfully reaches the goal tile. The agent has direct control of the steering angle and velocity of the car at each time step. See figure 1. Each domain is specified by a one-dimensional domain ID, the width of the car, which is unknown to the agent. During pretraining, the car width ranges from 0.050 to 0.099, and is sampled uniformly from this range before each training episode. Having driven these cars of varying widths, we evaluate each method's ability to adapt to driving new cars. We test adaptation to one in-distribution domain: width 0.075, and five out-of-distribution domains: 0.1, 0.125, 0.15, 0.175, and 0.20. We vary the car width because of its direct influence on the optimal trajectory of the car: wider cars must make wider turns to avoid catastrophe. Performance Metrics. We measure the return (sum of rewards over a full episode) and the number of collisions underwent in the target environment. For each method, we report the "average maximum reward" over adaptation time t, which is the average over 10 random seeds of the maximum over t adaptation episodes of the reward obtained in the target environment. Finally, to measure the safety of the adaptation process, we also report the cumulative number of collisions suffered by each method during adaptation, which more directly measures the extent to which different methods avoid catastrophic failures and perform safe adaptation. Results. For all RADA variants, we perform pretraining for 32 episodes: 2 initial rollouts with a random policy and 30 on-policy episodes. RADA\{caution,DR} is pretrained for the same 32 episodes but on a single training domain -the one with car width 0.1, which is the closest to the out-of-domain target car widths (recall the training car widths are 0.050-0.099). RARL trains model-free policies, which typically require more pretraining episodes: we use the default settings and train it for 1000 episodes in the pretraining environments. Fig 2 shows the average maximum reward after 10 adaptation episodes and the average total number of collisions for each method, as a function of the car width. All methods perform worse farther away from the training car widths, but RADA maintains its performance up to car width 0.15, and deteriorates more gracefully, maintaining good performance even up to car width 0.2, over two times the largest car width observed at pretraining time. Not only does RADA achieve the highest rewards, but it also achieves them while being the most safe -it suffers the least number of collisions during adaptation across all these environments. Comparing RADA ablations, cautious action selection during adaptation proves critical to performance, and RADA\caution does much worse than RADA throughout. Domain randomization and pretraining replay have relatively smaller impacts on the performance after ten adaptation episodes, but we will show that they impact training speed and stability. Training directly on the target domain (train-on-target), aside from naturally ing in more collisions, does not in the best performance even in terms of rewards. We believe that this is because pretraining on the training car widths has an additional advantage: it sets up a curriculum for training that makes it easier for the agent to solve the exploration problem of learning how to make the right turn. Finally, RARL policies are relatively robust to changes in car width, but perform uniformly worse than RADA: they yield lower reward and experience more collisions at all car widths. Adaptation speed. We now plot the over adaptation time in each target environment for both the average maximum reward and the running total boundary collisions to show adaptation speed in the target environments. Fig 3 shows the average maximum reward over adaptation time for various methods, and at various target car widths. We evaluate at one in-domain car width (0.075) and at five out-of-domain car widths (0.1, 0.125, 0.15, 0.175, and 0.2). Across all six, RADA yields the highest rewards at all times except for car width 0.1. Fig 4 shows similar plots for the average cumulative total boundary collisions over adaptation time. Once again, RADA outperforms all approaches at most domains, with the least overall collisions at all domains except for car width 0.1. Further, as seen in these cases, RADA\{caution,DR} adapts more slowly than other approaches, demonstrating the value of multi-domain pretraining. Further RADA\{caution,replay} leads to very unstable training, reflected here in the fact that maximum reward over adaptation time does not monotonically increase, unlike the other approaches. Epistemic uncertainty. RADA relies on the fact that a probabilistic dynamics model trained on a population of source domains with varying unknown values of the domain ID z, learns to capture Figure 5: How well does the pretrained dynamics model capture the epistemic uncertainty due to unknown car width? These overhead views present the trajectories predicted by the dynamics model (thin curves in desaturated colors) for a given starting state of the car and a specific action sequence. Overlaid on top of these are ground truth trajectories of the car (thick curves) corresponding to varying car widths (legend in the rightmost panel). See text for details. the epistemic uncertainty associated with z. Fig 5 visualizes how well the dynamics model learns to capture the variation in car behavior due to the unknown car width in our Duckietown setting. For a car in a given state, we evaluate a specific action sequence, and plot the car trajectories predicted by the dynamics model on an overhead view of the environment. These are shown in thin, desaturated curves in the panels in Fig 5. As the panels show, these trajectories originate at the car and spread out as they get farther away, representing the higher uncertainty after more actions have been performed. On top of these trajectories predicted by the model, we overlay thicker curves corresponding to ground truth trajectories of cars of various widths, executing the same action sequence. In each panel of Fig 5, these ground truth trajectories all lie largely within the support of the predicted trajectory distribution, indicating that the model captures the epistemic uncertainty due to varying car widths. A particularly illustrative case is in the left-most panel, where the aggressive action trajectory takes the car very close to the corner of the road. This leads to erratic behavior that causes the car to swerve into the side of the road after clipping the corner in some cases, or proceed successfully towards the goal near the center of the road in other cases. The distribution produced by the model's predicted trajectories are similarly bimodal, correctly capturing these two kinds of behavior. Appendix B Fig 7 shows how these predictions evolve over the course of RADA pretraining. Overall, these visualizations demonstrate that the dynamics model successfully learns to/ represent the epistemic uncertainty due to unknown car width. We have proposed RADA, a new approach to model-based reinforcement learning for safe, quick adaptation of RL agents in new environments with unknown dynamics. RADA relies on two key ideas: transferring knowledge from training in a variety of training environments, and using a maximin notion of risk-aversion during action selection in the target environment. We show in a physically accurate driving environment that RADA performs fast, safe adaptation to learn to drive cars around corners, even when they are up to two times larger than any cars it has driven at pretraining time. In the main paper, we perform experiments where we sample the car width dynamically at the beginning of each pretraining episode, from the full pretraining range (width 0.05 to 0.099), so that there are effectively an infinite number of pretraining environments available to sample from. A more realistic situation might involve having a fixed, small number of pretraining environments, among which we select one at random for each pretraining episode. We now report the of preliminary experiments to investigate the feasibility of RADA in this setting. For these experiments, we sample a small number (2, 5, or 10) of car widths from the original pretraining range and limit pretraining to those environments only. Overall, these indicate that RADA is feasible when the number of pretraining environments is finite and small. In Figure 5, we plotted the trajectory predictions by the fully pretrained dynamics model for a fixed starting state and action sequence. Here, we show how the dynamics model's trajectory predictions improve over pretraining time. To do this, we replot the leftmost panel in Fig 5, but this time using dynamics models from various stages of pretraining. Fig 7 shows the . At first, the predictions are compeletely random and the predicted trajectories occupy the entire state space. They become more accurate throughout the course of training, approaching the trajectories plotted in the leftmost Figure 7: How do the dynamics model's predicted trajectories evolve over pretraining time? These plots demonstrate the predicted trajectories for a fixed action sequence at different stages of RADA pretraining, starting at the first episode of training after the initial random rollouts. image of Fig 5. As the model trains in the pretraining environments, it gradually learns to model the epistemic uncertainty associated with the unknown car width.

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.

Represent the TLDR for finding the corresponding scientific article.

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.


Question: Represent the scientific article to find a short TLDR

Question: We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval. Humans are capable of learning visual concepts by jointly understanding vision and language BID12 BID8 BID15. Consider the example shown in Figure 1 -I. Imagine that someone with no prior knowledge of colors is presented with the images of the red and green cubes, paired with the questions and answers. They can easily identify the difference in objects' visual appearance (in this case, color), and align it to the corresponding words in the questions and answers (Red and Green). Other object attributes (e.g., shape) can be learned in a similar fashion. Starting from there, humans are able to inductively learn the correspondence between visual concepts and word semantics (e.g., spatial relations and referential expressions, Figure 1 -II), and unravel compositional logic from complex questions assisted by the learned visual concepts (Figure 1 -III, also see BID0).Motivated by this, we propose the neuro-symbolic concept learner (NS-CL), which jointly learns visual perception, words, and semantic language parsing from images and question-answer pairs. NS-CL has three modules: a neural-based perception module that extracts object-level representations from the scene, a visually-grounded semantic parser for translating questions into executable programs, and a symbolic program executor that reads out the perceptual representation of objects, classifies their attributes/relations, and executes the program to obtain an answer. Figure 1: Humans learn visual concepts, words, and semantic parsing jointly and incrementally. I. Learning visual concepts (red vs. green) starts from looking at simple scenes, reading simple questions, and reasoning over contrastive examples BID12. II. Afterwards, we can interpret referential expressions based on the learned object-based concepts, and learn relational concepts (e.g., on the right of, the same material as). III Finally, we can interpret complex questions from visual cues by exploiting the compositional structure. NS-CL learns from natural supervision (i.e., images and QA pairs), requiring no annotations on images or semantic programs for sentences. Instead, analogical to human concept learning, it learns via curriculum learning. NS-CL starts by learning representations/concepts of individual objects from short questions (e.g., What's the color of the cylinder?) on simple scenes (≤3 objects). By doing so, it learns object-based concepts such as colors and shapes. NS-CL then learns relational concepts by leveraging these object-based concepts to interpret object referrals (e.g., Is there a box right of a cylinder?). The model iteratively adapts to more complex scenes and highly compositional questions. NS-CL's modularized design enables interpretable, robust, and accurate visual reasoning: it achieves state-of-the-art performance on the CLEVR dataset (a). More importantly, it naturally learns disentangled visual and language concepts, enabling combinatorial generalization w.r.t. both visual scenes and semantic programs. In particular, we demonstrate four forms of generalization. First, NS-CL generalizes to scenes with more objects and longer semantic programs than those in the training set. Second, it generalizes to new visual attribute compositions, as demonstrated on the CLEVR-CoGenT (a) dataset. Third, it enables fast adaptation to novel visual concepts, such as learning a new color. Finally, the learned visual concepts transfer to new tasks, such as image-caption retrieval, without any extra fine-tuning. Our model is related to research on joint learning of vision and natural language. In particular, there are many papers that learn visual concepts from descriptive languages, such as image-captioning or visually-grounded question-answer pairs (; ; ; ; BID14, dense language descriptions for scenes , video-captioning BID10 and video-text alignment .Visual question answering (VQA) stands out as it requires understanding both visual content and language. The state-of-the-art approaches usually use neural attentions (; BID6 ;). Beyond question answering, Johnson et al. (2017a) proposed the CLEVR (VQA) dataset to diagnose reasoning models. CLEVR contains synthetic visual scenes and questions generated from latent programs. Table 1 compares our model with state-of-the-art visual reasoning models BID2; ) along four directions: visual features, semantics, inference, and the requirement of extra labels. explored an interpretable, object-based visual representation for visual reasoning. It performs well, but requires fully-annotated scenes during training. Our model also adopts an object-based visual representation, but the representation is learned only based on natural supervision (questions and answers). BID1 Figure 2: We propose to use neural symbolic reasoning as a bridge to jointly learn visual concepts, words, and semantic parsing of sentences.networks and answers the questions by question-conditioned attention over the object features. In contrast, NS-CL parses question inputs into programs and executes them on object features to get the answer. This makes the reasoning process interpretable and supports combinatorial generalization over quantities (e.g., counting objects). Our model also learns general visual concepts and their association with symbolic representations of language. These learned concepts can then be explicitly interpreted and deployed in other vision-language applications such as image caption retrieval. There are two types of approaches in semantic sentence parsing for visual reasoning: implicit programs as conditioned neural operations (e.g., conditioned convolution and dual attention) and explicit programs as sequences of symbolic tokens BID2 b; ). As a representative, BID2 build modular and structured neural architectures based on programs for answering questions. Explicit programs gain better interpretability, but usually require extra supervision such as groundtruth program annotations for training. This restricts their application. We propose to use visual grounding as distant supervision to parse questions in natural languages into explicit programs, with zero program annotations. Given the semantic parsing of questions into programs, proposed a purely symbolic executor for the inference of the answer in the logic space. Compared with theirs, we propose a quasi-symbolic executor for VQA.Our work is also related to learning interpretable and disentangled representations for visual scenes using neural networks. proposed convolutional inverse graphics networks for learning and inferring pose of faces, while learned disentangled representation of pose of chairs from images. proposed the neural scene de-rendering framework as an inverse process of any rendering process.; learned disentangled representations using deep generative models. In contrast, we propose an alternative representation learning approach through joint reasoning with language. We present our neuro-symbolic concept learner, which uses a symbolic reasoning process to bridge the learning of visual concepts, words, and semantic parsing of sentences without explicit annotations We treat attributes such as Shape and Color as neural operators. The operators map object representations into a visual-semantic space. We use similarity-based metric to classify objects.for any of them. We first use a visual perception module to construct an object-based representation for a scene, and run a semantic parsing module to translate a question into an executable program. We then apply a quasi-symbolic program executor to infer the answer based on the scene representation. We use paired images, questions, and answers to jointly train the visual and language modules. Shown in Figure 2, given an input image, the visual perception module detects objects in the scene and extracts a deep, latent representation for each of them. The semantic parsing module translates an input question in natural language into an executable program given a domain specific language (DSL). The generated programs have a hierarchical structure of symbolic, functional modules, each fulfilling a specific operation over the scene representation. The explicit program semantics enjoys compositionality, interpretability, and generalizability. The program executor executes the program upon the derived scene representation and answers the question. Our program executor works in a symbolic and deterministic manner. This feature ensures a transparent execution trace of the program. Our program executor has a fully differentiable design w.r.t. the visual representations and the concept representations, which supports gradient-based optimization during training. Visual perception. Shown in Figure 2, given the input image, we use a pretrained Mask R-CNN to generate object proposals for all objects. The bounding box for each single object paired with the original image is then sent to a ResNet-34 to extract the region-based (by RoI Align) and image-based features respectively. We concatenate them to represent each object. Here, the inclusion of the representation of the full scene adds the contextual information, which is essential for the inference of relative attributes such as size or spatial position. Concept quantization. Visual reasoning requires determining an object's attributes (e.g., its color or shape). We assume each visual attribute (e.g., shape) contains a set of visual concept (e.g., Cube). In NS-CL, visual attributes are implemented as neural operators, mapping the object representation into an attribute-specific embedding space. FIG1 shows an inference an object's shape. Visual concepts that belong to the shape attribute, including Cube, Sphere and Cylinder, are represented as vectors in the shape embedding space. These concept vectors are also learned along the process. We measure the cosine distances ·, · between these vectors to determine the shape of the object. Specifically, we compute the probability that an object o i is a cube by σ ShapeOf(o i), v Cube − γ τ, where ShapeOf(·) denotes the neural operator, v Cube the concept embedding of Cube and σ the Sigmoid function. γ and τ are scalar constants for scaling and shifting the values of similarities. We classify relational concepts (e.g., Left) between a pair of objects similarly, except that we concatenate the visual representations for both objects to form the representation of their relation. DSL and semantic parsing. The semantic parsing module translates a natural language question into an executable program with a hierarchy of primitive operations, represented in a domain-specific language (DSL) designed for VQA. The DSL covers a set of fundamental operations for visual reasoning, such as filtering out objects with certain concepts or query the attribute of an object. The operations share the same input and output interface, and thus can be compositionally combined to form programs of any complexity. We include a complete specification of the DSL used by our framework in the Appendix A. Our semantic parser generates the hierarchies of latent programs in a sequence to tree manner BID11. We use a bidirectional GRU BID7 to encode an input question, which outputs a fixed-length embedding of the question. A decoder based on GRU cells is applied to the embedding, and recovers the hierarchy of operations as the latent program. Some operations takes concepts their parameters, such as Filter(Red) and Query(Shape). These concepts are chosen from all concepts appeared in the input question. Figure 4 (B) shows an example, while more details can be found in Appendix B.Quasi-symbolic program execution. Given the latent program recovered from the question in natural language, a symbolic program executor executes the program and derives the answer based on the object-based visual representation. Our program executor is a collection of deterministic functional modules designed to realize all logic operations specified in the DSL. Figure 4 (B) shows an illustrative execution trace of a program. To make the execution differentiable w.r.t. visual representations, we represent the intermediate in a probabilistic manner: a set of objects is represented by a vector, as the attention mask over all objects in the scene. Each element, Mask i ∈ denotes the probability that the i-th object of the scene belongs to the set. For example, shown in Figure 4 (B), the first Filter operation outputs a mask of length 4 (there are in total 4 objects in the scene), with each element representing the probability that the corresponding object is selected out (i.e., the probability that each object is a green cube). The output "mask" on the objects will be fed into the next module (Relate in this case) as input and the execution of programs continues. The last module outputs the final answer to the question. We refer interested readers to Appendix C for the implementation of all operators. Optimization objective. The optimization objective of NS-CL is composed of two parts: concept learning and language understanding. Our goal is to find the optimal parameters Θ v of the visual perception module Perception (including the ResNet-34 for extracting object features, attribute operators. and concept embeddings) and Θ s of the semantic parsing module SemanticParse, to maximize the likelihood of answering the question Q correctly: DISPLAYFORM0 where P denotes the program, A the answer, S the scene, and Executor the quasi-symbolic executor. The expectation is taken over P ∼ SemanticParse(Q; Θ s).Recall the program executor is fully differentiable w.r.t. the visual representation. We compute the gradient w.r.t. DISPLAYFORM1 We use RE-INFORCE to optimize the semantic parser Θ s: DISPLAYFORM2, where the reward r = 1 if the answer is correct and 0 otherwise. We also use off-policy search to reduce the variance of REINFORCE, the detail of which can be found in Appendix D.Curriculum visual concept learning. Motivated by human concept learning as in Figure 1, we employ a curriculum learning approach to help joint optimization. We heuristically split the training samples into four stages (Figure 4 (A)): first, learning object-level visual concepts; second, learning relational questions; third, learning more complex questions with perception modules fixed; fourth, joint fine-tuning of all modules. We found that this is essential to the learning of our neuro-symbolic concept learner. We include more technical details in Appendix E. We demonstrate the following advantages of our NS-CL. First, it learns visual concepts with remarkable accuracy; second, it allows data-efficient visual reasoning on the CLEVR dataset (a); third, it generalizes well to new attributes, visual composition, and language domains. We train NS-CL on 5K images (<10% of CLEVR's 70K training images). We generate 20 questions for each image for the entire curriculum learning process. The Mask R-CNN module is pretrained on 4K generated CLEVR images with bounding box annotations, following. Classification-based concept evaluation. Our model treats attributes as neural operators that map latent object representations into an attribute-specific embedding space (FIG1). We evaluate the concept quantization of objects in the CLEVR validation split. Our model can achieve near perfect classification accuracy (∼99%) for all object properties, suggesting it effectively learns generic concept representations. The for spatial relations is relatively lower, because CLEVR does not have direct queries on the spatial relation between objects. Thus, spatial relation concepts can only be learned indirectly. Count-based concept evaluation. The SOTA methods do not provide interpretable representation on individual objects (a; ;). To evaluate the visual concepts learned by such models, we generate a synthetic question set. The diagnostic question set contains simple questions as the following form: "How many red objects are there?". We evaluate the performance on all concepts appeared in the CLEVR dataset. Table 2 summarizes the compared with strong baselines, including methods based on convolutional features (b) and those based on neural attentions . Our approach outperforms IEP by a significant margin (8%) and attention-based baselines by >2%, suggesting object-based visual representations and symbolic reasoning helps to interpret visual concepts. NS-CL jointly learns visual concepts, words and semantic parsing by watching images and reading paired questions and answers. It can be directly applied to VQA. Table 2: We also evaluate the learned visual concepts using a diagnostic question set containing simple questions such as "How many red objects are there?". NS-CL outperforms both convolutional and attentional baselines. The suggested object-based visual representation and symbolic reasoning approach perceives better interpretation of visual concepts. Table 3: We compare different variants of baselines for a systematic study on visual features and data efficiency. Using only 10% of the training images, our model is able to achieve a comparable with the baselines trained on the full dataset. See the text for details. TAB5 summarizes on the CLEVR validation split. Our model achieves the state-of-theart performance among all baselines using zero program annotations, including MAC and FiLM . Our model achieves comparable performance with the strong baseline TbD-Nets , whose semantic parser is trained using 700K programs in CLEVR (ours need 0). The recent NS-VQA model from Yi et al. FORMULA0 achieves better performance on CLEVR; however, their system requires annotated visual attributes and program traces during training, while our NS-CL needs no extra labels. Here, the visual perception module is pre-trained on ImageNet BID9. Without pretraining, the concept learning accuracies drop by 0.2% on average and the QA accuracy drops by 0.5%. Meanwhile, NS-CL recovers the underlying programs of questions accurately (> 99.9% accuracy). NS-CL can also detect ambiguous or invalid programs and indicate exceptions. Please see Appendix F for more details. NS-CL can also be applied to other visual reasoning testbeds. Please refer to Appendix G.1 for our on the Minecraft dataset .For a systematic study on visual features and data efficiency, we implement two variants of the baseline models: TbD-Object and MAC-Object. Inspired by BID1, instead of the input image, TbD-Object and MAC-Object take a stack of object features as input. TbD-Mask and MAC-Mask integrate the masks of objects by using them to guide the attention over the images. Table 3 summarizes the . Our model outperforms all baselines on data efficiency. This comes from the full disentanglement of visual concept learning and symbolic reasoning: how to execute program instructions based on the learned concepts is programmed. TbD-Object and MAC-Object demonstrate inferior in our experiments. We attribute this to the design of model architectures and have a detailed analysis in Appendix F.3. Although TbD-Mask and MAC-Mask do not perform better than the originals, we find that using masks to guide attentions speeds up the training. Besides achieving a competitive performance on the visual reasoning testbeds, by leveraging both object-based representation and symbolic reasoning, out model learns fully interpretable visual concepts: see Appendix H for qualitative on various datasets. Generalizing to new visual compositions. The CLEVR-CoGenT dataset is designed to evaluate models' ability to generalize to new visual compositions. It has two splits: Split A only contains gray, blue, brown and yellow cubes, but red, green, purple, and cyan cylinders; split B imposes the opposite color constraints on cubes and cylinders. If we directly learn visual concepts on split A, it overfits to classify shapes based on the color, leading to a poor generalization to split B.Our solution is based on the idea of seeing attributes as operators. Specifically, we jointly train the concept embeddings (e.g., Red, Cube, etc.) as well as the semantic parser on split A, keeping pretrained, frozen attribute operators. As we learn distinct representation spaces for different attributes, our model achieves an accuracy of 98.8% on split A and 98.9% on split B. Figure 6: Samples collected from four splits in Section 4.3 for illustration. Models are trained on split A but evaluated on all splits for testing the combinatorial generalization. Generalizing to new visual concepts. We expect the process of concept learning can take place in an incremental manner: having learned 7 different colors, humans can learn the 8-th color incrementally and efficiently. To this end, we build a synthetic split of the CLEVR dataset to replicate the setting of incremental concept learning. Split A contains only images without any purple objects, while split B contains images with at least one purple object. We train all the models on split A first, and finetune them on 100 images from split B. We report the final QA performance on split B's validation set. All models use a pre-trained semantic parser on the full CLEVR dataset. Our model performs a 93.9% accuracy on the QA test in Split B, outperforming the convolutional baseline IEP (b) and the attentional baseline TbD by 4.6% and 6.1% respectively. The acquisition of Color operator brings more efficient learning of new visual concepts. Having learned visual concepts on small-scale scenes (containing only few objects) and simple questions (only single-hop questions), we humans can easily generalize the knowledge to larger-scale scenes and to answer complex questions. To evaluate this, we split the CLEVR dataset into four parts: Split A contains only scenes with less than 6 objects, and questions whose latent programs having a depth less than 5; Split B contains scenes with less than 6 objects, but arbitrary questions; Split C contains arbitrary scenes, but restricts the program depth being less than 5; Split D contains arbitrary scenes and questions. Figure 6 shows some illustrative samples. As VQA baselines are unable to count a set of objects of arbitrary size, for a fair comparison, all programs containing the "count" operation over > 6 objects are removed from the set. ForCaption: There is a big yellow cylinder in front of a gray object.(a) An illustrative pair of image and caption in our synthetic dataset. Table 5: We introduce a new simple DSL for image-caption retrieval to evaluate how well the learned visual concepts transfer. Due to the difference between VQA and caption retrieval, VQA baselines are only able to infer the on a partial set of data. The learned object-based visual concepts can be directly transferred into the new domain for free. methods using explicit program semantics, the semantic parser is pre-trained on the full dataset and fixed. Methods with implicit program semantics learn an entangled representation for perception and reasoning, and cannot trivially generalize to more complex programs. We only use the training data from the Split A and then quantify the generalization ability on other three splits. Shown in Table 5, our NS-CL leads to almost-perfect generalization to larger scenes and more complex questions, outperforming all baselines by at least 4% in QA accuracy. The learned visual concepts can also be used in other domains such as image retrieval. With the visual scenes fixed, the learned visual concepts can be directly transferred into the new domain. We only need to learn the semantic parsing of natural language into the new DSL.We build a synthetic dataset for image retrieval and adopt a DSL from scene graph-based image retrieval . The dataset contains only simple captions: "There is an <object A> <relation> <object B>." (e.g., There is a box right of a cylinder). The semantic parser learns to extract corresponding visual concepts (e.g., box, right, and cylinder) from the sentence. The program can then be executed on the visual representation to determine if the visual scene contains such relational triples. For simplicity, we treat retrieval as classifying whether a relational triple exists in the image. This functionality cannot be directly implemented on the CLEVR VQA program domain, because questions such as "Is there a box right of a cylinder" can be ambiguous if there exist multiple cylinders in the scene. Due to the entanglement of the visual representation with the specific DSL, baselines trained on CLEVR QA can not be directly applied to this task. For a fair comparison with them, we show the in Table 5b on a subset of the generated image-caption pairs where the underlying programs have no ambiguity regarding the reference of object B. A separate semantic parser is trained for the VQA baselines, which translates captions into a CLEVR QA-compatible program (e.g., Exist(Filter(Box, Relate(Right, Filter(Cylinder))). Table 5c compares our NS-CL against typical image-text retrieval baselines on the full image-caption dataset. Without any annotations of the sentence semantics, our model learns to parse the captions into the programs in the new DSL. It outperforms the CNN-LSTM baseline by 30%. We further conduct experiments on MS-COCO images. Results are presented on the VQS dataset BID13. VQS contains a subset of images and questions from the original VQA 1.0 dataset BID3. All questions in the VQS dataset can be visually grounded: each question is associated with multiple image regions, annotated by humans as essential for answering the question. Figure 7 illustrates an execution trace of NS-CL on VQS.We use a syntactic dependency parser to extract programs and concepts from language BID2 ). The object proposals and features are extracted from models pre-trained on the MS-COCO dataset and the ImageNet dataset, respectively. Illustrated in Figure 7, our model FIG2 shows examples of the learned visual concepts, including object categories, attributes, and relations. Experiment setup and implementation details are in Appendix G.2.In this paper, we focus on a neuro-symbolic framework that learns visual concepts about object properties and relations. Indeed, visual question answering requires AI systems to reason about more general concepts such as events or activities . We leave the extension of NS-CL along this direction and its application to general VQA datasets BID3 as future work. We presented a method that jointly learns visual concepts, words, and semantic parsing of sentences from natural supervision. The proposed framework, NS-CL, learns by looking at images and reading paired questions and answers, without any explicit supervision such as class labels for objects. Our model learns visual concepts with remarkable accuracy. Based upon the learned concepts, our model achieves good on question answering, and more importantly, generalizes well to new visual compositions, new visual concepts, and new domain specific languages. The design of NS-CL suggests multiple research directions. First, constructing 3D object-based representations for realistic scenes needs further exploration BID1 BID5. Second, our model assumes a domain-specific language for describing formal semantics. The integration of formal semantics into the processing of complex natural language would be meaningful future work BID4 ). We hope our paper could motivate future research in visual concept learning, language learning, and compositionality. Our framework can also be extended to other domains such as video understanding and robotic manipulation. Here, we would need to discover semantic representations for actions and interactions (e.g., push) beyond static spatial relations. Along this direction, researchers have studied building symbolic representations for skills and learning instruction semantics from interaction in constrained setups. Applying neuro-symbolic learning frameworks for concepts and skills would be meaningful future work toward robotic learning in complex interactive environments. We first introduce the domain-specific language (DSL) designed for the CLEVR VQA dataset (a). Table 6 shows the available operations in the DSL, while TAB9 explains the type system. Scene −→ ObjectSet Return all objects in the scene. Filter (ObjectSet, ObjConcept) −→ ObjectSet Filter out a set of objects having the object-level concept (e.g., red) from the input object set. DISPLAYFORM0 Filter out a set of objects that have the relational concept (e.g., left) with the input object. AERelate (Object, Attribute) −→ ObjectSet (Attribute-Equality Relate) Filter out a set of objects that have the same attribute value (e.g., same color) as the input object. Intersection (ObjectSet, ObjectSet) −→ ObjectSet Return the intersection of two object sets. DISPLAYFORM1 Return the union of two object sets. Query (Object, Attribute) −→ ObjConcept Query the attribute (e.g., color) of the input object. AEQuery (Object, Object, Attribute) −→ Bool (Attribute-Equality Query) Query if two input objects have the same attribute value (e.g., same color). DISPLAYFORM2 Query if the set is empty. DISPLAYFORM3 Query the number of objects in the input set. CLessThan (ObjectSet, ObjectSet) −→ Bool (Counting LessThan) Query if the number of objects in the first input set is less than the one of the second set. CGreaterThan (ObjectSet, ObjectSet) −→ Bool (Counting GreaterThan) Query if the number of objects in the first input set is greater than the one of the second set. CEqual (ObjectSet, ObjectSet) −→ Bool (Counting Equal) Query if the number of objects in the first input set is the same as the one of the second set. Table 6: All operations in the domain-specific language for CLEVR VQA.We note that some function takes Object as its input instead of ObjectSet. These functions require the uniqueness of the referral object. For example, to answer the question " What's the color of the red object?", there should be one and only one red object in the scene. During the program execution, the input object set will be implicitly cast to the single object (if the set is non-empty and there is only one object in the set). Such casting is named Unique in related works (b The type system of the domain-specific language for CLEVR VQA. As shown in Appendix A, a program can be viewed as a hierarchy of operations which take concepts as their parameters. Thus, NS-CL generates the hierarchies of latent programs in a sequence to tree manner BID11 . The semantic parser adopts an encoder-decoder architecture, which contains four neural modules: a bidirectional GRU encoder IEncoder BID7 to encode an input question into a fixed-length embedding, an operation decoder OpDecoder that determines the operation tokens, such as Filter, in the program based on the sentence embedding, a concept decoder ConceptDecoder that selects concepts appeared in the input question as the parameters for certain operations (e.g., Filter takes an object-level concept parameter while Query takes an attribute), and a set of output encoders {OEncoder i} which encode the decoded operations by OpDecoder and output the latent embedding for decoding the next operation. The operation decoder, the concept decoder, and the output encoders work jointly and recursively to generate the hierarchical program layout. Algorithm 1 illustrates the algorithmic outline of the semantic parser. Algorithm 1: The String-to-Tree Semantic Parser. DISPLAYFORM0 The function parse takes two inputs: the current decoding state f and all concepts appeared in the question, as a set {c i}. The parsing procedure begins with encoding the input question by IEncoder as f 0, extracting the concept set {c i} from the input question, and invoking parse(f 0, {c i}).The concept set {c i} is extracted using hand-coded rules. We assume that each concept (including object-level concepts, relational concepts, and attributes) is associated with a single word in the question. For example, the word "red" is associated with the object-level concept Red, while the word "shape" is associated with the attribute Shape. Informally, we call these words concept words. For a given question Q, the corresponding concept set {c i} is composed of all occurrences of the concept words in Q. The set of concept words is known for the CLEVR dataset. For natural language questions, one could run POS tagging to find all concept words BID2 ). We leave the automatic discovery of concept words as a future work BID15. We use the word embedding of the concept words as the representation for the concepts {c i}. Note that, these "concept embeddings" are only for the program parsing. The visual module has separate concept embeddings for aligning object features with concepts in the visual-semantic space. We now delve into the main function parse(f, {c i}): we first decode the root operation op of the hierarchy by OpDecoder(f). If op requires a concept parameter (an object-level concept, a relational concept, or an attribute), ConceptDecoder will be invoked to choose a concept from all concepts {c i}. Assuming op takes two non-concept inputs (e.g., the operation Intersection takes two object sets as its input), there will be two branches for this root node. Thus, two output encoders OEncoder 0 and OEncoder 1 will be applied to transform the current state f into two sub-states f 1 and f 2. parse will be recursively invoked based on f 1 and f 2 to generate the two branches respectively. In the DSL, the number of non-concept inputs for any operation is at most 2.In our implementation, the input encoder IEncoder first maps each word in the question into an embedding space. The word embeddings are composed of two parts: a randomly initialized word embedding of dimension 256 and a positional embedding of dimension 128 . For a concept word, its word embedding only depends on which type it belongs to (i.e. object-level, relational or attribute). Thus, after being trained on a fixed dataset, the semantic parser can parse questions with novel (unseen) concept words. The sequence of word embeddings is then encoded by a two-layer GRU with a hidden dimension of 256 * 2 (bidirectional). The function parse starts from the last hidden state of the GRU, and works recursively to generate the hierarchical program layout. Both OpDecoder and ConceptDecoder are feed-forward networks. ConceptDecoder performs attentions over the representations of all concepts {c i} to select the concepts. Output encoders OEncoder 0 and OEncoder 1 are implemented as GRU cells. Another pre-processing of the sentence is to group consecutive object-level concept words into a group and treat them together as a single concept, inspired by the notion of "noun phrases" in natural languages. The computational intuition behind this grouping is that, the latent programs of CLEVR questions usually contain multiple consecutive Filter tokens. During the program parsing and execution, we aim to fuse all such Filters into a single Filter operation that takes multiple concepts as its parameter. A Running Example As a running example, consider again the question " What is the color of the cube right of the red matte object?". We first process the sentence (by rules) as: "What is the <Attribute 1 (color)> of the <(ObjConcept 1 (cube)> <RelConcept 1 (right)> of the <ObjConcept 2 (red matte object)>?". The expected parsing of this sentence is: DISPLAYFORM1 The semantic parser encode the word embeddings with IEncoder. The last hidden state of the GRU will be used as f 0. The word embeddings of the concept words form the set {c i} = {Attribute 1, ObjConcept 1, RelConcept 1, ObjConcept 2}. The function parse is then invoked recursively to generate the hierarchical program layout. Table 8 illustrates the decoding process step-by-step. In this section, we present the implementation of all operations listed in Table 6. We start from the implementation of Object-typed and ObjectSet-typed variables. Next, we discuss how to classify objects by object-level concepts or relational concept, followed by the implementation details of all operations. Object-typed and ObjectSet-typed variables. We consider a scene with n objects. An Objecttyped variable can be represented as a vector Object of length n, where Object i ∈ and i Object i = 1. Object i can be interpreted as the probability that the i-th object of the scene is being referred to. Similarly, an ObjectSet-typed variable can be represented as a vector ObjectSet of length n, where ObjectSet i ∈. ObjectSet i can be interpreted as the probability that the i-the object is in the set. To cast an ObjectSet-typed variable ObjectSet as an Object-typed variableStep Inputs Outputs Recursive Invocation DISPLAYFORM0 (End of branch.) Table 8: A step-by-step running example of the recursive parsing procedure. The parameter {c i} is omitted for better visualization. Object (i.e., the Unique operation), we compute: Object = softmax(σ −1 (ObjectSet)), where σ −1 (x) = log(x/(1 − x)) is the logit function. Denote o i as the visual representation of the i-th object, OC the set of all object-level concepts, and A the set of all object-level attributes. Each object-level concept oc (e.g., Red) is associated with a vector embedding v oc and a L1-normalized vector b oc of length |A|. b oc represents which attribute does this object-level concept belong to (e.g., the concept Red belongs to the attribute Color). All attributes a ∈ A are implemented as neural operators, denoted as u a (e.g., uColor). To classify the objects as being Red or not, we compute: DISPLAYFORM0 where σ denotes the Sigmoid function, ·, · the cosine distance between two vectors. γ and τ are scalar constants for scaling and shifting the values of similarities. By applying this classifier on all objects we will obtain a vector of length n, denoted as ObjClassify(Red). Similarly, such classification can be done for relational concepts such as Left. This will in an n × n matrix RelClassify(Left), where RelClassify(Left) j,i is the probability that the object i is left of the object j. To classify whether two objects have the same attribute (e.g., have the same Color), we compute: DISPLAYFORM1 We can obtain a matrix AEClassify(Color) by applying this classifier on all pairs of objects, where AEClassifier(Color) j,i is the probability that the object i and j have the same Color. Quasi-symbolic program execution. Finally, Table 9 summarizes the implementation of all operators. In practice, all probabilities are stored in the log space for better numeric stability. To tackle the optimization in a non-smooth program space, we apply an off-policy program search process to facilitate the learning of the semantic parser. Denote P(s) as the set of all valid programs in the CLEVR DSL for the input question s. We want to compute the gradient w.r.t. Θ s, the parameters of the semantic parser: DISPLAYFORM0 i ) DISPLAYFORM1 i | + γc)/(γc · τc) Table 9: All operations in the domain-specific language for CLEVR VQA. γ c = 0.5 and τ c = 0.25 are constants for scaling and shift the probability. During inference, one can quantify all operations as.where P ∼ SemanticParse(s; Θ s). In REINFORCE, we approximate this gradient via Monte Carlo sampling. An alternative solution is to exactly compute the gradient. Note that in the definition of the reward r, only the set of programs Q(s) leading to the correct answer will contribute to the gradient term. With the perception module fixed, the set Q can be efficiently determined by an off-policy exhaustive search of all possible programs P(s). In the third stage of the curriculum learning, we search for the set Q offline based on the quantified of concept classification and compute the exact gradient ∇Θ s. An intuitive explanation of the off-policy search is that, we enumerate all possible programs, execute them on the visual representation, and find the ones leading to the correct answer. We use Q(s) as the "groundtruth" program annotation for the question, to supervise the learning, instead of running the Monte Carlo sampling-based REINFORCE.Spurious program suppression. However, directly using Q(s) as the supervision by computing = p∈Q(S) − log Pr(p) can be problematic, due to the spuriousness or the ambiguity of the programs. This comes from two aspects: 1) intrinsic ambiguity: two programs are different but equivalent. For example P1: AEQuery(Color, Filter(Cube), Filter(Sphere)) and P2: Exist(Filter(Sphere, AERelate(Color, Filter(Cube)))) are equivalent. 2) extrinsic spuriousness: one of the program is incorrect, but also leads to the correct answer in a specific scene. For example, P1: Filter(Red, Relate(Left, Filter(Sphere))) and P2: Filter(Red, Relate(Left, Filter(Cube))) may refer to the same red object in a specific scene. Motivated by the REINFORCE process, to suppress such spurious programs, we use the loss function: DISPLAYFORM2 The corresponding gradient ∇ Θs is, DISPLAYFORM3 The key observation is that, given a sufficiently large set of scenes, a program can be identified as spurious if there exists at least one scene where the program leads to a wrong answer. As the training goes, spurious programs will get less update due to the sampling importance term Pr[p] which weights the likelihood maximization term. During the whole training process, we gradually add more visual concepts and more complex question examples into the model. Summarized in Figure 4 (A), in general, the whole training process is split into 3 stages. First, we only use questions from lesson 1 to let the model learn object-level visual concepts. Second, we train the model to parse simple questions and to learn relational concepts. In this step, we freeze the neural operators and concept embeddings of object-level concepts. Third, the model gets trained on the full question set (lesson 3), learning to understand questions of different complexities and various format. For the first several iterations in this step, we freeze the parameters in the perception modules. In addition, during the training of all stages, we gradually increase the number of objects in the scene: from 3 to 10.We select questions for each lesson in the curriculum learning by their depth of the latent program layout. For eaxmple, the program "Query(Shape, Filter(Red, Scene))" has the depth of 3, while the program "Query(Shape, Filter(Cube, Relate(Left, Filter(Red, Scene))))" has the depth of 5. Since we have fused consecutive Filter operations into a single one, the maximum depth of all programs is 9 on the CLEVR dataset. We now present the detailed split of our curriculum learning lessons:For lesson 1, we use only programs of depth 3. It contains three types of questions: querying an attribute of the object, querying the existence of a certain type of objects, count a certain type of objects, and querying if two objects have the same attribute (e.g., of the same color). These questions are almost about fundamental object-based visual concepts. For each image, we generate 5 questions of lesson 1.For lesson 2, we use programs of depth less than 5, containing a number of questions regarding relations, such as querying the attribute of an object that is left of another object. We found that in the original CLEVR dataset, all Relate operations are followed by a Filter operation. This setup degenerates the performance of the learning of relational concepts such as Left. Thus, we add a new question template into the original template set: Count(Relate( ·, Filter( ·, Scene))) (e.g., "What's the number of objects that are left of the cube?"). For each image, we generate 5 questions of lesson 2.For lesson 3, we use the full CLEVR question set. Curriculum learning is crucial for the learning of our neuro-symbolic concept learner. We found that by removing the curriculum setup w.r.t. the number of object in the scenes, the visual perception module will get stuck at an accuracy that is similar to a random-guess model, even if we only use stage-1 questions. If we remove the curriculum setup w.r.t. the complexity of the programs, the joint training of the visual perception module and the semantic parser can not converge. We conduct ablation studies on the accuracy of semantic parsing, the impacts of the ImageNet pretraining of visual perception modules, the data efficiency of our model, and the usage of object-based representations. F.1 SEMANTIC PARSING ACCURACY.We evaluate how well our model recovers the underlying programs of questions. Due to the intrinsic equivalence of different programs, we evaluate the accuracy of programs by executing them on the ground-truth annotations of objects. Invalid or ambiguous programs are also considered as incorrect. Our semantic parser archives > 99.9% QA accuracy on the validation split. The only extra supervision of the visual perception module comes from the pre-training of the perception modules on ImageNet BID9. To quantify the influence of this pre-training, we conduct ablation experiments where we randomly initialize the perception module following. The classification accuracies of the learned concepts almost remain the same except for Shape. The classification accuracy of Shape drops from 98.7 to 97.5 on the validation set while the overall QA accuracy on the CLEVR dataset drops to 98.2 from 98.9. We speculate that large-scale image recognition dataset can provide prior knowledge of shape. In this section, we study whether and how the number of training samples and feature representations affect the overall performance of various models on the CLEVR dataset. Specifically, we compare the proposed NS-CL against two strong baselines: TbD and MAC .Baselines. For comparison, we implement two variants of the baseline models: TbD-Object and MAC-Object. Inspired by BID1, instead of using a 2D convolutional feature map, TbD-Object and MAC-Object take a stack of object features as inputs, whose shape is k × d obj. k is the number of objects in the scene, and d obj is the feature dimension for a single object. In our experiments, we fix k = 12 as a constant value. If there are fewer than 12 objects in the scene, we add "null" objects whose features are all-zero vectors. We extract object features in the same way as NS-CL. Features are extracted from a pre-trained ResNet-34 network before the last residual block for a feature map with high resolution. For each object, its feature is composed of two parts: region-based (by RoI Align) and image-based features. We concatenate them to represent each object. As discussed, the inclusion of the representation of the full scene is essential for the inference of relative attributes such as size or spatial position on the CLEVR domain. TbD and MAC networks are originally designed to use image-level attention for reasoning. Thus, we implement two more baselines: TbD-Mask and MAC-Mask. Specifically, we replace the original attention module on images with a mask-guided attention. Denotes the union of all object masks as M. Before the model applies the attention on the input image, we multiply the original attention map computed by the model with this mask M. The multiplication silences the attention on pixels that are not part of any objects. Results. Table 3 summarizes the . We found that TbD-Object and MAC-Object approach show inferior compared with the original model. We attribute this to the design of the network architectures. Take the Relate operation (e.g., finds all objects left of a specific object x) as an example. TbD uses a stack of dilated convolutional layers to propagate the attention from object x to others. In TbD-Object, we replace the stack of 2D convolutions by several 1D convolution layers, operating over the k × d obj object features. This ignores the equivalence of objects (the order of objects should not affect the ). In contrast, MAC networks always use the attention mechanism to extract information from the image representation. This operation is invariant to the order of objects, but is not suitable for handling quantities (e.g., counting objects).As for TbD-Mask and MAC-Mask, although the mask-guided attention does not improve the overall performance, we have observed noticeably faster convergence during model training. TbD-Mask and MAC-Mask leverage the prior knowledge of object masks to facilitate the attention. Such prior has also been verified to be effective in the original TbD model: TbD employs an attention regularization during training, which encourages the model to attend to smaller regions. In general, NS-CL is more data-efficient than MAC networks and TbD. Recall that NS-CL answers questions by executing symbolic programs on the learned visual concepts. Only visual concepts (such as Red and Left) and the interpretation of questions (how to translate questions into executable programs) need to be learned from data. In contrast, both TbD and MAC networks need to additionally learn to execute (implicit or explicit) programs such as counting. For the experiments on the full CLEVR training set, we split 3,500 images (5% of the training data) as the hold-out validation set to tune the hyperparameters and select the best model. We then apply this model to the CLEVR validation split and report the testing performance. Our model reaches an accuracy of 99.2% using the CLEVR training set. We also extend the experiments to a new reasoning testbed: Minecraft worlds . The Minecraft reasoning dataset differs from CLEVR in both visual appearance and question types. FIG3 gives an example instance from the dataset. Besides different 3D visual appearance and image contexts, the Minecraft reasoning dataset introduces two new types of reasoning operations. We add them to our domain-specific language:1. FilterMost(ObjectSet, Concept) → ObjectSet: Given a set of objects, finds the "most" one. For example, FilterMost(Closest, set) locates the object in the input set that is cloest to the camera (e.g., what is the direction of the closest animal?) 2. BelongTo(Object, ObjectSet) → Bool: Query if the input object belongs to a set. Results. TAB11 summarizes the and FIG5 shows sample execution traces. We compare our method against the NS-VQA baseline , which uses strong supervision for both scene representation (e.g., object categories and positions) and program traces. In contrast, our method learns both by looking at images and reading question-answering pairs. NS-CL outperforms NS-VQA by 5% in overall accuracy. We attribute the inferior of NS-VQA to its derendering module. Because objects in the Minecraft world usually occlude with each other, the detected object bounding boxes are inevitably noisy. During the training of the derendering module, each detected bounding box is matched with one of the ground-truth bounding boxes and uses its class and pose as supervision. Poorly localized bounding boxes lead to noisy labels and hurt the accuracy of the derendering module. This further influences the overall performance of NS-VQA. We conduct experiments on the VQS dataset BID13. VQS is a subset of the VQA 1.0 dataset BID3 On the right, we show the original question and answer in natural language, as well as the latent program recovered by our parser. To answer this question, models are expected to attend to the man and his pen in the pocket. Setup. All models are trained on the first 63,509 images of the training set, and tested on the test split. For hyper-parameter tuning and model selection, the rest 5,000 images from the training set are used for validation. We use the multiple-choice setup for VQA: the models choose their most confident answer from 18 candidate answers for each question. To obtain the latent programs from natural languages, we use a pre-trained syntactic dependency parser BID2 ) for extracting programs and concepts that need to be learned. A sample question and the program obtained by our parser is shown in FIG4. The concept embeddings are initialized by the bag of words (BoW) over the GloVe word embeddings .Baselines. We compare our model against two representative baselines: MLP and MAC .MLP is a standard baseline for visual-question answering, which treats the multiple-choice task as a ranking problem. For a specific candidate answer, a multi-layer perceptron (MLP) model is used to encode a tuple of the image, the question, and the candidate answer. The MLP outputs a score for each tuple, and the answer to the question is the candidate with the highest score. We encode the image with a ResNet-34 pre-trained on ImageNet and use BoW over the GloVe word embeddings for the question and option encoding. We slightly modify the MAC network for the VQS dataset. For each candidate answer, we concatenate the question and the answer as the input to the model. The MAC model outputs a score from 0 to 1 and the answer to the question is the candidate with the highest score. The image features are extracted from the same ResNet-34 model. Results. TAB9 summarizes the . NS-CL achieves comparable with the MLP baseline and the MAC network designed for visual reasoning. Our model also brings transparent reasoning over natural images and language. Example execution traces generated by NS-CL are shown in FIG1. Besides, the symbolic reasoning process helps us to inspect the model and diagnose the error sources. See the caption for details. Another appealing benefit is that our reasoning model enjoys full interpretability. Figure 11, FIG5, and FIG1 show NS-CL's execution traces on CLEVR, Minecraft, and VQS, respectively. As a side product, our system detects ambiguous and invalid programs and throws out exceptions. As an example (Figure 11), the question "What's the color of the cylinder?" can be ambiguous if there are multiple cylinders or even invalid if there are no cylinders. Figure 14 and FIG6 include qualitative visualizations of the concepts learned from the CLEVR and Minecraft datasets, including object categories, attributes, and relations. We choose samples from the validation or test split of each dataset by generating queries of the corresponding concepts. We set a threshold to filter the returned images and objects. For quantitative evaluations of the learned concepts on the CLEVR dataset, please refer to Table 2 and Table 5 Figure 11: Visualization of the execution trace generated by our Neuro-Symbolic Concept Learner on the CLEVR dataset. Example A and B are successful executions that generate correct answers. In example C, the execution aborts at the first operator. To inspect the reason why the execution engine fails to find the corresponding object, we can read out the visual representation of the object, and locate the error source as the misclassification of the object material. Example D shows how our symbolic execution engine can detect invalid or ambiguous programs during the execution by performing sanity checks. FIG1: Illustrative execution trace generated by our Neuro-Symbolic Concept Learner on the VQS dataset. Execution traces A and B shown in the figure leads to the correct answer to the question. Our model effectively learns visual concepts from data. The symbolic reasoning process brings transparent execution trace and can easily handle quantities (e.g., object counting in Example A). In Example C, although NS-CL answers the question correctly, it locates the wrong object during reasoning: a dish instead of the cake. In Example D, our model misclassifies the sport as frisbee.

represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We design incremental sequence-to-action parsers for text-to-SQL task and achieve SOTA results. We further improve by using non-deterministic oracles to allow multiple correct action sequences.