Question: Represent the scientific article to find a short TLDR

Question: We present a novel approach to train a natural media painting using reinforcement learning. Given a reference image, our formulation is based on stroke-based rendering that imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents. Throughout human history, painting has been an essential element of artistic creation. There are many diverse and complex artistic domains with various styles such as watercolor, oil painting, sketching, and so on. As image processing and computer graphics have advanced, there has been a considerable effort to simulate these styles using non-photorealistic rendering (NPR) techniques .; generate compelling using stroke-based rendering. However, most prior methods in NPR are engineered for a specific application or task, and cannot easily adapt to new styles or medium. Recent developments in machine learning have ed in significant advancements in computer vision and computer graphics, including computer-based painting systems. Many visual generative methods based on generative adversarial networks as;;;; have demonstrated promising . Many of these machine learning methods have also been applied to stroke-based rendering tasks, including modeling the brush , generating brushstroke paintings in an artist's style , reconstructing drawings for specific paintings styles , and constructing stroke-based drawings (Ha & Eck (2017a);;; Jia et al. (2019a) ). In this paper, we focus on a more general and challenging problem of training a natural media painting agent for interactive applications. Given a reference image, our goal is to develop a stroke-based rendering approach that can imitate the human drawing or strokes used in generating the image. A key challenge is to develop a method that can learn from scratch without any supervision. In this regard, we present a technique that can handle all inputs and train an agent to manipulate natural painting media such as charcoal, pencil, watercolor, and so on. We build a model-based natural media environment using deep CNN and train a natural media painting agent using model-based reinforcement learning. In order to introduce controls to the agents for interactive applications, we use a constraint representation along with a different framework for training and use the constrained painting agent. These constraints enable the agent to interact with a human or other agents and generate various styles without retraining the model. The novel contributions of our work include: • A method to train an agent that produces a stream of actions subject to constraint for each action. These constraints can include restricting the start location, stroke width, color, and other stroke parameters. • A method to roll out constrained agents so the user can produce new stylistic effects interactively or automatically, as the agent is painting by modulating the action stream. • By incorporate coarse-to-fine strategy, our painting agents can generate high-resolution stylized images using various constraints and paintbrush configurations. We evaluate our algorithm on different paintbrush configurations and datasets to highlights its benefits over prior reinforcement learning based methods. We also employ differing constraint settings to validate our constrained agents and produce new stylistic effects with a single trained model. In this paper, we focus on the stroke-based rendering problem, which renders the reference image with brush strokes. In contrast to image analogy approaches (; ; ; ; ; ; Li et al. (2017;), stroke-based approaches can generate intermediate painting states for interactive purposes. They can also be deployed in various painting environment to generate different artistic effects. One category of approaches for stroke-based rendering uses a heuristics-based method to guide the agent while painting. Notable examples include the method in , which can produce stylized illustrations, and the method in , which can reproduce a colorful painting in different styles. However, it is difficult to extend these methods to different styles of painting because of the hand-engineered features. Another category of approaches uses machine learning techniques to learn the policy. Compared with predefined policies, these methods use machine learning techniques to learn the painting policy, which enables the agent to generate a more natural . Ha & Eck (2017b) train an RNN to learn the latent space of the sketch data and generate the paintings in this latent space, which requires the paired dataset for training. Other approaches use deep reinforcement learning to learn the policy without supervised data. There have been a few attempts to tackle related problems in this domain. Xie et al. (2012; 2013) propose a series of works to simulate strokes using reinforcement learning and inverse reinforcement learning. These prior approaches learn a policy either from reward functions or expert demonstrations. Comparing with these approaches, we use a more general setup which does not rely on rewards engineering. is based on the Deep Q network. trains the discriminator and the reinforcement learning framework at the same time. However, both methods can work on either a small action space or a small observation space. Jia et al. (2019b; a) use proximal policy optimization with curriculum learning and self-supervision to gradually increase the sampled action space and the frequency of positive rewards. train a differentiable painting environment model, which helps learn the painting policy with gradient-based methods. train a model-based RL using differentiable environment and DDPG . This approach is limited by the OpenCV based renderer and the uncontrollable agent. An overview of our approach is given in Fig. 1 This includes the training phase and extracting the underlying neural network as the painting agent is used for the roll-out process. We highlight all the symbols used in the paper in Table 1. In section 4, we present the details of natural media painting agent with a environment model for the agent to interact. Our approach is based on the training scheme described in and uses Deep Deterministic Policy Gradient (DDPG) to train the modelbased painting agent. For the roll-out algorithm, we apply the painting agent directly to the real environment R r as shown in Figure 1. In section 5, we present our representation of the underlying constraints and the underlying techniques to encode and decode these constraints using reinforcement learning. We use the unconstrained agent to identify the constraints, and encode them as observations to train the constrained painting agent, as shown in Figure 3. In order roll out the We use a actor-critic-based reinforcement learning framework to train the painting agent. For each step, the current states of the canvas and the reference image form the observation for the policy network. Based on the observation, the painting agent predicts an action for the neural renderer R n to execute and updates the canvas accordingly. Roll-out of the Natural Media Painting Agent (right): We extract the actor neural network as the painting agent for the roll-out process. We replace the neural renderer R n with the real renderer R r to get the precise synthetic . In this section, we present our algorithm for training a natural media environment model and training a painting agent to interact with this environment model. The renderer of our painting agent generates the corresponding canvas by the given action, which implements the blending functions and other synthetic programs. Unlike the previous learning-based painting approaches Jia et al. (2019a);; , we use a natural media painting renderer MyPaint (libmypaint contributors) for our experiment. Compared with self-defined painting simulators, it provides rich visual effects and sophisticated blending functions. use the same environment but have not explored the various configurations of the paintbrush. Using a pre-existing environment setup helps our approach concentrate on the learning algorithm to make it easy to generalize. The action of our painting agent consists of the configurations it use to interact with the environment. For the action space, we follow the stroke definition of using a quadratic Bezier curve (QBC) of three points ((x 0, y 0), (x 1, y 1), (x 2, y 2)). Each stroke is a 3-point curve. The pressure which affects the blending function is linear interpolated by the value of the start position (x 0, y 0) and the end position (x 2, y 2). We use one color (R, G, B) for each stroke, and the transparency of the pixels within the stroke is affected by the pressure (p 0, p 1). We use one brush size r for each stroke, and the actual stroke width within the stroke can also be affected by the pressure (p 0, p 1). Formally, we represent the action a t as: In practice, a t ∈ R 12. Each value is normalized to. The action is in a continuous space, which makes it possible to control the agent precisely. The observation of our painting agent consists of information to make decisions. In our setup, the observation is represented by the reference image s *, the current canvas s t at step t, and the constraint c t. Formally, o t = (s *, s t). For the constrained agent described in section 5, o t = (s *, s t, c t). c t is a vector drawn from the sub-space of the action space. To unify the observation representation for implementation, we upsample the constraint vector as a bitmap with size of s t to feed into the CNN of the policy network and the value network. Reward is a metric that enables our painting agent to measure effectiveness of the action. Aside from the reinforcement learning algorithm, the definition of the reward can be seen as a guidance for the policy π. For our problem, the goal of the painting agent is to reproduce the reference image s *. First, we define the loss function L between the current canvas s t at step t and the reference image s *. In practice, we use l 2 loss (Jia et al. (2019a) ) and WGAN loss ) for implementation. Then, we use the difference of the losses of two continuous steps as the reward r t. We normalize r t using Eq.2, such that r t ∈ (−∞, 1]. where L is a loss function defined as l 2 or WGAN. For our reinforcement learning setting, the objective is to maximize the discounted accumulative rewards q t = tmax t=1 r t γ t, where the discount factor γ ∈ For this painting environment, we use the rendering function R to represent the transition function. The action only modifies the current painting state s t with the current action a t as s t+1 = R(a t, s t). Inspired by and , we model the behaviors of the real environment renderer R r using a neural network as R n. We use a 4-layer fully connected neural network followed by 3 convolutional neural networks.) with different paintbrush configurations. The top row is rendered by the real renderer R r and the bottom row is rendered by the neural renderer R n. There are two main benefits of modeling the environment. First, the reinforcement learning algorithm may have millions of steps, and it can be computationally expensive to use the real environment because of sophisticated synthetic algorithms. If we simplify the environment model, it can save training time. Second, modeling the transition function of states using neural network makes the entire framework differentiable, and the loss can be backpropagated through the renderer network R n. In practice, we model R r by building a dataset consisting of paired data (a t, s t, s t+1). To simplify the problem, we always start from a blank canvas s 0 so that the R n only need to predict the stroke image from a single action. At run-time, we synthesize the blending function by making the stroke image transparent and adding it to the current canvas s t. The paired dataset {a (i), s (i) } can be collected by randomly sampling the a i and synthesising the corresponding s i. We treat different paintbrushes of MyPaint (libmypaint contributors) as separate models. It is worth noting that we use the environment model R n for the training process, but use the real environment R r for the roll-out process. After we train the natural media painting agent, we make the agent controllable to handle interactive scenarios and be used for generating stylized images. To achieve this goal, one straightforward way is to constrain the agent with a modified action space. However, this method is difficult to generalize to different constraints. For each different constraint c in the same subspace C in action space A, the policy needs to be retrained. For example, if we train a painting agent and constrain its stroke to width 0.8, we still need to train it again when we need an agent to output a stroke with width 0.9. In the rest of this section, we propose a constrained painting agent that, can follow any constraint c drawn from the subspace of action space A. First, we define the constraint representation, including the definition of the constraint, the encoding function, and the decoding function. Then we introduce the training and roll-out schemes of the constrained agent. The constraint is the vector c in the constraint space C, which is the sub-space of the action space A. While fixing the vector c, the agent has to explore the subtraction of A and C as A by sampling a from A and concatenating with c: We can define C by selecting and combining dimensions in A. For example, we can constrain the color of the stroke by defining C as the color space (R, G, B). We can constrain the start position of the stroke by defining C as (x 0, x 1). We can constrain the stroke width of the stroke by defining C as (r). Moreover, we can use the combination of the constraints to achieve complex effects. As shown in Figure 3, the unconstrained agent takes the reference image and the current canvas (s *, s t) as observation while the constrained agent takes the reference (s *, s t, c t) as an observation, which has an additional constraint c t at step t. c t is a vector drawn from the sub-space of the action space. To unify the observation representation for implementation, we encode the constraint vector c t as a bitmap to feed into the CNN of the policy network and the value network. To encode the constraint c t into observation, we upsample the constraint c t as a matrix c t which has the same size of s * and s t, to stack them and feed into our policy network and our value network. To decode the constraint c t into c t, we implement a downsample module within the policy network. The policy network pi can be seen as separate policy: π c and π a; π c outputs the downsampled constraint c and π a outputs the constrained action a. Then we concatenate a and c to form action a. Formally, we have c t = π c (c t), a t = π a (s *, s t, c t), and a t = a t ⊕ c t. After we introduce the constraint c t as part of the observation of the painting agent, we propose the corresponding training scheme. Because the constraint representation is designed for interactive purposes, we need to use either human experts or another agent to train the constrained agent. As shown in Figure 3, we use an unconstrained agent to generate constraints by cascading them together. For each step t, the unconstrained agent takes the reference image and the current canvas (s *, s t) as observations, and outputs the action a in action space A. Then we identify a subspace in A as constraint space C and transfer a t to c t, followed by upsampling the c t as c t, as defined in section 4. After that, the constrained agent takes the additional constraint c t as an observation (s *, s t, c t) and outputs action a t concatenated by c t and a. Figure 3: Framework for Training (Roll-out) the Constrained Agent We train the constrained agent by cascading an unconstrained agent. First, we identify a subspace of the action space and extract the constraint c from the action a. Then we upsample the constraint c and pass it on as an observation of the constrained agent. We use R n as renderer. For roll-out process (dashed line), we extract the constraint from either the user defined constraint or the action of the previous step. We use real renderer R r to get the best visual quality. For the roll-out of the constrained painting agent, we replace the unconstrained agent from the training scheme in Figure 3 with the constraint extracted from either human command or a painting agent's output for each step t. When the constraint is extracted from a human command, the agent can be ordered to place a paint stroke with a given position (x 0, y 0), color (R, G, B), and stroke width r. When the constraint is extracted from painting agent's output, it can be seen that we split the one agent that explores in entire action space A into two agents that explore separate action spaces A 0 and A 1, where A = A 0 + A 1. For training the natural media painting environment, we treat different paintbrushes as different models. We run 50,000 iterations for each model and record the l 2 loss on the validation dataset as shown in Table 2. The learning curves of the training processes are shown in Figure 4 (left). We train the unconstrained painting agent using the fixed envionment model with various dataset. We use hand-written digits images , character images (, face images (, object images as train painting agents. We run 10,000 episodes for each training task. Because of the difference among the dataset, we use 5 strokes to reproduce hand-written digits images, 20 strokes to reproduce character images, 100 strokes to reproduce face and object images. We measure the l 2 loss between the reference images and reproduced images throughout the training process shown as Figure 4 (middle). The reproduced is shown as Figure 5, and the corresponding l 2 loss is shown as Table 3 (left). Figure 5: We trained our natural media painting agents using MNIST, KanjiVG, CelebA, and ImageNet as reference images (left). We generate (right) using R r for the training and validating process. For the roll-out process, we employ a coarse-to-fine strategy to increase the resolution of the shown as Figure 6. First, we roll out the agent with a low-resolution reference image s * and get the reproductionŝ *. Then we divide s * andŝ * into patches and feed the agent as initial observation. The roll-out using various natural media are shown as Figure 7. We train the constrained painting agents using the learning parameters as unconstrained painting agents. We compute the l 2 distance between the reference images and reproduce for the training process with various constraint configurations. To control variates, we use same neural environment R n (charcoal) and dataset (CelebA) for these experiments. We use the color (R, G, B), the stroke width r, the start position (x 0, y 0) and all of them (x 0, y 0, r, R, G, B) as constraints. The learning curves of these constraints are shown as Figure 4 (right) and the l2 loss is shown as Table 3(right). We demonstrate the constrained painting agents as Figure 2, which uses pencil as paintbrush and incorporates coarse-to-fine strategy by dividing the reference images as 4×4 patches. To constrain the color of strokes, we build a color palette by clustering the colors of the reference image. For each action a, we constrain it from a color randomly selected from the color palette. To constrain the stroke width, we use a constant stroke width for the roll-out process. Figure 6: Coarse-to-fine Roll-out We roll out the trained agent using a coarse-to-fine strategy. We first roll out the model treating the reference image and canvas as one patch. Then we divide the reference image and updated canvas into patches and feed to the agent. In this figure, we roll out 100 strokes in each patch. In this paper, we train natural media painting agents that can generate artistic paintings using various natural media, and collaborate with humans and other agents to get different visual effects. We build a model of natural media environment using deep CNN and train a natural media painting agent using model-based reinforcement learning. To introduce controls to the agents for interactive purposes, we propose constraint representation, a framework for training a constrained painting agent, and various roll-out schemes to apply the agent. We demonstrate our algorithm by applying the trained model using various paintbrushes from MyPaint and constraints set up. The experimental show that our algorithm can reproduce reference images in multiple artistic styles. For future work, we aim to extend the proposed algorithm by building a unified model for differing paintbrush configuration. In addition, we will train a hierarchical agent that uses a constrained agent as the low-level policy. We would like to apply our approach on other reference images and use for interactive painting systems. A APPENDIX Figure 9: Roll-out using Various PaintbrushesWe roll out our natural media painting agents trained with various brushes in MyPaint. To increase the resolutions of the generated images, we incorporate the coarse-to-fine strategy. We use 8 × 8 patches for first row and 4 × 4 for second row. Figure 10: Reproduction of Starry Night using Charcoal We roll out our natural media painting agent trained with charcoal brush in MyPaint to reproduce Van Gogh's starry night. We incorporate the coarse-to-fine strategy by dividing the reference image and canvas into 16 × 16 patches. Figure 11: Reproduction of Starry Night using Watercolor We roll out our natural media painting agent trained with watercolor brush in MyPaint to reproduce Van Gogh's starry night. We incorporate the coarse-to-fine strategy by dividing the reference image and canvas into 16 × 16 patches.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We train a natural media painting agent using environment model. Based on our painting agent, we present a novel approach to train a constrained painting agent that follows the command encoded in the observation.

Represent the short summary for finding the corresponding scientific article (long)!

We use reinforcement learning to train an agent to solve a set of visual arithmetic tasks using provided pre-trained perceptual modules and transformations of internal representations created by those modules.


Question: represent the text to find a very short summary (1-2 sentences max):

Question: Delusional bias is a fundamental source of error in approximate Q-learning. To date, the only techniques that explicitly address delusion require comprehensive search using tabular value estimates. In this paper, we develop efficient methods to mitigate delusional bias by training Q-approximators with labels that are "consistent" with the underlying greedy policy class. We introduce a simple penalization scheme that encourages Q-labels used across training batches to remain (jointly) consistent with the expressible policy class. We also propose a search framework that allows multiple Q-approximators to be generated and tracked, thus mitigating the effect of premature (implicit) policy commitments. Experimental demonstrate that these methods can improve the performance of Q-learning in a variety of Atari games, sometimes dramatically. Q-learning lies at the heart of many of the recent successes of deep reinforcement learning (RL) (;, with recent advancements (e.g., van ; ; ;) helping to make it among the most widely used methods in applied RL. Despite these successes, many properties of Q-learning are poorly understood, and it is challenging to successfully apply deep Q-learning in practice. When combined with function approximation, Q-learning can become unstable (; ; ;). Various modifications have been proposed to improve convergence or approximation error (; 1999; Szepesvári & ; ; ;); but it remains difficult to reliably attain both robustness and scalability. identified a source of error in Q-learning with function approximation known as delusional bias. It arises because Q-learning updates the value of state-action pairs using estimates of (sampled) successor-state values that can be mutually inconsistent given the policy class induced by the approximator. This can in unbounded approximation error, divergence, policy cycling, and other undesirable behavior. To handle delusion, the authors propose a policy-consistent backup operator that maintains multiple Q-value estimates organized into information sets. Each information set has its own backed-up Q-values and corresponding "policy commitments" responsible for inducing these values. Systematic management of these sets ensures that only consistent choices of maximizing actions are used to update Q-values. All potential solutions are tracked to prevent premature convergence on any specific policy commitments. Unfortunately, the proposed algorithms use tabular representations of Q-functions, so while this establishes foundations for delusional bias, the function approximator is used neither for generalization nor to manage the size of the state/action space. Consequently, this approach is not scalable to RL problems of practical size. In this work, we propose CONQUR (CONsistent Q-Update Regression), a general framework for integrating policy-consistent backups with regression-based function approximation for Q-learning and for managing the search through the space of possible regressors (i.e., information sets). With suitable search heuristics, our framework provides a computationally effective means for minimizing the effects of delusional bias in Q-learning, while admitting scaling to practical problems. Our main contributions are as follows. First we define novel augmentations of standard Q-regression to increase the degree of policy consistency across training batches. While testing exact consistency is expensive, we introduce an efficient soft-consistency penalty that promotes consistency of new labels with earlier policy commitments. Second, drawing on the information-set structure of , we define a search space over Q-regressors to allow consideration of multiple sets of policy commitments. Third, we introduce heuristics for guiding the search over regressors, which is critical given the combinatorial nature of information sets. Finally, we provide experimental on the Atari suite demonstrating that CONQUR can offer (sometimes dramatic) improvements over Q-learning. We also show that (easy-to-implement) consistency penalization on its own (i.e., without search) can improve over both standard and double Q-learning. We assume a discounted, infinite horizon Markov decision process (MDP), M = (S, A, P, p 0, R, γ). The state space S can reflect both discrete and continuous features, but we take the action space A to be finite (and practically enumerable). We consider Q-learning with a function approximator Q θ to learn an (approximately) optimal Q-function , drawn from some approximation class parameterized by Θ (e.g., the weights of a neural network). When the approximator is a deep network, we generically refer to the algorithm as DQN, the method at the heart of many recent RL successes (; . For online Q-learning, at a transition s, a, r, s, the Q-update is given by: Batch versions of Q-learning, including DQN, are similar, but fit a regressor repeatedly to batches of training examples . Batch methods are usually more data efficient and stable than online Q-learning. Abstractly, batch Q-learning works through a sequence of (possibly randomized) data batches D 1, · · · D T to produce a sequence of regressors Q θ1,..., Q θ T = Q θ, estimating the Q-function. 1 For each (s, a, r, s) ∈ D k, we use a prior estimator Q θ k−1 to bootstrap the Q-label q = r + γ max a Q θ k−1 (s, a). We then fit Q θ k to this training data using a suitable regression procedure with an appropriate loss function. Once trained, the (implicit) induced policy π θ is the greedy policy w.r.t. Q θ, i.e., π θ (s) = arg max a∈A Q θ (s, a). Let F(Θ), resp. G(Θ), be the corresponding class of expressible Q-functions, resp. greedy policies. Intuitively, delusional bias occurs whenever a backed-up value estimate is derived from action choices that are not (jointly) realizable in G(Θ) . Standard Q-updates back up values for each (s, a) pair by independently choosing maximizing actions at the corresponding next states s. However, such updates may be "inconsistent" under approximation: if no policy in G(Θ) can jointly express all past action choices, backed up values may not be realizable by any expressible policy. show that delusion can manifest itself with several undesirable consequences. Most critically, it can prevent Q-learning from learning the optimal representable policy in G(Θ); it can also cause divergence. To address this, they propose a non-delusional policy consistent Q-learning (PCQL) algorithm that provably eliminates delusion. We refer to the original paper for details, but review the main concepts we need to consider below. The first key concept is that of policy consistency. For any S ⊆ S, an action assignment σ S: S → A associates an action σ(s) with each s ∈ S. We say σ is policy consistent if there is a greedy policy π ∈ G(Θ) s.t. π(s) = σ(s) for all s ∈ S. We sometimes equate a set SA of state-action pairs with an implied assignment π(s) = a for all (s, a) ∈ SA. If SA contains multiple pairs with the same state s, but different actions a, it is a multi-assignment (though we loosely use the term "assignment" in both cases when there is no risk of confusion). In (batch) Q-learning, each successive regressor uses training labels generated by assuming maximizing actions (under the prior regressor) are taken at its successor states. Let σ k reflect the collection of states and corresponding maximizing actions taken to generate labels for regressor Q θ k (assume it is policy consistent). Suppose we train Q θ k by bootstrapping on Q θ k−1 and consider a training sample (s, a, r, s). Q-learning generates label r + γ max a Q θ k−1 (s, a) for input (s, a). Notice, however, that taking action a * = argmax a Q θ k (s, a) at s may not be policy consistent with σ k. Thus Q-learning will estimate a value for (s, a) assuming the execution of a policy that cannot be realized given the limitations of the approximator. The PCQL algorithm prevents this by insisting that any action assignment σ used to generate bootstrapped labels is consistent with earlier assignments. Notice that this means Q-labels will often not be generated using maximizing actions relative to the prior regressor. The second key concept is that of information sets. One will generally not be able to use maximizing actions to generate labels, so tradeoffs can be made when deciding which actions to assign to different states. Indeed, even if it is feasible to assign a maximizing action a to state s early in training, say at batch k, since it may prevent assigning a maximizing a to s later, say batch k +, we may want to consider a different assignment to s to give more flexibility to maximize at other states later. PCQL doesn't try to anticipate the tradeoffs-rather it maintains multiple information sets, each corresponding to a different assignment to the states seen in the training data so far. Each gives rise to a different Q-function estimate, ing in multiple hypotheses. At the end of training, the best hypothesis is the one maximizing expected value w.r.t. an initial state distribution. PCQL provides strong convergence guarantees, but it is a tabular algorithm: the function approximator retricts the policy class, but is not used to generalize Q-values. Furthermore, its theoretical guarantees come at a cost: it uses exact policy consistency tests-tractable for linear approximators, but not practical for large problems; and it maintains all consistent assignments. As a , PCQL cannot be used for large RL problems of the type tackled by DQN. We develop the CONQUR framework to provide a practical approach to reducing delusion in Qlearning, specifically addressing the limitations of PCQL identified above. CONQUR consists of three main components: a practical soft-constraint penalty that promotes policy consistency; a search space to structure the search over multiple regressors (information sets, action assignments); and heuristic search schemes (expansion, scoring) to find good Q-regressors. We assume a set of training data consisting of quadruples (s, a, r, s), divided into (possibly nondisjoint) batches D 1,... D T for training. This perspective is quite general: online RL corresponds to |D i | = 1; off-line batch training (with sufficiently exploratory data) corresponds to a single batch (i.e., T = 1); and online or batch methods with replay are realized when the D i are generated by sampling some data source with replacement. For any data batch D, let χ(D) = {s : (s, a, r, s) ∈ D} denote the collection of successor states of D. An action assignment σ D for D is an assignment (or multi-assignment) from χ(D) to A: this dictates which action σ D (s) is considered "maximum" for the purpose of generating a Q-label for pair (s, a); i.e., (s, a) will be assigned training label r + γQ(s, σ(s)) rather than r + γ max a ∈A Q(s, a). The set of all such assignments is Σ(D) = A χ(D); note that it grows exponentially with |D|. 2 This is simple policy consistency, but with notation that emphasizes the policy class. Let Σ Θ (D) denote the set of all Θ-consistent assignments over D. The union σ 1 ∪ σ 2 of two assignments (over D 1, D 2, resp.) is defined in the usual way. Enforcing strict Θ-consistency as regressors θ 1, θ 2,..., θ T are generated is computationally challenging. Suppose assignments σ 1,..., σ k−1, used to generate labels for D 1,... D k−1, are jointly Θ-consistent (let σ ≤k−1 denote their multi-set union). Maintaining Θ-consistency when generating θ k imposes two requirements. First, one must generate an assignment σ k over D k s.t. σ ≤k−1 ∪ σ k is consistent. Even testing assignment consistency can be problematic: for linear approximators this is a linear feasibility program whose constraint set grows linearly with |D 1 ∪... ∪ D k |. For DNNs, this is a complex, and much more expensive, polynomial program. Second, the regressor θ k should itself be consistent with σ ≤k−1 ∪ σ k. Again, this imposes a significant constraint on the regression optimization: in the linear case, this becomes a constrained least-squares problem (solvable, e.g., as a quadratic program); while with DNNs, it could be solved, say, using a much more complicated projected SGD. However, the sheer number of constraints makes this impractical. Rather than enforcing consistency, we propose a simple, computationally tractable scheme that "encourages" it: a penalty term that can be incorporated into the regression itself. Specifically, we add a penalty function to the usual squared loss to encourage updates of the Q-regressors to be consistent with the underlying information set, i.e., the prior action assignments used to generate its labels. When constructing θ k, let D ≤k = ∪{D j : j ≤ k}, and σ ∈ Σ Θ (D ≤k) be the collective (possibly multi-) assignment used to generate labels for all prior regressors (including θ k itself). The multiset of pairs B = {(s, σ(s))|s ∈ χ(D ≤k)}, is called a consistency buffer. The collective assignment need not be consistent (as we elaborate below), nor does the regressor θ k need to be consistent with σ. Instead, we incorporate the following soft consistency penalty when constructing θ k: where. This penalizes Q-values of actions at state s that are larger than that of action σ(s). We note that σ is Θ-consistent if and only if min θ∈Θ C θ (B) = 0. We incorporate this penalty into our regression loss for batch D k: Here Q θ k is prior estimator on which labels are bootstrapped (other prior regressors may be used). The penalty effectively acts as a "regularizer" on the squared Bellman error, where λ controls the degree of penalization, allowing a tradeoff between Bellman error and consistency with the action assignment used to generate labels. It thus promotes consistency without incurring the expense of testing strict consistency. It is a simple matter to replace the classical Q-learning update with one using a consistency penalty: This scheme is quite general. First, it is agnostic as to how the prior action assignments are made, which can be the standard maximizing action at each stage w.r.t. the prior regressor like in DQN, Double DQN (DDQN), or other variants. It can also be used in conjunction with a search through alternate assignments (see below). Second, the consistency buffer B may be populated in a variety of ways. Including all max-action choices from all past training batches promotes full consistency in an attempt to minimize delusion. However, this may be too constraining since action choices early in training are generally informed by very inaccurate value estimates. Hence, B may be implemented in other ways to focus only on more recent data (e.g., with a sliding recency window, weight decay, or subsampling); and the degree of recency bias may adapt during training (e.g., becoming more inclusive as training proceeds and the Q-function approaches convergence). Reducing the size of B also has various computational benefits. We discuss other practical means of promoting consistency in Sec. 5. The proposed consistency penalty resembles the temporal-consistency loss of , but our aims are very different. Their temporal consistency notion penalizes changes in a next state's Q-estimate over all actions, whereas we discourage inconsistencies in the greedy policy induced by the Q-estimator, regardless of the actual estimated values. Ensuring optimality requires that PCQL track all Θ-consistent assignments. While the set of such assignments is shown to be of polynomial size , it is still impractical to track this set in realistic problems. As such, in CONQUR we recast information set tracking as a search problem and propose several strategies for managing the search process. We begin by defining the search space and discussing its properties. We discuss search procedures in Sec. 3.4. As above, assume training data is divided into batches D 1,... D T and we have some initial Qfunction estimate θ 0 (for bootstrapping D 1 's labels). The regressor θ k for D k can, in principle, be trained with labels generated by any assignment σ ∈ Σ Θ (D k) of actions to its successor states χ(D k), not necessarily maximizing actions w.r.t. θ k−1. Each σ gives rise to a different updated Q-estimator θ k. There are several restrictions we could place on "reasonable" σ-candidates: (ii) σ is jointly Θ-consistent with all σ j, for j < k, used to construct the prior regressors on which, and this inequality is strict for at least one s. Conditions (i) and (ii) are the strict consistency requirements of PCQL. We will, however, relax these below for reasons discussed in Sec. 3.2. Condition (iii) is inappropriate in general, since we may add additional assignments (e.g., to new data) that render all non-dominated assignments inconsistent, requiring that we revert to some dominated assignment. This gives us a generic search space for finding policy-consistent, delusion-free Q-function, as illustrated in Fig k can also be viewed as an information set). We assume the root n 0 is based on an initial regression θ 0, and has an empty action assignment σ 0. Nodes at level k of the tree are defined as follows. For each node n, and its regressor θ i k is trained using the following data set: The entire search space constructed in this fashion to a maximum depth of T. See Appendix B, Algorithm 1 for pseudocode of a simple depth-first recursive specification. The exponential branching factor in this search tree would appear to make complete search intractable; however, since we only allow Θ-consistent "collective" assignments we can bound the size of the tree-it is polynomial in the VC-dimension of the approximator. Theorem 1. The number of nodes in the search tree is no more than VCDim(G) ) where VCDim(·) is the VC-dimension of a set of boolean-valued functions, and G is the set of boolean functions defining all feasible greedy policies under Θ: A linear approximator with a fixed set of d features induces a policy-indicator function class G with VC-dimension d, making the search tree polynomial in the size of the MDP. Similarly, a fixed ReLU DNN architecture with W weights and L layers has VC-dimension of size O(W L log W) again rendering the tree polynomially sized. Even with this bound, navigating the search space exhaustively is generally impractical. Instead, various search methods can be used to explore the space, with the aim of reaching a "high quality" regressor at some leaf of the tree (i.e., trained using all T data sets/batches). We discuss several key considerations in the next subsection. Even with the bound in Theorem 1, traversing the search space exhaustively is generally impractical. Moreover, as discussed above, enforcing consistency when generating the children of a node, and their regressors, may be intractable. Instead, various search methods can be used to explore the space, with the aim of reaching a "high quality" regressor at some (depth T) leaf of the tree. We outline three primary considerations in the search process: child generation, node evaluation or scoring, and the search procedure. Generating children. Given node n i k−1, there are, in principle, exponentially many action assignments, or children, Σ Θ (D k) (though Theorem 1 significantly limits the number of children if we enforce consistency). For this reason, we consider heuristics for generating a small set of children. Three primary factors drive these heuristics. The first factor is a preference for generating high-value assignments. To accurately reflect the intent of (sampled) Bellman backups, we prefer to assign actions to state s ∈ χ(D k) with larger predicted Q-values over actions with lower values, i.e., a preference for a over a if Q θ (s, a). However, since the maximizing assignment may be Θ-inconsistent (in isolation, or jointly with the parent's information set, or with future assignments), candidate children should merely have higher probability of a high-value assignment. The second factor is the need to ensure diversity in the assignments among the set of children. Policy commitments at stage k constrain the possible assignments at subsequent stages. In many search procedures (e.g., beam search), we avoid backtracking, so we want the policy commitments we make at stage k to offer as much flexibility as possible in later stages. The third is the degree to which we enforce consistency. There are several ways to generate such high-value assignments. We focus on just one natural technique: sampling action assignments using a Boltzmann distribution. Specifically, let σ denote the assignment (information set) of some node (parent) at level k − 1 in the tree. We can generate an assignment σ k for D k as follows. Assume some permutation s 1,..., s |D k | of χ(D k). For each s i in turn, we sample a i with probability proportional to e τ Q θ k−1 (s i,ai). This can be done without regard to consistency, in which case we would generally use the consistency penalty when constructing the regressor θ k for this child to "encourage" consistency rather than enforce it. If we want strict consistency, we can use rejection sampling without replacement to ensure a i is consistent with σ j k−1 ∪ σ ≤i−1 (we can also use a subset of σ j k−1 as a less restrictive consistency buffer). 3 The temperature parameter τ controls the degree to which we focus on purely maximizing assignments versus more diverse, random assignments. While stochastic sampling ensures some diversity, this procedure will bias selection of high-value actions to states s ∈ χ(D k) that occur early in the permutation. To ensure sufficient diversity, we use a new random permutation for each child. Scoring children. Once the children of some expanded node are generated (and, optionally, their regressors constructed), we need some way of evaluating the quality of each child as a means of deciding which new nodes are most promising for expansion. Several techniques can be used. We could use the average Q-label (overall, or weighted using some initial state distribution), Bellman error, or loss incurred by the regressor (including the consistency penalty or other regularizer). However, care must be taken when comparing nodes at different depths of the search tree, since deeper nodes have a greater chance to accrue rewards or costs-simple calibration methods can be used. Alternatively, when a simulator is available, rollouts of the induced greedy policy can be used evaluate the quality of a node/regressor. Notice that using rollouts in this fashion incurs considerable computational expense during training relative to more direct scoring based on properties on the node, regressor, or information set. Search Procedure. Given any particular way of generating/sampling and scoring children, a variety of different search procedures can be applied: best-first search, beam search, local search, etc. all fit very naturally within the CONQUR framework. Moreover, hybrid strategies are possible-one we develop below is a variant of beam search in which we generate multiple children only at certain levels of the tree, then do "deep dives" using consistency-penalized Q-regression at the intervening levels. This reduces the size of the search tree considerably and, when managed properly, adds only a constant-factor (proportional to beam size) slowdown to standard Q-learning methods like DQN. We now outline a specific instantiation of the CONQUR framework that can effectively navigate the large search space that arises in practical RL settings. We describe a heuristic, modified beamsearch strategy with backtracking and priority scoring. Pseudocode is provided in Algorithm 2 (see Appendix B); here we simply outline some of the key refinements. Our search process grows the tree in a breadth-first manner, and alternates between two phases. In an expansion phase, parent nodes are expanded, generating one or more child nodes with action assignments sampled from the Boltzmann distribution. For each child, we create target Q-labels, then optimize the child's regressor using consistency-penalized Bellman error (Eq. 2) as our loss. We thus forego strict policy consistency, and instead "encourage" consistency in regression. In a dive phase, each parent generates one child, whose action assignment is given by the usual max-actions selected by the parent node's regressor as in standard Q-learning. No additional diversity is considered in the dive phase, but consistency is promoted using consistency-penalized regression. From the root, the search begins with an expansion phase to create c children-c is the splitting factor. Each child inherits its parent's consistency buffer from which we add the new action assignments that were used to generate that child's Q-labels. To limit the size of the tree, we only track a subset of the children, the frontier nodes, selected using one of several possible scoring functions. We select the top -nodes for expansion, proceed to a dive phase and iterate. It is possible to move beyond this "beam-like" approach and consider backtracking strategies that will return to unexpanded nodes at shallower depths of the tree. We consider this below as well. Other work has considered multiple hypothesis tracking in RL. One particularly direct approach has been to use ensembling, where multiple Q-approximators are updated in parallel (Faußer & ; ;) then combined straightforwardly to reduce instability and variance. An alternative approach has been to consider population-based methods inspired by evolutionary search. For example, combine a novelty-search and quality diversity technique to improve hypothesis diversity and quality in RL. consider augmenting an off-policy RL method with diversified population information from an evolutionary algorithm. Although these techniques do offer some benefit, they do not systematically target an identified weakness of Q-learning, such as delusion. We experiment using the Atari test suite to assess the performance of CONQUR. We first assess the impact of using the consistency penalty in isolation (without search) as a "regularizer" that promotes consistency with both DQN and DDQN. We then test the modified beam search described in Appendix B to assess the full power of CONQUR. We first study the effects of introducing the soft-policy consistency in isolation, augmenting both DQN and DDQN with the consistency penalty term. We train our models using an open-source implementation of both DQN and DDQN (with the same hyperparameters). We call these modified algorithms DQN(λ) and DDQN(λ), respectively, where λ is the penalty coefficient defined in Eq. 2. Note that λ = 0 recovers the original methods. This is a lightweight modification that can be applied readily to any regression-based Q-learning method, and serves to demonstrate the effectiveness of soft-policy consistency penalty. Since we don't consider search (i.e., don't track multiple hypotheses), we maintain a small consistency buffer using only the current data batch by sampling from the replay buffer-this prevents getting "trapped" by premature policy constraints. As the action assignment is the maximizing action of some network, σ(s) can be computed easily for each batch. This in a simple algorithmic extension that adds only an additional penalty term to the original TD-loss. We train and evaluate DQN(λ) and DDQN(λ) for λ = {0.25, 0.5, 1, 1.5, 2} on 19 Atari games. In training, λ is initialized to 0 and slowly annealed to the desired value to avoid premature commitment to poor action assignments. Without annealing, the model tends fit to poorly informed action assignments during early phases of training, and thus fails to learn a good model. The best λ is generally different across games, depending on the nature of the game and the extent of delusional bias. Though a single λ = 0.5 works well across all games tested, Fig. 2 illustrates the effect of increasing λ on two games. In Gravitar, increasing λ generally in better performance for both DQN and DDQN, whereas in SpaceInvaders, λ = 0.5 gives improvement over both baselines, but performance starts to degrade for λ = 2. We compare the performance of the algorithms using each λ value separately, as well as using the best λ for each game. Under the best λ, DQN(λ) and DDQN(λ) outperform their "potentially delusional" counterparts on all except 3 and 2 games, respectively. In 9 of these games, each of DQN(λ) and DDQN(λ) beats both baselines. With a constant λ = 0.5, each algorithm still beats their respective baseline in 11 games. These suggest that consistency penalization (independent of the general CONQUR model) can improve the performance of DQN and DDQN by addressing the delusional bias that is critical to learning a good policy. Moreover, we see that consistency penalization seems to have a different effect on learned Q-values than double Q-learning, which addresses maximization bias. Indeed, consistency penalization, when applied to DQN, can achieve gains that are greater than DDQN (in 15 games). Third, in 9 games DDQN(λ) provides additional performance gains over DQN(λ). A detailed description of the experiments and further can be found in Appendix C. Table 1: Results of CONQUR with 8 (split 2) nodes on 9 games using the proposed scoring function compared to evaluation using rollouts. We test the full CONQUR framework using the modified beam search discussed above. Rather than training a full Q-network, for effective testing of its core principles, we leverage pre-trained networks from the Dopamine package. 4. These networks have the same architecture as in and are trained on 200M frames with sticky actions using DQN. We use CONQUR to retrain only the last (fully connected) layer (implicitly freezing the other layers), which can be viewed as a linear Q-approximator over the features learned by the CNN. We run CONQUR using only 4M addtional frames to train our Q-regressors. 5 We consider splitting factors c of 2 and 4; impose a limit on the frontier size of 8 or 16; and an expansion factor of 2 or 4. The dive phase is always of length 9 (i.e., 9 batches of data), giving an expansion phase every 10 iterations. Regressors are trained using the loss in Eq. 2 and the consistency buffer comprises all prior action assignments. (See Appendix D for details, hyperparameter choices and more .) We run CONQUR with λ = {1, 10, 100, 1000} and select the best performing policy. We initially test two scoring approaches, policy evaluation using rollouts and scoring using the loss function (Bellman error with consistency penalty). Results comparing the two on a small selection of games are shown in Table 1. While rollouts, not surprisingly, tend to give rise to better-performing policies, consistent-Bellman scoring is competitive. Since the latter much less computationally intense, and does not require sampling the environment, we use it throughout our remaining experiments. We compare CONQUR with the value of the pre-trained DQN. We also evaluate a "multi-DQN" baseline that applies multiple versions of DQN independently, warm-starting from the same pretrained DQN. It uses the same number of frontier nodes as CONQUR, and is otherwise trained identically as CONQUR but with direct Bellman error (no consistency penalty). This gives DQN the same advantage of multiple-hypothesis tracking as CONQUR but without policy consistency. We test on 59 games, comparing CONQUR with frontier size 16 and expansion factor 4 and splitting factor 4 with backtracking (as described in the Appendix D) ed in significant improvements to the pre-trained DQN, with an average score improvement of 125% (excluding games with non-positive pre-trained score). The only games without improvement are Montezuma's Revenge, Tennis, PrivateEye and BankHeist. This demonstrates that, even when simply retraining the last layer of a highly tuned DQN network, removing delusional bias has the potential to offer strong improvements in policy performance. It is able exploit the reduced parameterization to obtain these gains with only 4M frames of training data. Roughly, a half-dozen games have outsized score improvements, including Solaris (11 times greater value), Tutankham (6.5 times) and WizardOfWor (5 times). Compared to the stronger multi-DQN baseline (with 16 nodes), CONQUR wins by at least a 10% margin in 20 games, while 22 games see improvements of 1-10% and 8 games show little effect (plus/minus 1%) and 7 games show a decline of greater than 1% (most are 1-6% with the exception of Centipede at -12% and IceHockey at -86%). Results are similar when comparing CONQUR and multi-DQN each with 8 nodes: 9 games exhibit 10%+ improvement, 21 games show 1-8% improvement, 12 games perform comparably and 7 games do worse under CONQUR. A table of complete appears in Appendix D.3, Table 4, and training curves (all games, all λ) in Fig. 11. Increasing the number of nodes from 8 to 16 generally leads to better performance for CONQUR, with 38 games achieving strictly higher scores with 16 nodes: 16 games with 10%+ improvement, 5 games tied and the remaining 16 games performing worse (only a few with a 5%+ decline). Fig. 3 shows the (smoothed) effect of increasing the number of nodes for a fixed λ = 10. The y-axis represents the rollout value of the best frontier node (i.e., the greedy policy of its Q-regressor) as a function of the training iteration. For both Alien and Solaris, the multi-DQN (baseline) training curve is similar with both 8 and 16 nodes, but CONQUR improves Alien from 3k to 4.3k while Solaris improves from 2.2k to 3.5k. Fig. 4 and Fig. 5 (smoothed, best frontier node) shows node policy values and training curves, respectively, for Solaris. When considering nodes ranked by their policy value, comparing nodes of equal rank generated by CONQUR and by multi-DQN (baseline), we see that CONQUR nodes dominate their multi-DQN counterparts: the three highest-ranked nodes achieve a score improvement of 18%, 13% and 15%, respectively, while the remaining nodes achieve improvements of roughly 11-12%. Fig. 6 (smoothed, best frontier node) shows the effects of varying λ. In Alien, increasing λ from 1 to 10 improves performance, but it starts to decline for higher values of 100 and 1000. This is similar to patterns observed in 4.1 and represents a trade-off between emphasizing consistency and not over-committing to action assignments. In Atlantis, stronger penalization tends to degrade performance. In fact, the stronger the penalization, the worse the performance. We have introduced CONQUR, a framework for mitigating delusional bias in value-based RL that relaxes some of the strict assumptions of exact delusion-free algorithms to ensure scalability. Its two main components are (a) a tree-search procedure used to create and maintain diverse, promising Q-regressors (and corresponding information sets); and (b) a consistency penalty that encourages "maximizing" actions to be consistent with the FA class. CONQUR embodies elements of both value-based and policy-based RL: it can be viewed as using partial policy constraints to bias the value estimator or as a means of using candidate value functions to bias the search through policy space. Empirically, we find that CONQUR can improve the quality of existing approximators by removing delusional bias. Moreover, the consistency penalty applied on its own, directly in DQN or DDQN, itself can improve the quality of the induced policies. Given the generality of the CONQUR framework, there remain numerous interesting directions for future research. Other methods for nudging regressors to be policy-consistent include exact consistency (constrained regression), other regularization schemes that bias the regressor to fall within the information set, etc. Given its flexibility, more extensive exploration of search strategies (e.g., best first), child-generation strategies, and node scoring schemes should be examined within CONQUR. Our (full) experiments should also be extended beyond those that warm-start from a DQN model, as should testing CONQUR in other domains. Other connections and generalizations are of interest as well. We believe our methods can be extended to both continuous actions and soft max-action policies. We suspect that there is a connection between maintaining multiple "hypotheses" (i.e., Q-regressors) and notions in distributional RL, which maintains distributions over action values. We describe an example, taken directly from , to show concretely how delusional bias causes problems for Q-learning with function approximation. The MDP in Fig. 7 illustrates the phenomenon: use a linear approximator over a specific set of features in this MDP to show that: (a) No π ∈ G(Θ) can express the optimal (unconstrained) policy (which requires taking a 2 at each state); (b) The optimal feasible policy in G(Θ) takes a 1 at s 1 and a 2 at s 4 (achieving a value of 0.5). (c) Online Q-learning (Eq. 1) with data generated using an ε-greedy behavior policy must converge to a fixed point (under a range of rewards and discounts) corresponding to a "compromise" admissible policy which takes a 1 at both s 1 and s 4 (value of 0.3). Algorithm 1 CONQUR SEARCH (Generic, depth-first) Training set S ← {} 4: end for end if 14: end for Q-learning fails to find a reasonable fixed-point because of delusion. Consider the backups at (s 2, a 2) and (s 3, a 2). Supposeθ assigns a "high" value to (s 3, a 2), so that Qθ(s 3, a 2) > Qθ(s 3, a 1) as required by π θ *. They show that any suchθ also accords a "high" value to (s 2, a 2). But Qθ(s 2, a 2) > Qθ(s 2, a 1) is inconsistent the first requirement. As such, any update that makes the Q-value of (s 2, a 2) higher undercuts the justification for it to be higher (i.e., makes the "max" value of its successor state (s 3, a 2) lower). This occurs not due to approximation error, but the inability of Q-learning to find the value of the optimal representable policy. The pseudocode of (depth-first) version of the CONQUR search framework is listed in Algorithm 1. As discussed in Sec. 3.5, a more specific instantiation of the CONQUR algorithm is listed in Algorithm. 2. Both DQN and DDQN uses a delayed version of the Q-network Q θ − (s, a) for label generation, but in a different way. In DQN, Q θ − (s, a) is used for both value estimate and action assignment σ DQN (s) = argmax a Q θ k (s, a), whereas in DDQN, Q θ − (s, a) is used only for value estimate and the action assignment is computed from the current network σ DDQN (s) = argmax a Q θ k (s, a). With respect to delusional bias, action assignment of DQN is consistent for all batches after the latest network weight transfer, as σ DQN (s) is computed from the same Q θ − (s, a) network. DDQN, on the other hand, could have very inconsistent assignments, since the action is computed from the current network that is being updated at every step. Select top scoring nodes n 1,..., n ∈ P 10: for each selected node n i do Generate c children n i,1,..., n i,c using Boltzmann sampling on D k with Q θ i 12: for each child n i,j do 13: Let assignment history σ i,j be σ i ∪ {new assignment} 14: Determine regressor θ i,j by applying update if k is a refinement ("dive") level then for each frontier node n i,j ∈ F do 25: Update regressor θ i,j by applying update to θ . In particular, we modify existing DqnAgent and DdqnAgent by adding a consistency penalty term to the original TD loss. We use TF-Agents implementation of DQN training on Atari with the default hyperparameters, which are mostly the same as that used in the original DQN paper . For conveniece to the reader, some important hyperparameters are listed in Table 2. The reward is clipped between [−1, 1] following the original DQN. We empirically evaluate our modified DQN and DDQN agents trained with consistency penalty on 15 Atari games. Evaluation is run using the training and evaluation framework for Atari provided in TF-Agents without any modifications. C.4 DETAILED Fig. 8 shows the effects of varying λ on both DQN and DDQN. Table 3 summarizes the best penalties for each game and their corresponding scores. Fig. 9 shows the training curves of the best penalization constants. Finally, Fig. 10 shows the training curves for a fixed penalization of λ = 1/2. steps, and within each window, we take the largest policy value (and over ≈2-5 multiple runs). This is done to reduce visual clutter. Our use a frontier queue of size (F) 8 or 16 (these are the top scoring leaf nodes which receive gradient updates and rollout evaluations during training). To generate training batches, we select the best node's regressor according to our scoring function, from which we generate training samples (transitions) using ε-greedy. Results are reported in Table 4 and 5, and related figures where max number of nodes are 8 or 16. We used Bellman error plus consistency penalty as our scoring function. During the training process, we also calibrated the scoring to account for the depth difference between the leaf nodes at the frontier versus the leaf nodes in the candidate pool. We calibrated by taking the mean of the difference between scores of the current nodes in the frontier with their parents. We scaled this difference by multiplying with a constant of 2.5. In our implementation, we initialized our Q-network with a pre-trained DQN. We start with the expansion phase. During this phase, each parent node splits into l children nodes and the Q-labels are generated using action assignments from the Boltzmann sampling procedure, in order to create high quality and diversified children. We start the dive phase until the number of children generated is at least F. In particular, with F = 16 configuration, we performed the expansion phase at the zero-th and first iterations, and then at every tenth iteration starting at iteration 10, then at 20, and so on until ending at iteration 90. In the F = 8 configuration, the expansion phase occurred at the zero-th and first iterations, then at every tenth iterations starting at iterations 10 and 11, then at iterations 20 and 21, and so on until ending at iterations 90 and 91. All other iterations execute the "dive" phase. For every fifth iteration, Q-labels are generated from action assignments sampled according to the Boltzmann distribution. For all other iterations, Q-labels are generated in the same fashion as the standard Q-learning (taking the max Q-value). The generated Q-labels along with the consistency penalty are then converted into gradient updates that applies to one or more generated children nodes. Each iteration consists of 10k transitions sampled from the environment. Our entire training process has 100 iterations which consumes 1M transitions or 4M frames. We used RMSProp as the optimizer with a learning rate of 2.5 × 10 −6. One training iteration has 2.5k gradient updates and we used a batch size of 32. We replace the target network with the online network every fifth iteration and reward is clipped between [−1, 1]. We use a discount value of γ = 0.99 and ε-greedy with ε = 0.01 for exploration. Details of hyper-parameter settings can be found in Table 6 (for 16 nodes) and Table 7 (for 8 nodes). We empirically evaluate our algorithms on 59 Atari games , and followed the evaluation procedure as in. We evaluate our agents every 10th iterations (and also the initial and first iteration) by suspending our training process. We evaluate on 500k frames, and we cap the length of the episodes for 108k frames. We used ε-greedy as the evaluation policy with ε = 0.001. Fig. 11 shows training curves of CONQUR with 16 nodes under different penalization strengths λ. Each plotted step of each training curve (including the baseline) shows the best performing node's policy value as evaluated with full rollouts. Table 4 shows the summary of the highest policy values achieved for all 59 games for CONQUR and the baseline under 8 and 16 nodes. Table 5 shows a similar summary, but without no-op starts (i.e. policy actions are applied immediately). Both the baseline and CONQUR improve overall, but CONQUR's advantage over the baseline is amplified. This may suggest that for more deterministic MDP environments, CONQUR may have even better improvements. The on 16 and 8 nodes use a splitting factor of 4 and 2, respectively. Table 4: Summary of scores with ε-greedy (ε = 0.001) evaluation with up to 30 no-op starts. Mini-batch size Size of the mini batch data used to train the Qnetwork. ε train ε-greedy policy for exploration during training. 0.01 ε eval ε-greedy policy for evaluating Q-regressors. 0.001 Training calibration parameter Calibration to adjust the difference between the nodes from the candidate pool m which didn't selected during both the expansion nor the dive phases. The calibration is performed based on the average difference between the frontier nodes and their parents. We denote this difference as. Discount factor γ Discount factor during the training process. 0.99

Represent the short summary for finding the corresponding scientific article (long)

We developed a search framework and consistency penalty to mitigate delusional bias.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: The paper proposes and demonstrates a Deep Convolutional Neural Network (DCNN) architecture to identify users with disguised face attempting a fraudulent ATM transaction. The recent introduction of Disguised Face Identification (DFI) framework proves the applicability of deep neural networks for this very problem. All the ATMs nowadays incorporate a hidden camera in them and capture the footage of their users. However, it is impossible for the police to track down the impersonators with disguised faces from the ATM footage. The proposed deep convolutional neural network is trained to identify, in real time, whether the user in the captured image is trying to cloak his identity or not. The output of the DCNN is then reported to the ATM to take appropriate steps and prevent the swindler from completing the transaction. The network is trained using a dataset of images captured in similar situations as of an ATM. The comparatively low clutter in the images enables the network to demonstrate high accuracy in feature extraction and classification for all the different disguises. The widespread acceptance of Automated Teller Machine (ATM) and their omnipresence in the banking sector has engendered numerous security concerns. One of the most imperative concerns being, verifying the authenticity of the user. Evidently, most of the ATMs across the globe simply rely on a card and a Personal Identification Number (PIN) for authentication. However, in either case, it is plausible that the user is not authorised to transact. For instance, illegal practices like phishing, shoulder surfing, card fraud, stolen card can cause substantial monetary loss to the owner. To overcome and identify such practices, ATMs have an inbuilt camera which records 24x7. The current state of art ATM security works in the following way: After a fraudulent transaction, the owner of the corresponding bank account reports about the fraud. The police then investigates and goes through the footage recorded by the ATM camera to find the face of the imposter. Once the face is identified, the police searches for the imposter. Clearly, this security measure can be easily gamed by using artifacts or alterations like wigs, caps, eyeglasses, beard to cover the face for intentional disguises. As a , BID6 stated that such face alterations can substantially degrade the performance of the system. Hence, this approach has a very low success rate which is unacceptable in banking sectors. Additionally, BID7 explained different openings and vulnerabilities that exist at the time of transactions due to fake entries and fake cards. Apparently, this chaos can be prevented by ensuring that the transaction proceeds only if the face is undisguised and reveal identity of the user. The proposed system extracts the user's face from the footage and checks if the face is disguised. The system is trained cleverly to identify such faces by an extensive pool of disguised and undisguised faces. If the face is disguised, the system will not allow the transaction to be proceeded, thereby preventing the imposter from stealing. To achieve this, the proposed system uses Deep Convolutional Neural Networks for image classification using statistical dimensionality reduction method. Deep networks have proved to be exceptional in computer vision problems BID9 BID4. BID9 stated a three-layer cascading style which superficially captures the high level features and refines them to detect deeper features. Analogously, the proposed system uses a five-layer architecture, first 3 layers comprises of a convolutional layers followed by a pooling layers to learn the features of the following types of images: Disguised, Partially disguised and Undisguised.2 PROPOSED SYSTEM 2.1 EXISTING MECHANISMS Plenty of research work has been published in response to the ATM security problems and a lot of it relates to using machine learning to authenticate users. BID3 proposed a facebased authentication as identity test for users and the system uses facial recognition with biometric features. T. BID10 stated the applicability of image processing by amalgamation of Face Recognition System (FRS) in the identity verification process engaged in ATMs. BID2 proposed a framework to classify local facial regions of both visible and thermal face images into biometric (regions without disguise) and non-biometric (regions with disguise) classes and used the biometric patches for facial feature extraction and matching. The fact that none of the above mentioned mechanisms are implemented in the current state-ofart endorses that they are not at par with the requirements and have to be compromised due to certain trade-offs. The mechanisms use an extensive pool of user information and glut of computing resources such as cloud storage, network bandwidth which makes them in-feasible and erratic.According to T. Suganya FORMULA0, transactor's picture should be clicked and matched to existing records after his card and PIN are verified. However, such computation cannot be relied upon as it highly depends on the remote server which holds the data and the available bandwidth. Such systems try to authenticate too much and exceed the computational limits of the machine. Also, emergency situations where the card owner is unavailable to authenticate the transaction is where the current systems BID3, BID10, BID5 suffer. Moreover, with reference to BID1, clearly Eigenface based method can be spoofed by using face masks or photos of account holder. The proposed solution takes advantage of the fact that fraudsters hide their identity to complete the transaction and not get caught in police investigation. To achieve this, fraudsters cover their faces with disguises while performing the transaction. We propose and demonstrate the use of a Convolutional Neural Network to identify face disguises, thereby preventing fraudulent transactions. An ATM equipped with the proposed DCNN model will not allow any user with disguised face to complete the transaction and prompt for removing the disguise to complete the transaction. The DCNN is trained to identify in real time whether the user interacting with the ATM is having disguised face or not without consuming any additional data or remote resources. The output is swiftly reported to the ATM to further proceed the transaction based on the computed label. The proposed system makes sure that every user interacting with the ATM shows his/her identity to the ATM and essentially allows police to identify the user corresponding to a fraud transaction. The DCNN model is implemented in python 2.7 using TensorFlow(r1.3) and is executed on an Ubuntu 16.04 operating system. Stochastic Gradient Descent method is used to train the model and Kingma and Ba's Adam algorithm to control the learning rate. The dataset used to train and test the contains 1986 images of faces covered with any combination of 4 disguises namely scarf, helmet, eyeglasses and fake beard. The images are published by BID8 and matches the requirement to an acceptable level. The images are manually classified in 3 classes, which are -disguised, undisguised and partially disguised. FIG0 exemplifies the dataset in short. While feeding to the network the 1986 images were randomly split into 1500 and 486 images which is approximately 75% and 25% split ratio. 1500 images were used for training and remaining 486 for testing. 30 batches of 50 randomly chosen samples were used for every training cycle. After every training cycle, the training set was shuffled to randomize the data and ensure generalization in learning process. • Disguised Faces which are not recognizable by humans are labelled as disguised. These images particularly contain more than one disguises and effectively hide the identity of the person in the image. 1372 images belong to disguised class. We introduced this label to adopt the network to allow users having unintentional disguises such as spectacles and natural beard. These images are recognizable by humans and the apparent disguises are part of the persons identity. There are 212 samples of partially disguised faces. Faces showing clear and effective identity of the person in the image are labelled as undisguised. A total of 402 images belong to the undisguised class. The Rectified Linear Unit (ReLU) activation function is used to model the neurons. The following equation defines ReLU function. DISPLAYFORM0 The activation is simply thresholded at zero. Due to its linear and non-saturating form, it significantly accelerates the convergence of Stochastic Gradient Descent (SGD) compared to the sigmoid or tanh functions. The network parameters are initialised with a standard deviation of 0.1 to facilitate SGD.Furthermore, softmax classification function is used to classify each sample and present the predicted output to cost function to calculate training error. Cross entropy is the loss measure used to measure training error of the learner and it is as defined below The training steps are executed using the tensorflow.train.AdamOptimizer function in TensorFlow, with 1e-4 as the initial learning rate and cross entropy as the cost functions needed to be optimized. The AdamOptimizer uses Kingma and Ba's Adam algorithm to control the learning rate. Adam offers several advantages and foremost is that it uses moving averages of the parameters (momentum). BID0 ] discusses the reasons for why this is beneficial. Simply put, this enables Adam to use a larger effective step size, and the algorithm will converge to this step size without fine tuning. DISPLAYFORM1 The DCNN model contains 3 convolutional layers each followed by a pooling layer. After the 3rd pooling layer two fully connected hidden layers are introduced to classify the features learned by the convolutional layers. The output layer uses softmax function to calculate probabilities of the image to belong to each class. FIG2 shows the model architecture along with specific network parameters and respective input and output image dimensions. The first convolutional layer uses 32 feature maps with kernel dimensions as (7x7x3). Similarly, the 2nd and 3rd convolutional layers use 64 and 128 feature maps and kernels with dimensions (5x5x32) and (5x5x64) respectively. All 3 pooling layers use (2x2) max pooling with stride length of 2. The 3 convolutional layers are accompanied with bias vectors of sizes FORMULA0, FORMULA0, FORMULA0 respectively. The following 2 fully connected hidden layers have weight matrices of dimensions FORMULA0 and FORMULA0 respectively and bias vectors of size (512x1) each. The output layer contains 3 neurons and weight matrix of size (3x512) with (3x1) sized bias vector. In all, the network incorporates 9,962,819 trainable parameters which are eventually learned by the network in training step and used to classify the unknown examples while testing. The performance of the model is evaluated on a test set which is disjoint from training set, simulating the real life situation. It reports a final accuracy of 90.535% after around 60 training cycles. Figure 3 shows evolution of test accuracy over 60 training cycles and Figure 4 demonstrates accuracy for every training step for first 5 training cycles. Neural Networks are proven to work efficiently in numerous classification and regression tasks due to their flexibility and high fault tolerance. The system using Deep Convolutional Neural Network demonstrates high accuracy and works efficiently in real time without needing to access data on cloud. Speed being indispensable to banking transactions is one of the significant advantages of this model. The performance can be significantly improved by training through a large dataset obtained from actual ATMs.

represent the short summary for finding the corresponding scientific article (long)

Proposed System can prevent impersonators with facial disguises from completing a fraudulent transaction using a pre-trained DCNN.

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.