Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Achieving machine intelligence requires a smooth integration of perception and reasoning, yet models developed to date tend to specialize in one or the other; sophisticated manipulation of symbols acquired from rich perceptual spaces has so far proved elusive. Consider a visual arithmetic task, where the goal is to carry out simple arithmetical algorithms on digits presented under natural conditions (e.g. hand-written, placed randomly). We propose a two-tiered architecture for tackling this kind of problem. The lower tier consists of a heterogeneous collection of information processing modules, which can include pre-trained deep neural networks for locating and extracting characters from the image, as well as modules performing symbolic transformations on the representations extracted by perception. The higher tier consists of a controller, trained using reinforcement learning, which coordinates the modules in order to solve the high-level task. For instance, the controller may learn in what contexts to execute the perceptual networks and what symbolic transformations to apply to their outputs. The ing model is able to solve a variety of tasks in the visual arithmetic domain,and has several advantages over standard, architecturally homogeneous feedforward networks including improved sample efficiency. Recent successes in machine learning have shown that difficult perceptual tasks can be tackled efficiently using deep neural networks BID18. However, many challenging tasks may be most naturally solved by combining perception with symbol manipulation. The act of grading a question on a mathematics exam, for instance, requires both sophisticated perception (identifying discrete symbols rendered in many different writing styles) and complex symbol manipulation (confirming that the rendered symbols correspond to a correct answer to the given question). In this work, we address the question of creating machine learning systems that can be trained to solve such perceptuo-symbolic problems from a small number of examples. In particular, we consider, as a first step toward fullblown exam question grading, the visual arithmetic task, where the goal is to carry out basic arithmetic algorithms on hand-written digits embedded in an image, with the wrinkle that an additional symbol in the image specifies which of a handful of algorithms (e.g. max, min, +, *) should be performed on the provided digits. One straightforward approach to solving the visual arithmetic task with machine learning would be to formulate it as a simple classification problem, with the image as input, and an integer giving the correct answer to the arithmetic problem posed by the image as the label. A convolutional neural network (CNN; BID17 could then be trained via stochastic gradient descent to map from input images to correct answers. However, it is clear that there is a great deal of structure in the problem which is not being harnessed by this simple approach, and which would likely improve the sample efficiency of any learning algorithm that was able to exploit it. While the universal approximation theorem BID9 suggests that an architecturally homogeneous network such as a CNN should be able to solve any task when it is made large enough and given sufficient data, imposing model structure becomes important when one is aiming to capture human-like abilities of strong generalization and learning from small datasets BID16 .In particular, in this instance we would like to provide the learner with access to modules implementing information processing functions that are relevant for the task at hand -for example, modules that classify individual symbols in the image, or modules that perform symbolic computations on stored representations. However, it is not immediately clear how to include such modules in standard deep networks; the classifiers need to somehow be applied to the correct portion of the image, while the symbolic transformations need to be applied to the correct representations at the appropriate time and, moreover, will typically be non-differentiable, precluding the possibility of training via backpropogation. In this work we propose an approach that solves this type of task in two steps. First, the machine learning practitioner identifies a collection of modules, each performing an elementary information processing function that is predicted to be useful in the domain of interest, and assembles them into a designed information processing machine called an interface BID30 that is coupled to the external environment. Second, reinforcement learning (RL) is used to train a controller to make use of the interface; use of RL alleviates any need for the interface to be differentiable. For example, in this paper we make use of an interface for the visual arithmetic domain that contains: a discrete attention mechanism; three pre-trained perceptual neural networks that classify digits/classify arithmetic symbols/detect salient locations (respectively); several modules performing basic arithmetic operations on stored internal representations. Through the use of RL, a controller learns to sequentially combine these components to solve visual arithmetic tasks. We propose a novel recipe for constructing agents capable of solving complex tasks by sequentially combining provided information processing modules. The role of the system designer is limited to choosing a pool of modules and gathering training data in the form of input-output examples for the target task. A controller is then trained by RL to use the provided modules to solve tasks. We evaluate our approach on a family of visual arithmetic tasks wherein the agent is required to perform arithmetical reduction operations on handwritten digits in an image. Our experiments show that the proposed model can learn to solve tasks in this domain using significantly fewer training examples than unstructured feedforward networks. The remainder of the article is organized as follows. In Section 2 we describe our general approach and lay down the required technical machinery. In Section 3 we describe the visual arithmetic task domain in detail, and show how our approach may be applied there. In Section 4 we present empirical demonstrating the advantages of our approach as it applies to visual arithmetic, before reviewing related work in Section 5 and concluding with a discussion in Section 6. Our approach makes use of standard reinforcement learning formalisms BID27. The external world is modelled as a Partially Observable Markov Decision Process (POMDP), E. Each time step E is in a state s t, based upon which it emits an observation o t that is sent to the learning agent. The agent responds with an action a t, which causes the environment to emit a reward r t. Finally, the state is stochastically updated according to E's dynamics, s t+1 ∼ P (·|s t, a t). This process repeats for T time steps. The agent is assumed to choose a t according to a parameterized policy that maps from observation-action histories to distributions over actions, i.e. a t ∼ π θ (·|h t), where h t = o 0, a 0,..., o t and θ is a parameter vector. We make extensive use of the idea of an interface as proposed in BID30. An interface is a designed, domain-specific machine that mediates a learning agent's interaction with the external world, providing a representation (observation and action spaces) which is intended to be more conducive to learning than the raw representation provided by the external world. In this work we formalize an interface as a POMDP I distinct from E, with its own state, observation and action spaces. The interface is assumed to be coupled to the external world in a particular way; each time step E sends an observation to I, which potentially alters its state, after which I emits its own observation to the agent. When the agent responds with an action, it is first processed by I, which once again has the opportunity to change its state, after which I sends an action to E. The agent may thus be regarded as interacting with a POMDP C comprised of the combination of E and I. C's observation and action spaces are the same as those of I, its state is the concatenation of the states of I and E, and its dynamics are determined by the nature of the coupling between I and E. BID30 learn to control interfaces in order to solve purely algorithmic tasks, such as copying lists of abstractly (rather than perceptually) represented digits. One of the main insights of the current work is that the idea of interfaces can be extended to tasks with rich perceptual domains by incorporating pre-trained deep networks to handle the perceptual components. We train controllers using the actor-critic algorithm (see e.g. BID27 BID2 BID20 BID26). We model the controller as a policy π θ that is differentiable with respect to its parameters θ. Assume from the outset that the goal is to maximize the expected sum of discounted rewards when following π θ: DISPLAYFORM0 where DISPLAYFORM1 is the probability of that trajectory under π θ, and γ ∈ is a discount factor. We look to maximize this objective using gradient ascent; however, it is not immediately clear how to compute ∇ θ J(θ), since the probability of a trajectory P π θ (τ) is a function of the environment dynamics, which are generally unknown. Fortunately, it can be shown that an unbiased estimate of ∇ θ J(θ) can be obtained by differentiating a surrogate objective function that can be estimated from sample trajectories. Letting R t = T −1 i=t γ i−t r i, the surrogate objective is: DISPLAYFORM2 The standard REINFORCE algorithm BID29 consists in first sampling a batch of trajectories using π θ, then forming an empirical estimate f (θ) of F(θ). ∇ θ f (θ) is then computed, and the parameter vector θ updated using standard gradient ascent or one of its more sophisticated counterparts (e.g. ADAM; BID13).The above gradient estimate is unbiased (i.e. E [∇ θ f (θ)] = ∇ θ J(θ)) but can suffer from high variance. This variance can be somewhat reduced by the introduction of a baseline function b t (h) into Equation: DISPLAYFORM3 It can be shown that including b t (h) does not bias the gradient estimate and may lower its variance if chosen appropriately. The value function for the current policy, et al., 2000) -however this will rarely be known. A typical compromise is to train a function V ω (h), parameterized by a vector ω, to approximate V π θ (h) at the same time as we are training π θ. Specifically, this is achieved by minimizing a sample-based estimate of DISPLAYFORM4 DISPLAYFORM5 We employ two additional standard techniques BID20. First, we have V ω (h) share the majority of its parameters with π θ, i.e. ω = θ. This allows the controller to learn useful representations even in the absence of reward, which can speed up learning when reward is sparse. Second, we include in the objective a term which encourages π θ to have high entropy, thereby favouring exploration. Overall, given a batch of N sample trajectories from policy π θ, we update θ in the direction of the gradient of the following surrogate objective:1 N T k t as the first term does not provide a useful training signal for V θ. We now describe the Visual Arithmetic task domain in detail, as well as the steps required to apply our approach there. We begin by describing the external environment E, before describing the interface I, and conclude the section with a specification of the manner in which E and I are coupled in order to produce the POMDP C with which the controller ultimately interacts. Tasks in the Visual Arithmetic domain can be cast as image classification problems. For each task, each input image consists of an (n, n) grid, where each grid cell is either blank or contains a digit or letter from the Extended MNIST dataset BID1. Unless indicated otherwise, we use n = 2. The correct label corresponding to an input image is the integer that from applying a specific (task-varying) reduction operation to the digits present in the image. We consider 5 tasks within this domain, grouped into two kinds. Changing the task may be regarded as changing the external environment E. Single Operation Tasks. In the first kind of task, each input image contains a randomly selected number of digits (2 or 3 unless otherwise stated) placed randomly on the grid, and the agent is required to output an answer that is a function of the both the specific task being performed and the digits displayed in the image. We consider 4 tasks of this kind: Sum, Product, Maximum and Minimum. Example input images are shown in FIG0, Top Row. Combined Task. We next consider a task that combines the four Single Operation tasks. Each input example now contains a capital EMNIST letter in addition to 2-to-3 digits. This letter indicates which reduction operation should be performed on the digits: A indicates add/sum, M indicates multiplication/product, X indicates maximum, N indicates minimum. Example input images are shown in FIG0, Bottom Row. Succeeding on this task requires being able to both carry out all the required arithmetic algorithms and being able to identify, for any given input instance, which of the possible algorithms should be executed. def update_interface(ExternalObs e, string action): if action == "right": fovea_x += 1 elif action == "left": fovea_x -= 1 elif action == "down": fovea_y += 1 elif action == "up": fovea_y -= 1 elif action == "+": store += digit elif action == " * ": store * = digit elif action == "max": store = max(store, digit) elif action == "min": store = min(store, digit) elif action == "+1": store += 1 elif action == "classify_op": op = op_classifier(get_glimpse(e, fovea_x, fovea_y)) elif action == "classify_digit": digit = digit_classifier(get_glimpse(e, fovea_x, fovea_y)) elif action == "update_salience": salience_map = salience_detector(e, fovea_x, fovea_y) else:raise Exception("Invalid action") obs = (fovea_x, fovea_y, store, op, digit, salience_map) return obs We now describe the interface I that is used to solve tasks in this domain. The first step is to identify information processing functions that we expect to be useful. We can immediately see that for Visual Arithmetic, it will be useful to have modules implementing the following functions:1. Detect and attend to salient locations in the image.2. Classify a digit or letter in the attended region.3. Manipulate symbols to produce an answer. We select modules to perform each of these functions and then assemble them into an interface which will be controlled by an agent trained via reinforcement learning. A single interface, depicted in FIG1, is used to solve the various Visual Arithmetic tasks described in the previous section. This interface includes 3 pre-trained deep neural networks. Two of these are instances of LeNet , each consisting of two convolutional/max-pool layers followed by a fully-connected layer with 128 hidden units and RELU non-linearities. One of these LeNets, the op classifier, is pre-trained to classify capital letters from the EMNIST dataset. The other LeNet, the digit classifier, is pre-trained to classify EMNIST digits. The third network is the salience detector, a multilayer perceptron with 3 hidden layers of 100 units each and RELU non-linearities. The salience network is pre-trained to output a salience map when given as input scenes consisting of randomly scattered EMNIST characters (both letters and digits). In the Visual Arithmetic setting, E may be regarded as a degenerate POMDP which emits the same observation, the image containing the EMNIST letters/digits, every time step. I sends the contents of its store field (see FIG1) to E every time step as its action. During training, E responds to this action with a reward that depends on both the time step and whether the action sent to E corresponds to the correct answer to the arithmetic problem represented by the input image. Specifically, for all but the final time step, a reward of 0 is provided if the answer is correct, and −1/T otherwise. On the final time step, a reward of 0 is provided if the answer is correct, and −1 otherwise. Each episode runs for T = 30 time steps. At test time, no rewards are provided and the contents of the interface's store field on the final time step is taken as the agent's guess for the answer to the arithmetic problem posed by the input image. For the controller, we employ a Long Short-Term Memory (LSTM) BID8 with 128 hidden units. This network accepts observations provided by the interface (see FIG1 as input, and yields as output both π θ (·|h) (specifically a softmax distribution) from which an action is sampled, and V θ (h) (which is only used during training). The weights of the LSTM are updated according to the actor-critic algorithm discussed in Section 2.3. In this section, we consider experiments applying our approach to the Visual Arithmetic domain. These experiments involve the high-level tasks described in Section 3.1. For all tasks, our reinforcement learning approach makes use of the interface described in Section 3.2 and the details provided in Section 3.3.Our experiments look primarily at how performance is influenced by the number of external environment training samples provided. For all sample sizes, training the controller with reinforcement learning requires many thousands of experiences, but all of those experiences operate on the small provided set of input-output training samples from the external environment. In other words, we assume the learner has access to a simulator for the interface, but not one for the external environment. We believe this to be a reasonable assumption given that the interface is designed by the machine learning practitioner and consists of a collection of information processing modules which will, in most cases, take the form of computer programs that can be executed as needed. We compare our approach against convolutional networks trained using cross-entropy loss, the de facto standard when applying deep learning to image classification tasks. These feedforward networks can be seen as interacting directly with the external environment (omitting the interface) and running for a single time step, i.e. T = 1. The particular feedforward architecture we experiment with is the LeNet , with either 32, 128 or 512 units in the fully-connected layer. Larger, more complex networks of course exist, but these will likely require much larger amounts of data to train, and here we are primarily concerned with performance at small sample sizes. For training the convolutional networks, we treat all tasks as classification problems with 101 classes. The first 100 classes correspond to the integers 0-99. All integers greater than or equal to 100 (e.g. when multiplying 3 digits) are subsumed under the 101st class. Experiments showing the sample efficiency of the candidate models on the Single Operation tasks are shown in FIG2. Similar for the Combined task are shown in FIG3. In both cases, our reinforcement learning approach is able to leverage the inductive bias provided by the interface to achieve good performance at significantly smaller sample sizes than the relatively architecturally homogeneous feedforward convolutional networks. in cognitive science. Production systems date back to Newell and Simon's pioneering work on the study of high-level human problem solving, manifested most clearly in the General Problem Solver (GPS; BID23 . While production systems have fallen out of favour in mainstream AI, they still enjoy a strong following in the cognitive science community, forming the core of nearly every prominent cognitive architecture including ACT-R , SOAR BID22 BID15, and EPIC . However, the majority of these systems use hand-coded, symbolic controllers; we are not aware of any work that has applied recent advances in reinforcement learning to learn controllers for these systems for difficult tasks. A related body of work concerns recurrent neural networks applied to supervised learning problems. BID19, for example, use reinforcement learning to train a recurrent network to control the location of an attentional window in order to classify images containing MNIST digits placed on cluttered s. Our approach may be regarded as providing a recipe for building similar kinds of models while placing greater emphasis on tasks with a difficult algorithmic components and the use of structured interfaces. Neural Abstract Machines. One obvious point of comparison to the current work is recent research on deep neural networks designed to learn to carry out algorithms on sequences of discrete symbols. Some of these frameworks, including the Differentiable Forth Interpreter BID25 and TerpreT , achieve this by explicitly generating code, while others, including the Neural Turing Machine (NTM;, Neural Random-Access Machine (NRAM; BID14, Neural Programmer (NP; BID21, and Neural Programmer-Interpreter (NPI; BID24 avoid generating code and generally consist of a controller network that learns to perform actions using a differentiable external computational medium (i.e. a differentiable interface) in order to carry out an algorithm. Our approach is most similar to the latter category, the main difference being that we have elected not to require the external computational medium to be differentiable, which provides it with greater flexibility in terms of the components that can be included in the interface. In fact, our work is most similar to BID30, which also uses reinforcement learning to learn algorithms, and from which we borrowed the idea of an interface, the main difference being that we have included deep networks in our interfaces in order to tackle tasks with non-trivial perceptual components. Visual Arithmetic. Past work has looked at learning arithmetic operations from visual input. BID10 train a multi-layer perceptron to map from images of two 7-digit numbers to an image of a number that is some task-specific function of the numbers in the input images. Specifically, they look at addition, subtraction, multiplication and Roman-Numeral addition. Howeover, they do not probe the sample efficiency of their method, and the digits are represented using a fixed computer font rather than being hand-written, making the perceptual portion of the task significantly easier. Gaunt et al. FORMULA0 addresses a task domain that is similar to Visual Arithmetic, and makes use of a differentiable code-generation method built on top of TerpreT. Their work has the advantage that their perceptual modules are learned rather than being pre-trained, but is perhaps less general since it requires all components to be differentiable. Moreover, we do not view our reliance on pre-trained modules as particularly problematic given the wide array of tasks deep networks have been used for. Indeed, we view our approach as a promising way to make further use of any trained neural network, especially as facilities for sharing neural weights mature and enter the mainstream. Additional work has focused more directly on the use of neural modules and adaptively choosing groups of modules to apply depending on the input. Endto-End Module Networks BID11 use reinforcement learning to train a recurrent neural network to lay out a feedforward neural network composed of elements of a stored library of neural modules (which are themselves learnable). Our work differs in that rather than having the layout of the network depend solely on the input, the module applied at each stage (i.e. the topology of the network) may depend on past module applications within the same episode, since a decision about which module to apply is made at every time step. Systems built using our framework can, for example, use their modules to gather information about the environment in order to decide which modules to apply at a later time, a feat that is not possible with Module Networks. In Deep Sequential Neural Networks (DSNN; BID3), each edge of a fixed directed acyclic graph (DAG) is a trainable neural module. Running the network on an input consists in moving through the DAG starting from the root while maintaining a feature vector which is repeatedly transformed by the neural modules associated with the traversed edges. At each node of the DAG, an outgoing edge is stochastically selected for traversal by a learned controller which takes the current features as input. This differs from our work, where each module may be applied many times rather than just once as is the case for the entirely feedforward DSNNs (where no module appears more than once in any path through the DAG connecting the input to the output). Finally, PathNet is a recent advance in the use of modular neural networks applied to transfer learning in RL BID4 ). An important difference from our work is that modules are recruited for the entire duration of a task, rather than on the more fine-grained step-by-step basis used in our approach. There are number of possible future directions related to the current work, including potential benefits of our approach that were not explored here. These include the ability to take advantage of conditional computation; in principle, only the subset of the interface needed to carry out the chosen action needs to be executed every time step. If the interface contains many large networks or other computationally intensive modules, large speedups can likely be realized along these lines. A related idea is that of adaptive computation time; in the current work, all episodes ran for a fixed number of time steps, but it should be possible to have the controller decide when it has the correct answer and stop computation at that point, saving valuable computational resources. Furthermore, it may be beneficial to train the perceptual modules and controller simultaneously, allowing the modules to adapt to better perform the uses that the controller finds for them. Finally, the ability of reinforcement learning to make use of discrete and non-differentiable modules opens up a wide array of possible interface components; for instance, a discrete knowledge base may serve as a long term memory. Any generally intelligent system will need many individual competencies at its disposal, both perceptual and algorithmic; in this work we have proposed one path by which a system may learn to coordinate such competencies. We have proposed a novel approach for solving tasks that require both sophisticated perception and symbolic computation. This approach consists in first designing an interface that contains information processing modules such as pre-trained deep neural networks for processing perceptual data and modules for manipulating stored symbolic representations. Reinforcement learning is then used to train a controller to use the interface to solve tasks. Using the Visual Arithmetic task domain as an example, we demonstrated empirically that the interface acts as a source of inductive bias that allows tasks to be solved using a much smaller number of training examples than required by traditional approaches.

We use reinforcement learning to train an agent to solve a set of visual arithmetic tasks using provided pre-trained perceptual modules and transformations of internal representations created by those modules.

Represent the TLDR for finding the corresponding scientific article.

Introduce an approach to allow agents to learn  PPDDL action models incrementally over multiple planning problems under the framework of reinforcement learning.


Question: represent the scientific article to find a short tldr

Question: Animals develop novel skills not only through the interaction with the environment but also from the influence of the others. In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers. Specifically, we first define a metric to measure the distance between policies then quantitatively derive the definition of uniqueness. Unlike previous precarious joint optimization approaches, the social uniqueness motivation in our work is imposed as a constraint to encourage the agent to learn a policy different from the existing agents while still solve the primal task. The ing algorithm, namely Interior Policy Differentiation (IPD), brings about performance improvement as well as a collection of policies that solve a given task with distinct behaviors The paradigm of Reinforcement Learning (RL), inspired by cognition and animal studies , can be described as learning by interacting with the environment to maximize a cumulative reward . From the perspective of ecology, biodiversity as well as the development of various skills are crucial to the continuation and evolution of species (Darwin, 1859;). Thus the behavioral diversity becomes a rising topic in RL. Previous works have tried to encourage the emergence of behavioral diversity in RL with two approaches: The first approach is to design interactive environments which contain sufficient richness and diversity. For example, show that rich environments enable agents to learn different locomotion skills even using the standard RL algorithms. Yet designing a complex environment requires manual efforts, and the diversity is limited by the obstacle classes. The second approach to increase behavioral diversity is to motivate agents to explore beyond just maximizing the reward for the given task. proposed to maximize a heuristically defined novelty metric between policies through task-novelty joint optimization, but the final performance of agents is not guaranteed. In this work, we address the topic of policy differentiation in RL, i.e., to improve the diversity of RL agents while keeping their ability to solve the primal task. We draw the inspiration from the Social Influence in animal society (; ; van ; ;) and formulate the concept of social influence in the reinforcement learning paradigm. Our learning scheme is illustrated in Fig 1. The target agent not only learns to interact with the environment to maximize the reward but also differentiate the actions it takes in order to be different from other existing agents. Since the social influence often acts on people passively as a sort of peer pressure, we implement the social influence in terms of social uniqueness motivation and consider it as a constrained optimization problem. In the following of our work, we first define a rigorous policy distance metric in the policy space to compare the similarity of the agents. Then we develop an optimization constraint using the proposed metric, which brings immediate rather than episodic feedback in the learning process. A novel method, namely Interior Policy Differentiation (IPD), is further I should learn to run as fast as I can I should try to be different Figure 1: The illustration of learning with social influence. Instead of focusing only on the primal task, an additional constraint is introduced to the target agent, motivating it to not only perform well in the primal task but also take actions differently to other existing agents. proposed as a better solution for the constrained policy optimization problem. We benchmark our method on several locomotion tasks and show it can learn various diverse and well-behaved policies for the given tasks based on the standard Proximal Policy Optimization (PPO) algorithm . Intrinsic motivation methods. The Variational Information Maximizing Exploration (VIME) method is designed by to tackle the sparse reward problems. In VIME, an intrinsic reward term based on the maximization of information gains is added to contemporary RL algorithms to encourage exploration. The curiosity-driven methods, proposed by and Burda et al. (2018a) define intrinsic rewards according to prediction errors of neural networks. i.e., when taking previous unseen states as inputs, networks trained with previous states will tend to predict with low accuracy, so that such prediction errors can be viewed as rewards. Burda et al. (2018b) proposed Random Network Distillation (RND) to quantify intrinsic reward by prediction differences between a fixed random initialized network and another randomly initialized network trained with previous state information. proposed Competitive Experience Replay (CER), in which they use two actors and a centralized critic, and defined an intrinsic reward by the state coincidence of two actors. The values of intrinsic rewards are fixed to be ±1 for the two actors separately. All of those approaches leverage the weighted sum of the external rewards, i.e., the primal rewards provided by environments, and intrinsic rewards that provided by different heuristics. A challenging problem is the trade-off between external rewards and intrinsic rewards. The Task-Novelty Bisector (TNB) learning method introduced by aims to solve such problem by jointly optimize the extrinsic rewards and intrinsic rewards. Specifically, TNB updates the policy in the direction of the angular bisector of the two gradients, i.e., gradients of the extrinsic and intrinsic objective functions. However, the foundation of such joint optimization is not solid. Besides, creating an extra intrinsic reward function and evaluating the novelty of states or policies always requires additional neural networks such as auto-encoders. Thus extra computation expenses are needed . Diverse behaviors from rich environments and algorithms. introduce the Distributed Proximal Policy Optimization (DPPO) method and enable agents with simulated bodies to learn complex locomotion skills in a diverse set of challenging environments. Although the learning reward they utilize is straightforward, the skills their policy learned are quite impressive and effective in traveling terrains and obstacles. Their work shows that rich environments can encourage the emergence of different locomotion behaviors, but extra manual efforts are required in designing such environments. The research of shows that different RL algorithms may converge to different policies for the same task. The authors find that algorithms based on policy gradient tend to converge to the same local optimum in the game of Pitfall, while off-policy and value-based algorithms are prone to learn sophisticated strategies. On the contrary, in this paper, we are more interested in how to learn different policies through a single learning algorithm and learn the capability of avoiding local optimum. , maintains model uncertainty given the data collected from the environment via an ensemble of deep neural networks. To encourage the emergence of behavioral diversity in RL, we first define a metric to measure the difference between policies, which is the foundation for the later algorithm we propose. We denote the learned policies as {π θi ; θ i ∈ Θ, i = 1, 2, ...}, wherein θ i represents parameters of the i-th policy, Θ denotes the whole parameter space. In the following, we omit π and denote a policy π θi as θ i for simplicity unless stated otherwise. Mathematically, a metric should satisfy three important properties, namely the identity, the symmetry as well as the triangle inequality. Definition 1 A metric space is an ordered pair (M, d) where M is a set and d is a metric on M, i.e., a function d: M × M → R such that for any x, y, z ∈ M, the following holds: We use the Total Variance Divergence D T V to measure the distance between policies. Concretely, for discrete probability distributions p and q, this distance is defined as is a metric on Θ, thus (Θ, D ρ T V) is a metric space. 2003;; ), and similar can be get 3 It can be extended to continuous state and action spaces by replacing the sums with integrals. 4 The factor 1 2 in is omitted in our work for conciseness. Consequently, to motivate RL with the social uniqueness, we hope our method can maximize the uniqueness of a new policy, i.e., max θ U(θ|Θ ref), where the Θ ref includes all the existing policies. In practice, the calculation of D ρ T V (θ i, θ j) is based on Monte Carlo estimation. i.e., we need to sample s from ρ(s). Although in finite state space we can get precise estimation after establishing ergodicity, problem arises when we are facing continuous state cases. i.e. it is difficult to efficiently get enough samples. Formally, we denote the domain of ρ(s) as S and denote the domain of ρ θ (s) as S θ ⊂ S, where ρ θ (s):= ρ(s|s ∼ θ) and in finite time horizon problems ρ(s|s ∼ θ) = P (s 0 = s|θ) + P (s 1 = s|θ) +... + P (s T = s|θ). As we only care about the reachable regions, the domain S can be divided In order to improve the sample efficiency, we propose to approximate, where θ is a certain fixed behavior policy that irrelevant to θ i, θ j. Such approximation requires a necessary condition: The domain of possible states are similar between different policies: When such condition holds, we can use ρ(s|s ∼ θ) as our choice of ρ(s), and the properties in Definition 1 still holds. In practice, the Condition 1 always holds as we can ensure this by adding sufficiently large noise on θ, while the permitted state space is always limited. And for more general cases, to satisfy the properties in Definition 1, we must sample s from S θ ∪ S θj, accordingly, where N represents random action when a policy have never been trained or visited such state domain. Plugging Eq. into Eq., the objective function of policy differentiation is While the first two terms are related to the policy θ, the last term is only related to the domain S θ. If we enable sufficient exploration in training as well as in the initialization of θ, the last term will disappear (i.e. S θj ⊂ S θ). Hence we can also use D Proposition 1 (Unbiased Single Trajectory Estimation) The estimation of ρ θ (s) using a single trajectory τ is unbiased. The proof of Proposition 1 is in Appendix B. Given the definition of uniqueness and a practically unbiased sampling method, the next step is to develop an efficient learning algorithm. In the traditional RL paradigm, maximizing the expectation of cumulative rewards g = t=0 γ t r t is commonly used as the objective. i.e. max θ∈Θ E τ ∼θ [g], where τ ∼ θ denotes a trajectory τ sampled from the policy θ using Monte Carlo methods. To improve the behavioral diversity of different agents, the learning objective must take both reward from the primal task and the policy uniqueness into consideration. Previous approaches (; ; a; b;) often directly write the weighted sum of the reward from the primal task and the intrinsic reward g int = t=0 γ t r int,t, where r int,t denotes the intrinsic reward (e.g., as the uniqueness reward in our case) as follows, where 0 < α < 1 is a weight parameter. Such an objective is sensitive to the selection of α as well as the formulation of r int. For example, in our case formulating the intrinsic reward r int as will in significantly different . Besides, a trade-off arises in the selection of α: while a large α may undermine the contribution of intrinsic reward, a small α could ignore the importance of the reward, leading to the failure of agent in solving the primal task. To tackle these issues, we draw inspiration from the observation that social uniqueness motivates people in passive ways. In other words, it plays more like a constraint rather than an additional target. Therefore, we change the multi-objective optimization problem in Eq. into a constrained optimization problem as: where r 0 is a threshold indicating minimal permitted uniqueness, and r int,t denotes a moving average of r int,t. Further discussion on the selection of r 0 will be deliberated in Appendix D. From the perspective of optimization, Eq. can be viewed as a penalty method which replaces the constrained optimization problem in Eq. with the penalty term r int and the penalty coefficient 1−α α > 0, where the difficulty lies in the selection of α. The work of ) tackles this challenge by the Task Novel Bisector (TNB) in the form of Feasible Direction Methods (FDMs) . As a heuristic approximation, that approach requires reward shaping and intensive emphasis on r int,t. Instead, in this work we propose to solve the constrained optimization problem Eq. by resembling the Interior Point Methods (IPMs) . In vanilla IPMs, the constrained optimization problem in Eq. is solved by reforming it to an unconstrained form with an additional barrier term in the objective as The limit of Eq. when α → 0 then leads to the solution of Eq.. Readers please refer to Appendix G for more discussion on the correspondence between those novel policy seeking methods and constrained optimization methods. However, directly applying the IPMs is computationally challenging and numerically unstable, especially when α is small. Luckily, in our proposed RL paradigm where the behavior of an agent is influenced by its peers, a more natural way can be used. Precisely, since the learning process is based on sampled transitions, we can simply bound the collected transitions in the feasible region by permitting previous trained M policies θ i ∈ Θ ref, i = 1, 2,..., M sending termination signals during the training process of new agents. In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it. Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies. At the end of the training, we then naturally obtain a new policy that has sufficient uniqueness. In this way, we no longer need to consider the trade-off problem between intrinsic and extrinsic rewards deliberately. The learning process of our method is thus more robust and no longer suffer from objective inconsistency. As our formulation of the constrained optimization problem Eq. is inspired by IPMs, we name our approach as Interior Policy Differentiation (IPD) method. The MuJoCo environment We demonstrate our proposed method on the OpenAI Gym where the physics engine is based on MuJoCo . Concretely, we test on three locomotion environments, the Hopper-v3 (11 observations and 3 actions), Walker2d-v3 (11 observations and 2 actions), and HalfCheetah-v3 (17 observations and 6 actions). In our experiments, all the environment parameters are set as default values. Uniqueness beyond intrinsic stochasticity Experiments in show that policies that perform differently can be produced by simply selecting different random seeds before training. Before applying our method to improve behavior diversity, we firstly benchmark how much uniqueness can be generated from the stochasticity in the training process of vanilla RL algorithms as well as the random weight initialization. In this work, we mainly demonstrate our proposed method based on PPO . The extension to other popular algorithms is straightforward. We also compare our proposed method with the TNB and weighted sum reward (WSR) approaches as different ways to combine the goal of the task and the uniqueness motivation . More implementation details are depicted in Appendix D. According to Theorem 2, the uniqueness r int in equation under our uniqueness metric can be unbiased approximated by i.e., we utilize the metric directly in learning new policies instead of applying any kind of reshaping. We implement WSR, TNB, and our method in the same experimental settings and for each method, 10 different policies are trained and try to be unique with regard to all previously trained policies sequentially. Concretely, the 1st policy is trained by ordinary PPO without any social influence. The 2nd policy should be different from 1st policy, and the 3rd should be different from the previous two policies, and so on. Fig.2 shows the qualitative of our method. We visualize the motion of agents by drawing multiple frames representing the pose of agents at different time steps in the same row. The horizontal interval between consecutive frames is proportional to the velocity of agents. The settings of the frequency of highlighted frames and the correlation between interval and velocity are fixed for each environment. The visualization starts from the beginning of each episode and therefore the readers can get sense of the process of acceleration as well as the pattern of motion of agents clearly. Fig. 3 shows our experimental in terms of uniqueness (the x-axis) and the performance (the y-axis). Policies in the upper right are the more unique ones with higher performance. In Hopper and HalfCheetah, our proposed method distinctively outperforms other methods. In Walker2d, both WSR and our method work well in improving the uniqueness of policies, but none of the three methods can find way to surpass the performance of PPO apparently. Detailed comparison on the task related rewards are carried out in Table 1. A box figure depicting the performance of each trained policy and their reward gaining curve are disposed in Fig.5 and Fig.6 in Appendix C. And Fig.7 in Appendix C provides more detailed from the view of uniqueness. In addition to averaged reward, we also use success rate as another metrics to compare the performance of different approaches. In this work, we consider a policy is success when its performance is at least as good as the averaged performance of policies trained without social influences. To be specific, we use the averaged final performance of PPO as the baseline. If a new policy, which aims at performing differently to solve the same task, surpasses the baseline during its training process, it will be regarded as a successful policy. Through the success rate, we know the policy does not learn unique behavior at the expense of performance. Table 1 shows the success rate of all the methods, including the PPO baseline. The show that our method can always surpass the average baseline during training. Thus the performance of our method can always be insured. In our experiments, we observed noticeable performance improvements in the Hopper and the HalfCheetah environments. For the environment of Hopper, in many cases, the agents trained with PPO tend to learn a policy that jumps as far as possible and then fall to the ground and terminate this episode (please refer to Fig.11 in Appendix E). Our proposed method can prevent new policies from always falling into the same local minimum. After the first policy being trapped in a local minimum, the following policies will try other approaches to avoid the same behavior, explore other feasible action patterns, and thereafter the performance may get improved. Such property shows that our method can be a helpful enhancement of the traditional RL scheme, which can be epitomized as policies could make mistakes, but they should explore more instead of hanging around the same local minimum. The similar feature attributes to the reward growth in the environment of HalfCheetah. Moreover, we can illuminate the performance improvement of HalfCheetah from another perspective. The environment of HalfCheetah is quite different from the other two for there is no explicit termination signal in its default settings (i.e., no explicit action like falling to the ground would trigger termination). At the beginning of the learning process, an agent will act randomly, ing in massive repeat, trivial samples as well as large control costs. In our learning scheme, since the agent also interacts with the peers, it can receive termination signals from the peers to prevent wasting too much effort acting randomly. During the learning process in our method, an agent will first learn to terminate itself as soon as possible to avoid heavy control costs by imitating previous policies and then learns to behave differently to pursue higher reward. From this point of view, such learning process can be regarded as a kind of implicit curriculum. As the number of policies learned with social influence grows, the difficulty of finding a unique policy may also increase. Later policies must keep away from all previous solutions. The of our ablation study on how the performance changes under different scales of social influence (i.e., the number of peers) is shown in Fig. 4, where the thresholds are selected according to our previous ablation study in Sec. D. The performance decrease is more obvious in Hopper than the other two environments for the action space of Hopper is only 3 dimensional. Thus the number of possible diverse policies can be discovered is limited. In this work, we develop an efficient approach to motivate RL to learn diverse strategies inspired by social influence. After defining the distance between policies, we introduce the definition of policy uniqueness. Regarding the problem as constrained optimization problem, our proposed method, Interior Policy Differentiation (IPD), draws the key insight of the Interior Point Methods. And our experimental demonstrate IPD can learn various well-behaved policies, and our approach can help agents to avoid local minimum and can be interpreted as a kind of implicit curriculum learning in certain cases. The first two properties are obviously guaranteed by D ρ T V. As for the triangle inequality, Figure 7: Maximal and minimal between policy uniqueness in Hopper, Walker2d and HalfCheetah environments. The are averaged over all possible combinations of 10 policies. As TNB and WSR optimize the uniqueness reward directly, their uniqueness sometimes can exceed our proposed method. However, such direct optimization will lead to decreasing in task related performance as cost. To tackle the trade-off problem, carefully hyper-parameter tuning and reward shaping is always a must. Detailed comparison on the task related rewards are carried out in Table 1 D IMPLEMENTATION DETAILS Calculation of D T V We use deterministic part of policies in the calculation of D T V, i.e., we remove the Gaussian noise on the action space in PPO and use Network Structure We use MLP with 2 hidden layers as our actor models in PPO. The first hidden layer is fixed to have 32 units. Our ablation study on the choice of unit number in the second layer is detailed in Table. 2, Table3 and Fig.8. Moreover, we choose to use 10, 64 and 256 hidden units for the three tasks respectively in all of the main experiments, after taking the success rate (Table. 2), performance (Table. 3) and computation expense (i.e. the preference to use less unit when the other two factors are similar) into consideration. Training Timesteps We fix the training timesteps in our experiments. The timesteps are fixed to be 1M in Hopper-v3, 1.6M for Walker2d-v3 and 3M for HalfCheetah. Threshold Selection In our proposed method, we can control the magnitude of policy uniqueness flexibly by adjusting the constraint threshold r 0. Choosing different thresholds will lead to different policy behaviors. Concretely., a larger threshold may drive the agent to perform more differently while smaller threshold imposes a lighter constraint on the behavior of the agent. Intuitively, a larger threshold will lead to relatively poor performance for the learning algorithm is less likely to find a feasible solution to Eq.. Besides, we do not use constraints in the form of Eq. as we need not force every single action of a new agent to be different from others. Instead, we are more care about the long term differences. Therefore, we use the cumulative uniqueness as constraints, We test our method with different choices of threshold values. The performance of agents under different thresholds are shown in Fig. 9 and more detailed analysis of their success rate is presented in Table. 2. F IMPLEMENTATION OF EQ. We do not use constraints in the form of Eq. as we need not force every single action of a new agent to be different from others. Instead, we are more care about the long term differences. Therefore, we use the cumulative uniqueness as constraints. Moreover, the constraints can be applied after the first t S timesteps (e.g. t S = 20) for the consideration of similar starting sequences. We note here, the WSR, TNB and IPD methods correspond to three approaches in constrained optimization problem. For simplicity, we consider Eq. with a more concise notion g int,t − g 0,t ≥ 0, where g int,t = t t=0 r int,t, i.e., max As the optimization of policy is based on batches of trajectory samples and is implemented with stochastic gradient descent, Eq. can be further simplified as: where g t (θ) denotes the average over a trajectory. The Penalty Method considers the constraints of Eq. by putting constraint g(θ) into a penalty term, and then solve the unconstrained problem using an iterative manner, and the limit when α → 0 lead to the solution of the primal constrained problem. As an approximation, WSR choose a fixed weight term α, and use the gradient of ∇ θ f + 1−α α ∇ θ g instead of ∇ θ f + 1−α α ∇ θ min{g(θ), 0}, thus the final solution will intensely rely on the selection of α. The Taylor series of g(θ) at pointθ is g(θ + λ p) = g(θ) + ∇ θ g(θ) T λ p + O(||λ p||) The Feasible Direction Method (FDM) considers the constraints of Eq. by first finding a direction p satisfies so that for small λ, we have and g(θ + λ p) = g(θ) + λ∇ θ g(θ) T p > 0 if g(θ) > 0 The TNB method, by using the bisector of gradients ∇ θ f and ∇ θ g, select p to be Clearly, Eq. satisfies Eq., but it is more strict than Eq. as the ∇ θ g term always exists during the optimization of TNB. In TNB, the learning stride is fixed to be, leading to problem when ∇ θ f → 0, which shows the final optimization will heavily rely on the selection of g. i.e., the shape of g is crucial for the success of TNB. or use the barrier term of −α log g(θ) instead: where α, the barrier factor, is a small positive number. As α is small, the barrier term will introduce only minuscule influence on the objective. On the other hand, when θ get closer to the barrier, the objective will increase fast. It is clear that the solution of the objective with barrier term will get closer to the primal objective as α getting smaller. Thus in practice, such methods will choose a sequence of {α i} such that 0 < α i < α k+1 and α i → 0 as k → ∞ The limit of Eq. Directly applying this method is computationally challenging and numerically unstable, especially when α is small. A more natural way can be used: since the learning process is based on sampled transitions, we can simply bound the collected transitions in the feasible region by permitting previous trained M policies θ i ∈ Θ ref, i = 1, 2,..., M sending termination signals during the training process of new agents. In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it. Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies. At the end of the training, we then naturally obtain a new policy that has sufficient uniqueness. In this way, we no longer need to consider the trade-off problem between intrinsic and extrinsic rewards deliberately. The learning process of our method is thus more robust and no longer suffer from objective inconsistency. Algorithm.1 shows the pseudo code of IPD based on PPO, where the blue lines show the addition to primal PPO algorithm.

Represent the short summary for finding the corresponding scientific article (long)

A new RL algorithm called Interior Policy Differentiation is proposed to learn a collection of diverse policies for a given primal task.

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the article for finding a single-sentence TLDR

Question: Generative adversarial networks (GANs) have been extremely effective in approximating complex distributions of high-dimensional, input data samples, and substantial progress has been made in understanding and improving GAN performance in terms of both theory and application. However, we currently lack quantitative methods for model assessment. Because of this, while many GAN variants being proposed, we have relatively little understanding of their relative abilities. In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training. We observe consistency across the various proposed metrics and, interestingly, the test-time metrics do not favour networks that use the same training-time criterion. We also compare the proposed metrics to human perceptual scores. Generative adversarial networks (GANs) aim to approximate a data distribution P, using a parameterized model distribution Q. They achieve this by jointly optimizing generative and discriminative networks BID9. GANs are end-to-end differentiable, and samples from the generative network are propagated forward to a discriminative network, and error signals are then propagated backwards from the discriminative network to the generative network. The discriminative network is often viewed as learned, adaptive loss function for the generative network. GANs have achieved state-of-the-art for a number of applications BID8, producing more realistic, sharper samples than other popular generative models, such as variational autoencoders BID22. Because of their success, many GAN frameworks have been proposed. However, it has been difficult to compare these algorithms and understand their strengths and weaknesses because we are currently lacking in quantitative methods for assessing the learned generators. In this work, we propose new metrics for measuring how realistic samples generated from GANs are. These criteria are based on a formulation of divergence between the distributions P and Q BID33 BID38: DISPLAYFORM0 Here, different choices of µ, υ, and F can correspond to different f -divergences BID33 or different integral probability metrics (IPMs) BID38. Importantly, J(Q) can be estimated using samples from P and Q, and does not require us to be able to estimate P (x) or Q(x) for samples x. Instead, evaluating J(Q) involves finding the function f ∈ F that is maximally different with respect to P and Q.This measure of divergence between the distributions P and Q is related to the GAN criterion if we restrict the function class F to be neural network functions parameterized by the vector φ and the class of approximating distributions to correspond to neural network generators G θ parameterized by the vector θ, allowing formulation as a min-max problem: H is a Reproducing Kernel Hilbert Space (RKHS) and · L is the Lipschitz constant. For the LS-DCGAN, we used b = 1 and a = 0 BID28. DISPLAYFORM1 Metric µ υ Function Class GAN (GC) log f − log(1 − f) X → R +, ∃M ∈ R: |f (x)| ≤ M Least-Squares GAN (LS) −(f − b) DISPLAYFORM2 In this formulation, Q θ corresponds to the generator network's distribution and D φ corresponds to the discriminator network (see BID33 for details).We propose using J(θ) to evaluate the performance of the generator network G θ for various choices of µ and υ, corresponding to different f -divergences or IPMs between distributions P and Q θ, that have been successfully used for GAN training. Our proposed metrics differ from most existing metrics in that they are adaptive, and involve finding the maximum over discriminative networks. We compare four metrics, those corresponding to the original GAN (GC) BID8, the Least-Squares GAN (LS) BID28,the Wasserstein GAN (IW), and the Maximum Mean Discrepency GAN (MMD) criteria. Choices for µ, υ, and F for these metrics are shown in TAB0. Our method can easily be extended to other f -divergences or IPMs. To compare these and previous metrics for evaluating GANs, we performed many experiments, training and comparing multiple types of GANs with multiple architectures on multiple data sets. We qualitatively and quantitatively compared these metrics to human perception, and found that our proposed metrics better reflected human perception. We also show that rankings produced using our proposed metrics are consistent across metrics, thus are robust to the exact choices of the functions µ and υ in Equation 2.We used the proposed metrics to quantitatively analyze three different families of GANs: Deep Convolutional Generative Adversarial Networks (DCGAN) BID34, Least-Squares GANs (LS-DCGAN), and Wasserstein GANs (W-DCGAN), each of which corresponded to a different proposed metric. Interestingly, we found that the different proposed metrics still agreed on the best GAN framework for each dataset. Thus, even though, e.g. for MNIST the W-DCGAN was trained with the IW criterion, LS-DCGAN still outperformed it for the IW criterion. Our analysis also included carrying out a sensitivity analysis with respect to various factors, such as the architecture size, noise dimension, update ratio between discriminator and generator, and number of data points. Our empirical show that: i) the larger the GAN architecture, the better the ; ii) having a generator network larger than the discriminator network does not yield good ; iii) the best ratio between discriminator and generator updates depend on the data set; and iv) the W-DCGAN and LS-DCGAN performance increases much faster than DCGAN as the number of training examples grows. These metrics thus allow us to tune the hyper-parameters and architectures of GANs based on our proposed method. GANs can be evaluated using manual annotations, but this is time consuming and difficult to reproduce. Several automatically computable metrics have been proposed for evaluating the performance of probabilistic general models and GANs in particular. We review some of these here, and compare our proposed metrics to these in our experiments. Many previous probabilistic generative models were evaluated based on the pointwise likelihood of the test data, the criterion also used during training. While GANs can be used to generate samples from the approximate distribution, its likelihood on test samples cannot be evaluated without simplifying assumptions. As discussed in BID41, likelihood often does not provide good rankings of how realistic samples look, the main goal of GANs. We evaluted the efficacy of the log-likelihood of the test data, as estimated using Annealed Importance Sampling (AIS) BID43. AIS has been to estimate the likelihood of a test sample x by considering many intermediate distributions that are defined by taking a weighted geometric mean between the prior (input) distribution, p(z), and an approximation of the joint distribution p σ (x, z) = p σ (x|z)p(z). Here, p σ (x|z) is a Gaussian kernel with fixed standard deviation σ around mean G θ (z). The final estimate depends critically on the accuracy of this approximation. In Section 4, we demonstrate that the AIS estimate of p(x) is highly dependent on the choice of this hyperparameter. The Generative Adversarial Metric BID18 measures the relative performance of two GANs by measuring the likelihood ratio of the two models. Consider two GANs with their respective trained partners, M 1 = (D 1, G 1) and M 2 = (D 2, G 2), where G 1 and G 2 are the generators and D 1 and D 2 are the discriminators. The hypothesis H 1 is that M 1 is better than M 2 if G 1 fools D 2 more than G 2 fools D 1, and vice versa for the hypothesis H 0. The likelihood-ratio is defined as: DISPLAYFORM0 where M 1 and M 2 are the swapped pairs (D 1, G 2) and (D 2, G 1), and p(x|y = 1, M) is the likelihood of x generated from the data distribution p(x) by model M and p(y = 1|x; D) indicates that discriminator D thinks x is a real sample. To evaluate this, we measure the ratio of how frequently G 1, the generator from model 1, fools D 2, the discriminator from model 2, and vice-versa: DISPLAYFORM1, where x 1 ∼ G 1 and x 2 ∼ G 2. There are two main caveats to the Generative Adversarial Metric. First, the measurement only provides comparisons between pairs of models. Second, the metric has a constraint where the two discriminators must have an approximately similar performance on a calibration dataset, which can be difficult to satisfy in practice. The Inception Score BID36 (IS) measures the performance of a model using a third-party neural network trained on a supervised classification task, e.g. Imagenet. The IS computes the expectation of divergence between the distribution of class predictions for samples from the GAN compared to the distribution of class to the distribution of class labels used to train the third-party network, DISPLAYFORM2 Here, the class prediction given a sample x is computed using the third-party neural network. In BID36 ), Google's Inception Network BID40 trained on Imagenet was the third-party neural network. IS is the most widely used metric to measure GAN performance. However, summarizing samples as the class prediction from a network trained for a different task discards much of the important information in the sample. In addition, it requires another neural network that is trained separately via supervised learning. We demonstrate an example of a failure case of IS in the Experiments section. The Fréchet Inception Distance (FID) BID14 extends upon IS. Instead of using the final classification outputs from the third-party network as representations of samples, it uses a representation computed from a late layer of the third-party network. It compares the mean m Q and covariance C Q of the Inception-based representation of samples generated by the GAN to the mean m P and covariance C P of the same representation for training samples: DISPLAYFORM3 This method relies on the Inception-based representation of the samples capturing all important information and the first two moments of the distributions being descriptive of the distribution. Classifier Two-Sample Tests (C2ST) BID27 proposes training a classifier, similar to a discriminator, that can distinguish real samples from P from generated samples from Q, and using the error rate of this classifier as a measure of GAN performance. In their work, they used single-layer and k-nearest neighbor (KNN) classifiers trained on a representation of the samples computed from a late layer of a third-party network (in this case, ResNet BID13). C2ST is an IPM BID38, like the MMD and Wasserstein metrics we propose, with µ(f) = f and υ(f) = f, but with a different function class F, corresponding to the family of classifiers chosen (in this case, single-layer networks or KNN, see see our detailed explanation in Appendix 5). The accuracy of a classifier trained to distinguish samples from distributions P and Q is just one way to measure the distance between these distributions, and, in this work, we propose a general family. Given a generator G θ with parameters θ which generates samples from the distribution Q θ, we propose to measure the quality of G θ by estimating divergence between the true data distribution P and Q θ for different choices of divergence measure. We train both G θ and D ϕ on a training data set, and measure performance on a separate test set. See Algorithm 1 for details. We consider metrics from two widely studied divergence and distance measures, f -divergence BID32 and the Integral Probability Metric (IPM) BID31. In our experiments, we consider the following four metrics that are commonly used to train GANs. Below, ϕ represents the parameters of the discriminator network and θ represents the parameters of the generator network. Training a standard GAN corresponds to minimizing the following BID9: DISPLAYFORM0 where p(z) is the prior distribution of the generative network and G θ (z) is a differentiable function from z to the data space represented by a neural network with parameter θ. D ϕ is trained with a sigmoid activation function, thus its output is guaranteed to be positive. A Least-Squares GAN corresponds to training with a Pearson χ 2 divergence BID28: DISPLAYFORM0 Following BID28, we set a = 0 and b = 1 when training D ϕ. The maximum mean discrepancy metric considers the largest difference in the expectations over a unit ball of RKHS H, DISPLAYFORM0 where H is the RKHS with kernel k(·, ·) BID11. In this case, we do not need to train a discriminator D ϕ to evaluate our metric. Improved Wasserstein Distance (IW) proposed the use of the dual representation of the Wasserstein distance BID42 ) for training GANs. The Wasserstein distance is an IPM which considers the 1-Lipschitz function class ϕ: DISPLAYFORM1 Note that IW and MMD BID39 were recently proposed to evaluate GANs, but have not been compared before. DISPLAYFORM2 Initialize critic network parameter ϕ.3: DISPLAYFORM3 Sample data points from X, {x m} ∼ X tr. Sample points from generative model, {s m} ∼ G θ. ϕ ← ϕ + η∇ ϕ J({x m}, {s m}; ϕ). Sample points from generative model, {s m} ∼ G θ. return J(ϕ, X te, {s m}). The goals in our experiments are two-fold. First, we wanted to evaluate the metrics we proposed for evaluating GANs. Second, we wanted to use these metrics to evaluate GAN frameworks and architectures. In particular, we evaluated how size of the discriminator and generator networks affected performance, and the sensitivity of each algorithm to training data set size. GAN frameworks. We conducted our experiments on three types of GANs: Deep Convolutional Generative Adversarial Networks (DCGAN), Least-Squares GANs (LS-DCGAN), and Wasserstein GANs (W-DCGAN). Note that to not confuse the test metric names with the GAN frameworks we evaluated, we use different abbreviations. GC is the original GAN criterion, which is used to train DCGANs. The LS criterion is used to train the LS-DCGAN, and the IW is used to train the W-DCGAN.Evaluation criteria. We evaluated these three families of GANs with six metrics. We compared our four proposed metrics to the two most commonly used metrics for evaluating GANs, the IS and FID.Because the optimization of a discriminator is required both during training and test time, we will call the discriminator learned for evaluaton of our metrics the critic, in order to not confuse the two discriminators. We also compared these metrics to human perception, and had three volunteers evaluate and compare sets of images, either from the training data set or generated from different GAN frameworks during training. Data sets. In our experiments, we considered the MNIST , CIFAR10, LSUN Bedroom, and Fashion MNIST datasets. MNIST consists of 60,000 training and 10,000 test images with a size of 28 × 28 pixels, containing handwritten digits from the classes 0 to 9. From the 60,000 training examples, we set aside 10,000 as validation examples to tune various hyper-parameters. Similarly, FashionMNIST consists exactly the same number of training and test examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. The CIFAR10 dataset 1 consists of images with a size of 32 × 32 × 3 pixels, with ten different classes of objects. We used 45,000, 5,000, and 10,000 examples as training, validation, and test data, respectively. The LSUN Bedroom dataset consists of images with a size of 64×64 pixels, depicting various bedrooms. From the 3,033,342 images, we used 90,000 images as training data and 90,000 images as validation data. The learning rate was selected from discrete ranges and chosen based on a held-out validation set. Hyperparameters. TAB0 in the Appendix shows the learning rates and the convolutional kernel sizes that were used for each experiment. The architecture of each network is presented in the Appendix in Figure 10. Additionally, we used exponential-mean-square kernels with several different sigma values for MMD. A pre-trained logistic regression and pre-trained residual network were used for IS and FID on the MNIST and CIFAR10 datasets, respectively. For every experiment, we retrained 10 times with different random seeds, and report the mean and standard deviation. The log-likelihood measurement is the most commonly used metric for generative models. We measured the log-likelihood using AIS 2 on GANs is strange, as shown in Figure 1. We measured the log-likelihood of the DCGAN on MNIST with three different variances, σ 2 = 0.01, 0.025, and 0.05. The figure illustrates that the log-likelihood curve over the training epochs varies substantially depending on the variance, which indicates that the fixed Gaussian observable model might not be the ideal assumption for GANs. Moreover, we observe a high log-likelihood at the beginning of training, followed by a drop in likelihood, which then returns to the high value. The IS and MMD metrics do not require training a critic. It was easy to find samples for which IS and MMD scores did not match their visual quality. For example, Figure 2 shows samples generated by a DCGAN when it failed to train properly. Even though the failed DCGAN samples are much darker than the samples on the right, the IS for the left samples is higher/better than for the right samples. As the Imagenet-trained network is likely trained to be somewhat invariant to overall intensity, this issue is to be expected. A failure case for MMD is shown in FIG2. The samples on the right are dark, like the previous examples, but still textually recognizable, whereas the samples on the left are totally meaningless. However, MMD gives lower/worse distances to the left samples. The average intensity of the pixels of the left samples are closer to that for the training data, suggesting that MMD is overly sensitive to image intensity. Thus, IS is under-sensitive to image intensity, while MMD if oversensitive to it. In Section 4.2.1, we conduct more systematic experiments by measuring the correlation between these metrics to human perceptual scores. To both compare the metrics as well as different GAN frameworks, we evaluated the six metrics on different GAN frameworks. TAB1, and 4 present the on MNIST, CIFAR10, and LSUN respectively. As each type of GAN was trained using one of our proposed metrics, we investigated whether the metric favors samples from the model trained using the same metric. Interestingly, we do not see this behavior, and our proposed metrics agree on which GAN framework produces samples closest to the test data set. Every metric, except for MMD, showed that LS-DCGAN performed best for MNIST and CIFAR10, while W-DCGAN performed best for LSUN. As discussed below, we found DCGAN to be unstable to train, and thus excluded GC as a metric for experiments except for this first data set. For Fashion-MNIST, FID's ranking disagreed with IW and LS.We observed similar for a range of different critic CNN architectures (number of feature maps in each convolutional layer):,,, and (see Supp. FIG0).We evaluated a larger variety of GAN frameworks using pre-trained GANs downloaded from (pyt). In particular, we evaluated on EBGAN, BEGAN BID4, W-DCGAN GP BID12, and DRAGAN BID23. TAB4 presents the evaluation . Critic architectures were selected to match those of these pre-trained GANs. For both MNIST and FashionMNIST, the three metrics are consistent and they rank DRAGAN the highest, followed by LS-DCGAN and DCGAN.The standard deviations for the IW distance are higher than for LS divergence. We computed the Wilcoxon rank sum in order to test that whether medians of the distributions of distances are the same for DCGAN, LS-DCGAN, and W-DCGAN. We found that the different GAN frameworks have significantly different performance according to the LS-GAN criterion, but not according to the IW criterion (p < .05, Wilcoxon rank-sum test). Thus LS is more sensitive than IW.We evaluated the consistency of the metrics with respect to the size of validation set. We trained our three GAN frameworks for 100 epochs with training 90,000 examples from the LSUN Bedroom dataset. We then trained LS and IW critics using both 300 and 90,000 validation examples. We looked at how often the critic trained with 300 examples agreed with that trained with 90,000 examples. The LS critics agreed 88% of the time, while the IW critics agreed only 55% of the time (slightly better than chance). Thus, LS is more robust to validation data set size. Another advantage is that measuring the LS distance is faster than measuring the IW distance, as estimating IW involves regularizing with a gradient penalty time BID12. Computing the gradient penalty term and tuning its regularization coefficient requires extra computational time. As mentioned above, we found training a critic using the GC criterion (corresponding to a DCGAN) to be unstable. It has previously been speculated that this is the case because the support of the data and model distributions possibly becoming disjoint, and the Hessian of the GAN objective being non-Hermitian BID29. LS-DCGAN and W-DCGAN are proposed to address this by providing non-saturating gradients. We also found DCGAN to be difficult to train, and thus only report using the corresponding criterion GC for MNIST. Note that this is different than training a discriminator as part of standard GAN training because we are training from a random initialization, not from the previous version of the discriminator. Our experience was that the LS-DCGAN was the simplest and most stable model to train. We visualized the 2D subspace of the loss surface of the GANs in Supp. Fig. 29. Here, we took the parameters of three trained models (corresponds to red vertices in the figure) and applied barycentric interpolation with respect to three parameters (see details from BID20). DCGAN surfaces have much sharper slopes when compared to the LS-DCGAN and W-DCGAN, and LS-DCGAN has the most gentle surfaces. In what follows, we show that this geometric view is consistent with our finding that LS-DCGAN is the easiest and the most stable to train. We compared the LS, IW, MMD, and IS metrics to human perception for the CIFAR10 dataset. To accomplish this, we asked five volunteers to choose which of two sets of 100 samples, each generated using a different generator, looked most realistic. Before surveying, the volunteers were trained to choose between real samples from CIFAR10 and samples generated by a GAN. Supp. FIG1 displays the user interface for the participants, and Supp. FIG2 shows the fraction of labels that the volunteers agreed upon. TAB5 ) presents the fraction of pairs for which each metric agrees with humans (higher is better). IW has a slight edge over LS, and both outperform IS and MMD. In Figure 3, we show examples in which all humans agree and metrics disagrees with human perception. All such examples are shown in Supp. Fig. 21 Figure 3: Pairs of generated image sets for which human perception and metrics disagree. Here, we selected one such example for each metric for which the difference in that metric's scores was high. For each pair, humans perceived the set of images on the left to be more realistic than those on the right, while the metric predicted the opposite. Below each pair of images, we indicate the metric's score for the left and right image sets. Several works have demonstrated an improvement in performance by enlarging deep network architectures BID24 BID37 BID13 BID16. Here, we investigate performance changes with respect to the width and depth of the networks. First, we trained three GANs with varying numbers of feature map sizes, as shown in TAB8 (a-d).Note that we double the number of feature maps in TAB8 for both the discriminators and generators. (a) Samples from (e) in TAB8, MMD= 0.03, IS= 5.11(b) Samples from (f) in TAB8, MMD= 0.49, IS= 6.15 In FIG1, the performance of the LS score increases logarithmically as the number of feature maps is doubled. A similar behaviour is observed in other metrics as well (see S.M. FIG4). We then analyzed the importance of size in the discriminative and generative networks. We considered two extreme feature map sizes, where we choose a small and large number of feature maps for the generator and discriminator, and vice versa (see label (e) and (f) in TAB8, and are shown in TAB7. For LS-DCGAN, it can be seen that a large number of feature maps for the discriminator has a better score than a large number of feature maps for the generator. This can also be qualitatively verified by looking at the samples from architectures (a), (e), (f), and (d) in FIG4. For W-DCGAN, we observe the agreement between the LS and IW metric and conflict with MMD and IS. When we look at the samples from the W-DCGAN in FIG2, it is clear that the model with a larger number of feature maps in the discriminator should get a better score; this is another example of false intuition propagated by MMD and IS. One interesting observation is that when we compare the score and samples from architecture (a) and (e) from TAB8, architecture (a) is much better than (e) (see FIG4). This demonstrates that having a large generator and small discriminator is worse than having a small architecture for both networks. Overall, we found that having a larger generator than discriminator does not give good , and that it is more desirable to have a larger discriminator than generator. Similar were also observed for MNIST, as shown in S.M. Figure 20. This somewhat supports the theoretical from BID2, where the generator capacity needs to be modulated in order for approximately pure equilibrium to exist for GANs. Lastly, we experimented with how performance changes with respect to the dimension of the noise vectors. The source of the sample starts by transforming a noise vector into a meaningful image. It is unclear how the size of noise affects the ability of the generator to generate a meaningful image. Che et al. FORMULA0 TAB8 )."Small" and "large" number of filters for discriminator and generator respectively (ref.(e) in TAB8 )."Large" and "small" number of filters for discriminator and generator respectively (ref.(f) in TAB8 )."Large" number of filters for both discriminator and generator (ref. (d) in TAB8 ). for DCGAN. Our experiments show that this depends on the model. Given a fixed size architecture (d) from TAB8, we observed the performance of LS-DCGAN and W-DCGAN by varying the size of noise vector z. TAB9 illustrates that LS-DCGAN gives the best score with a noise dimension of 50 and W-DCGAN gives best score with a noise dimension of 150 for both IW and LS. The outcome of LS-DCGAN is consistent with the in BID5. It is possible that this occurs because both models fall into the category of f -divergences, whereas the W-DCGAN behaves differently because its metric falls under a different category, the Integral Probability Metric. In practice, we alternate between updating the discriminator and generator, and yet this is not guaranteed to give the same as the solution to the min-max problem in Equation 2. Hence, the update ratio can influence the performance of GANs. We experimented with three different update ratios, 5: 1, 1: 1, and 1: 5, with respect to the discriminator and generator update. We applied these ratios to both the MNIST and CIFAR10 datasets on all models. FIG5 presents the LS scores on both MNIST and CIFAR10 and this is consistent with the IW metric as well (see S.M. FIG2). However, we did not find that any one update ratio was superior over others between the two datasets. For CIFAR10, the 1: 1 update ratio worked best for all models, and for MNIST, different ratios worked better for different models. Hence, we conclude that number of update ratios for each model needs to be dynamically tuned. The corresponding samples from the models trained by different update ratios are shown in S.M. Figure In practice, DCGANs are known to be unstable, and the generator tends to suffer as the discriminator gets better due to disjoint support between the data and generator distributions BID9 Here, we explore the sensitivity of three different kinds of GANs with respect to the number of training examples. We have trained GANs with 10,000, 20,000, 30,000, 40,000, and 45,000 examples on CIFAR10. FIG6 shows that the LS score curve of DCGAN grows quite slowly when compared to W-DCGAN and LS-DCGAN. The three GANs have a relatively similar loss when they are trained with 10,000 training examples. However, the DCGAN only gained 0.0124 ± 0.00127 by increasing from 10,000 to 40,000 training examples, whereas the performance of W-DCGAN and LS-DCGAN improved by 0.03016 ± 0.00469 and 0.0444 ± 0.0033, respectively. Thus, we empirically observe that W-DCGAN and LS-DCGAN have faster performance increases than a DCGAN as the number of training examples grows. In this paper, we proposed to use four well-known distance functions as an evaluation metrics, and empirically investigated the DCGAN, W-DCGAN, and LS-DCGAN families under these metrics. Previously, these models were compared based on visual assessment of sample quality and difficulty of training. In our experiments, we showed that there are performance differences in terms of average experiments, but that some are not statistically significant. Moreover, we thoroughly analyzed the performance of GANs under different hyper-parameter settings. There are still several types of GANs that need to be evaluated, such as GRAN BID18, IW-DCGAN BID12, BEGAN BID4, MMDGAN, and CramerGAN . We hope to evaluate all of these models under this framework and thoroughly analyze them in the future. Moreover, there has been an investigation into taking ensemble approaches to GANs, such as Generative Adversarial Parallelization BID19. Ensemble approaches have been empirically shown to work well in many domains of research, so it would be interesting to find out whether ensembles can also help in min-max problems. Alternatively, we can also try to evaluate other log-likelihood-based models like NVIL BID30, VAE BID22, DVAE BID17, DRAW BID10, RBMs BID15 BID35, , etc. Model evaluation is an important and complex topic. Model selection, model design, and even research direction can change depending on the evaluation metric. Thus, we need to continuously explore different metrics and rigorously evaluate new models. In this paper, we considered four distance metrics that belong to two class of metrics, φ-divergence and IPMs. BID38 have shown that the optimal risk function is associated with a binary classifier with P and Q distributions conditioned on a class when the discriminant function is restricted to certain F (Theorem 17 from BID38).Let the optimal risk function be: DISPLAYFORM0 where F is the set of discriminant functions (classifier), y ∈ −1, 1, and L is the loss function. By following derivation, we can see that the optimal risk function becomes IPM: DISPLAYFORM1 where DISPLAYFORM2 The second equality is derived by separating the loss for class 1 and class 0. The third equality is from the way how we chose L(1,f(x)) and L(0,f(x)). The last equality is derived from that fact that F is symmetric around zero (f ∈ F => −f ∈ F). Hence, this shows that with appropriately choosing L, MMD and Wasserstein distance can be understood as the optimal L-risk associated with binary classifier with specific set of F functions. For example, Wasserstein distance and MMD distances are equivalent to the optimal risk function with 1-Lipschitz classifiers and a RKHS classifier with an unit length. We trained two critics on training data and validation data, respectively, and evaluated on test data from both critics. We trained six GANs (GAN, LS-DCGAN, W-DCGAN GP, DRAGAN, BEGAN, EBGAN) on MNIST and FashionMNIST. We trained these GANs with 50,000 training examples. At test time, we used 10,000 training and 10,000 validation examples for training the critics, and evaluated on 10,000 test examples. Here, we present the test scores from the critics trained on training and validation data. The are shown in Table??. Note that we also have the IW and FID evaluation on these models in the paper. For FashionMNIST, we find that test scores with a critic trained on training and validation data are very close. Hence, we do not see any indication of overfitting. On the other hand, there are gaps between the scores for the MNIST dataset and the test scores from critics trained on the validation set. which gives better performance than the ones that are trained on the training set. FIG1: The participants are trained by selecting between random samples generated by GANs versus samples from data distribution. They get a positive reward if they selected the data samples and a negative reward if they select the samples from the model. After enough training, they choose the better group of samples among two randomly select set of samples. (a) Samples from (e) in TAB8, MMD= 0.03, IS= 5.11(b) Samples from (f) in TAB8,, MMD= 0.49, IS= 6.15 TAB8 ).(b) "Small" and "large" number of filters for discriminator and generator respectively (ref.(e) in TAB8 ).(c) "Large" and "small" number of filters for discriminator and generator respectively (ref.(f) in TAB8 ).(d) "Large" number of filters for both discriminator and generator (ref. (d) in TAB8 ). TAB8 ).(b) "Small" and "large" number of filters for discriminator and generator respectively (ref.(e) in TAB8 ).(c) "Large" and "small" number of filters for discriminator and generator respectively (ref.(f) in TAB8 ).(d) "Large" number of filters for both discriminator and generator (ref. (d) in TAB8 ). Figure 31: The training curve of critics to show that the training curve converges. IW distance curves in (a) increase because we used linear output unit for the critic network (by design choice). This can be simply bounded by adding a sigmoid at the output of the critic network.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An empirical evaluation on generative adversarial networks

Represent the TLDR for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.