Question:
represent the text to find a very short summary (1-2 sentences max)

Examples:

Given This paper presents preliminary ideas of our work for auto- mated learning of Hierarchical Goal Networks in nondeter- ministic domains. We are currently implementing the ideas expressed in this paper. Many domains are amenable to hierarchical problem-solving representations whereby complex problems are represented and solved at different levels of abstraction. Examples include some navigation tasks where hierarchical A* has been shown to be a natural solution solving the navigation problem over different levels of abstraction BID29 BID66; dividing a reinforcement learning task into subtasks where policy control is learned for subproblems and combined to form a solution for the overall problem BID9 BID10 BID12; abstraction planning, where concrete problems are transformed into abstract problem formulations, these abstract problems are solved as abstract plans, and in turn these abstract plans are refined into concrete solutions BID31 BID4; and hierarchical task network (HTN) planning where complex tasks are recursively decomposed into simpler tasks BID8 BID68 BID15 BID43. These paradigms have in common a divideand-conquer method to problem solving that is amenable to stratified representation of the subproblems. Among the various formalisms, HTN planning has been a recurrent research focus over the years. An HTN planner formulates a plan using actions and HTN methods. The latter describe how and when to reduce complex tasks into simpler subtasks. HTN methods are used to recursively decompose tasks until so-called primitive tasks are reached corresponding to actions that can be performed directly in the world. The HTN planners SHOP and SHOP2 have routinely demonstrated impressive gains in performance (runtime and otherwise) over standard planners. The primary reason for these performance gains is because of the capability of HTN planners to exploit domain-specific knowledge BID67. HTNs provide a natuCopyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. ral knowledge-modeling representation for many domains, including military planning BID38 BID40, strategy formulation in computer games BID23 BID22, manufacturing processes BID47 BID61, project planning BID62 BID63, story-telling BID6, web service composition, and UAV planning BID20 Despite these successes, HTN planning suffers from a representational flaw centered around the notion of task. A task is informally defined as a description of an activity to be performed (e.g., find the location of robot r15) (e.g., the task "dislodge red team from Magan hill" in some adversarial game) and syntactically represented as a logical atom (e.g., (locate r15)). (e.g., "(dislodge redteam Magan)"). Beyond this syntax, there is no explicit semantics of what tasks actually mean in HTN representations. HTN planners obviate this issue by requiring that a complete collection of tasks and methods is given, one that decomposes every complex task in every plausible situation. However, the knowledge engineering effort of creating a complete set of tasks and methods can be significant BID17. Furthermore, researchers have pointed out that the lack of tasks' semantics make using HTNs problematic for execution monitoring problems BID13 BID14 ). Unlike goals, which are conditions that can be evaluated against the current state of the world, tasks have no explicit semantics other than decomposing them using methods. For example, suppose that a team of robots is trying to locate r15 and, using HTN planning, it generates a plan calling for the different robots to ascertain r15's location. While executing the plan generate a complex plan in a gaming task to dislodge red team from Magan hill, the HTN planner might set a complex plan to cutoff access to Magan, surround it, weaken the defenders with artillery fire and then proceed to assault it. If sometime while executing the plan, the opponent abandons the hill, the plan would continue to be executed despite the fact that the task is already achieved. This is due to the lack of task semantics, so their fulfillment cannot be checked against the current state; instead their fulfillment is only guaranteed when the execution of the generated plans is completed. Hierarchical Goal Networks (HGNs) solve these limitations by representing goals (not tasks) at all echelons of the hierarchy BID56. Hence, goal fulfillment can be directly checked against the current state. In particular, even when a goal g is decomposed into other goals (i.e., in HGN, HGN methods decompose goals into subgoals), the question if the goal is achieved can be answered directly by checking if it is valid in the current state. So in the previous example, when the opponent abandons the hill, an agent executing the plan knows this goal has been achieved regardless of how far it got into executing the said plan. Another advantage of HGNs is that it relaxes the complete domain requirement of HTN planning BID57; in HTN planning a complete set of HTN methods for each task is needed to generate plans. Even if the HGN methods are incomplete, it is still possible to generate solution plans by falling back to standard planning techniques such as heuristic planning BID24 to achieve any open goals. Nevertheless, having a collection of well-crafted HGN methods can lead to significant improvement in performance over standard planning techniques BID59.When the HGN domain is complete (i.e., there is no need to revert to standard planning techniques to solve any problem in the domain), its expressiveness is equivalent to Simple Hierarchical Ordered Planning BID59, which is the particular variant of HTN planning used by the widely used SHOP and SHOP2 BID45 ) HTN planners. SHOP requires the user to specify a total order of the tasks; SHOP2 drops this requirement allowing partial-order between the tasks BID44. Both have the same representation capabilities although SHOP2 is usually preferred since it doesn't force the user to provide a total order for the method's subtasks BID44.In this work, we propose the automated learning of HGNs for ND domains extending our previous work on learning HTNs for deterministic domains BID21. While work exists on learning goal hierarchies BID53 BID32 BID49, these works are based on formalisms that have more limited representations than HGNs and in fact predate them. Aside from HGNs, researchers have explored other ways to address the limitation associated with the lack of tasks' semantics. For instance, TMKs (Task-Method-Knowledge models) require not only the tasks and methods to be given but also the semantics of the tasks themselves as (preconditions,effects) pairs BID41 BID42. While this solves the issue with the lack of tasks' semantics it may exacerbate the knowledge engineering requirement of HTNs: the knowledge engineer must not only encode the methods and tasks but also must encode their semantics and ensure that the methods are consistent with the given tasks' semantics. To deal with incomplete HTN domains, researchers have proposed translating the methods into a collection of actions so that standard planning techniques can be used BID1 ). There are two limitations with this approach. First, HTN planning is strictly more expressive than standard planning BID15, hence the translation will be incomplete in many domains. Second, for domains when translating methods into actions is possible, it may in exponentiallymany actions on the number of methods. HGNs are more in line with efforts combining HTN and standard planning approaches BID30 BID17; the main difference is that HGNs eliminate the use of tasks all-together while still preserving the expressiveness of Simple Hierarchical Ordered Planning BID59.The problem of learning hierarchical planning knowledge has been a frequent research subject over the years. For example, ICARUS (Choi and Langley 2005) learns HTN methods by using skills (i.e., abstract definitions of semantics of complex actions) represented as Horn clauses. The crucial step is a teleoreactive process where planning is used to fill gaps in the HTN planning knowledge. For example, if the learned HTN knowledge is able to get a package from an starting location to a location L1 and the HTN knowledge is also able to get the package from a location L2 to its destination, but there is no HTN knowledge on how to get the package from L1 to L2, then an standard planner is used to generate a plan to get the package from L1 to L2 and skills are used to learn new HTN methods from the plan generated to fill the gap on how to get from L1 to L2.Another example is HTN-Maker (Hogg, Muñoz-Avila, and Kuter 2008). HTN-Maker uses task semantics defined as (preconditions,effects) pairs, exactly like TMKs mentioned before, to identify sequences of contiguous actions in the input plan trace where the preconditions and effects are met. Task hierarchies are learned when an action sequence is identified as achieving a task and the action sequence is a sub-sequence of another larger action sequence achieving another task. This includes the special case when the sub-sequence and the sequence achieve the same task. In such a situation recursion is learned. HTN-Maker learns incrementally after each training case is given. HTNLearn BID70 transforms the input traces into a constraint satisfaction problem. Like HTN-Maker, it also assumes (preconditions,effects) as the task semantics to be given as input. HTNLearn process the input traces converting them into constraints. For example, if a literal p is observed before an action a and a is a candidate first sub-task for a method m, then a constraint c is added indicating that p is a precondition of m. These constrains are solved by a MAXSAT solver, which returns the truth value for each constraint. For example, if c is true then p is added as a precondition of m. As a of the MAXSAT process, HTNLearn is not able to converge to a 100% correct domain (the evaluation of HTNLearn computes the error rates in the learned domain).Similar to HTN planning, hierarchical decompositions have been used in hierarchical reinforcement learning BID50 BID10. The hierarchical structure of the reinforcement learning problem is analogous to an instance of the decomposition tree that an HTN planner might generate. Given this hierarchical structure, hierarchical reinforcement learners perform value-function composition for a task based on the value functions learned over its subtasks recursively. However, the possible hierarchical decompositions must be provided in advance. Hierarchical goal networks (HGNs) BID56 are an alternative representation formalism to HTNs. In HGNs, goals, instead of tasks, are decomposed at every level of the hierarchy. HGN methods have the same fo.rm as HTN methods but instead of decomposing a task, they decompose a goal; analogously instead of subtasks, HGN methods have subgoals. If the domain description is incomplete, HGNs can fall back to STRIPS planners to fill gaps in the domain. On the other hand, total-order HGNs are as expressive as totalorder HTNs BID59 and its partial-order variant ) is as expressive as partial-order HTNs.Inductive learning has been used to learn rules indicating goal-subgoal relations in X-learn BID53. This is akin to learning macro-operators BID39 BID5; the learned rules and macro-operators provide search control knowledge to reach the goals more rapidly but they don't add expressibility to standard planning. SOAR learns goal-subgoal relations BID32. It uses as input annotated behavior trace structures, indicating the decisions that led to the plans; this is used to generate a goal-subgoal relations. Another work on learning goal-subgoal relations is reported in BID49 ). It uses case-based learning techniques to store goal-subgoal relations, which are then reused by using similarity metrics. These works assume some form of the input traces, unstructured in BID49 ) and structured in BID32, to be annotated with the subgoals as they are accomplished in the traces. In our proposed work, the input traces are not annotated and, more importantly, we are learning HGNs. Goal regression techniques have been used to generate a plan starting from the goals that must be achieved BID52 BID36. The of goal regression can be seen as a hierarchy recursively generated by indicating for each goal what subgoals must be achieved. The goal-subgoal relations ing from goal regression are a direct consequence of the domain's operators: the goals are effects of the operators and the preconditions are the subgoals. In contrast, in a HGN, the hierarchies of goals represent relations between the HGN methods and are not necessarily implied directly from the actions. Making an analogy with HTN methods, HGN methods capture additional domain-specific knowledge BID45 or generate plans with desirable properties (e.g., taking into account quality considerations) again not explicitly represented in the actions BID27.Work on learning hierarchical plan knowledge is related to learning of context-free grammars (CFGs), which aims at eliciting a finite set of production rules from a finite set of strings BID48 BID55 ). The precise definition of the learning problem varies constraining the ing CFG by, among others, providing a target function (e.g., obtaining a CFG with the minimum number of production rules) or assuming that negative examples (i.e., strings that must not be generated by the CFG) are given. To learn CFGs, algorithms search for production rules that generate the training set (and none of the negative examples when provided). Grammar learning is exploited by the Greedy Structure Hypothesizer (GSH) BID35, which uses probabilistic context-free grammars learning techniques to learn a hierarchical structure of the input plan traces. GSH doesnt learn preconditions since its goals are not to generate the grammars for planning but to reflect users preferences. The difference between learning CFG and learning hierarchical planning knowledge is twofold. First, characters that form a string have no meaning. In contrast, actions in a given plan are defined by their preconditions and effects. This means that plausible strings generated by the grammars may be invalid when viewed as plans. Second, learning HGNs requires not only learning the task decomposition but also the preconditions. This is an important difference: HTNs are strictly more expressive than CFGs BID16. Intuitively, HTNs are akin to context-sensitive grammars in that they constraint when a decomposition can take place. Context-sensitive grammars are also strictly more expressive then CFGs BID60.Finally, as we will see in the next the next section, our proposed work is related to the notion of planning landmarks BID25. Given a planning problem P, defined as a triple (s 0, g,A), indicating the initial state, the goals and the actions respectively, a planning landmark is either an action a ∈ A, or state atom p ∈ s (s is an state, represented as a collection of atoms) that occurs in any solution plan trace solving P. Given the problem description P, planning systems can identify automatically landmarks for P. Planning landmarks have been widely used for automated planning ing in planners such as LAMA BID54 and the HGN planner GoDel BID57 ). We want to learn HGNs for fully observable nondeterministic (FOND) planning BID19 Speck, Ortlieb, and Mattmüller 2015; BID69. In such domains, actions may have multiple outcomes. For example, in the Minecraft simulation, when a character swings a sword to hit a monster, there are two possible outcomes: either the sword hits the monster or the monster parries it and the sword doesn't hit anything. As discussed before, HTN learners require the tasks semantics to be given either as Horn clauses defining the tasks or as (preconditions,effects) pairs. The latter is used, for example, in the nondeterministic HTN learner ND-HTNMaker, a state-of-the-art HTN learner, to pinpoint locations in the traces where the various tasks are fulfilled. ND-HTNMaker enforces a right recursive structure: exactly one primitive task followed by none or exactly one compound task. The main objective of enforcing this right recursive structure is to deal with nondeterminism: if, for example, the character swings the sword (e.g., a primitive task), the follow-up compound task handles the nondeterminism: one method decomposing a compound task t will simply perform the action to swing at the monster followed by t, thereby ensuring that method can be triggered as many times as needed until the monster is hit (and dies). Other methods decomposing t handle the case when the monster has been dealt with (e.g., a method handling the case when "character next to a dead monster"). This ensures that methods learned by HTN-Maker are provable correct BID26. Correctness can be loosely defined as follows: any solution generated by a sound nondeterministic HTN planner such as ND-SHOP BID33 ) using the learned methods and the nondeterministic actions is also a solution when using the nondeterministic actions (i.e., without the methods).Like in the deterministic case, the inputs will be a collection of actions A and a collection of traces s 0 a 0 s 1 a 1... a n s n+1, where each a i ∈ A. Only this time, any action a i ∈ A may have multiple outcomes; so each occurrence of a i in the input traces will reflect one such outcome. Planning in nondeterministic domains requires to account for all possible outcomes. As such, BID7 proposed a categorization of solutions for nondeterministic domains. It distinguishes between weak, strong cyclic and strong solutions for a problem (s 0, g,A). A solution is represented as a policy π: S → A, a mapping from the possible states in the world S to actions A, indicating for any given state s ∈ S, what action π(s) to take. Given a policy π, an execution trace is any sequence s 0 π(s 0) s 1 π(s 1)... π(s n) s n+1, where s i is an state that can be reached from state s i−1 after applying action π(s i−1).A solution policy π is weak if there exists an execution trace from s 0 to a state satisfying g. Weak solutions guarantee that a goal state can be successfully reached sometimes. For example, in the Minecraft simulation, a policy that assumes a computer-controlled character will always hit any monster it encounters when swinging the sword is considered a weak solution. In particular, this solution would not account for the situation when the monster parries the character's sword attack; e.g., the monster might counter-attack and disable the agent and the agent has not planned what to do in such a situation. Under the fairness assumption, stating that "every action executed infinitely often will exhibit all its effects infinitely often" (D'Ippolito, Rodrıguez, and Sardina 2018), a solution π is either strong cyclic or strong if every terminal state entails the goals and for every state s that the agent might finds itself in after executing π from s 0, there exists an execution trace from the state s to a state satisfying g. The difference is that in strong cyclic solutions the same state might be visited more than once whereas in strong solutions this never happens. For example, a strong cyclic solution might have the character swing the sword against the monster and if the monster parries the attack, the character takes an step back to avoid the monster's counter-attack and step towards the monster while taking another swing at it; this can be repeated as many times as needed until the monster dies. Strong solutions are ideal since they never visit the same state but in some domains they might not exists. For instance, there are no strong solutions in the Minecraft simulation mentioned as the monster can repeatedly parry the character's attacks. The same occurs in the robot navigation domain BID7 ), created to model nondeterminism. In this domain a robot is navigating between offices and when it encounters a closed door for an office it wants to access, the robot will open it. There is an another agent acting in the environment that closes doors at random. So the robot might need to repeatedly execute the action to open the same door. Solving nondeterministic planning problems is difficult because of what has been dubbed the explosion of states as a of the nondeterminism BID19. One demonstrated way to counter this is by adding domain-specific knowledge as described in BID33 ). While the algorithm described is generic for a variety of ways to encode the domain-specific knowledge, it showcases hierarchical planning techniques outperforming an state-of-the-art nondeterministic planner in some domains including the robot navigation domain. The show either speedups of several orders of magnitude or the ability to solve problems of sizes, measured by the number of goals to achieve, previously impossible to solve. Relation to probabilistic domains. In this work we are neither assuming a probability distribution over the possible actions' outcomes to be given nor we aim to learn such a distribution. Once an HGN domain is learned, hierarchical reinforcement learning techniques BID10 can be used to learn a probability distribution over the various possible goal decompositions and exploit the learned distribution during problem solving as done in BID27.We propose to learn bridge atoms and their hierarchical structure with the important constraint that the learned hierarchical structure must encode the domain's nondeterminism in a sound way. For instance, the nondeterministic version of the logistics transportation domain in BID26 ) extends the deterministic version as follows: when loading a package p into vehicle v in a location l there are two possible outcomes: either p is inside v or p is still at l (i.e., the load action failed). Regardless of possibly repeating the same action multiple times, traces will bring the package to the airport, transport it by air to the destination city, and deliver it. So the kinds of decompositions we are aiming to learn should also work on nondeterministic domains; on the other hand a learned hierarchy would be unsound if, for example, it assumes that the load truck action always succeeds and immediately proceeds to deliver the package to an airport. This will lead to weak solutions. To correctly handle nondeterminism, we propose forcing a right-recursive structure on lower echelons of the learned HGNs. This takes care of the nondeterminism and combine well with the higher decompositions. For instance, in the transportation domain we identified a goal g airp, for the package p reaching the airport, identified as a bridge atom, and then have all methods achieving g airp be right recursive; e.g., methods of the form (: method g airp prec (g g airp) <), where g is some intermediate goal such as loading the package into a vehicle. Our aim is the automated learning of HGN methods. This includes learning the goals, the goal-subgoal structure of the HGN methods and their applicability conditions. Specifically, the learning problem can be defined as follows: given a set of actions A and a collection of traces Π generated using actions in A, to obtain a collection of HGN methods. A collection of methods M is correct if given any (initial state, goal) pair (s 0, g), and any solution plan π generated by a sound HGN planner using M and A, π is a correct plan solving the planning problem (s 0, g,A). An HGN method m is a construct of the form (:method head(m) preconditions(m) subgoals(m) <(m)) corresponding to the goal decomposed by m (called the head of m), the preconditions for applying m and the subgoals decomposing head(m). Figure 1 shows an example of an HGN method in the logistics transportation domain BID65. (the question marks indicate variables. It recursively decomposes the goal of delivering ? pack1 into ? loc2 into three subgoals: delivering? pack1 to the airport? airp1 in the same city as its current location? loc1, delivering? pack1 to the airport? airp2 in the same city as the destination location? loc2, and recursively achieve the head goal):Head: Package-delivery Preconditions: (at ?pack ? loc1 ? city1) (airport ? airp1 ? city1) (airport ? airp2 ?city2) (location ? loc2 ? city2) (= ?city1 ?city2) Subgoals: g 1: (package-at ?pack ? airp1) g 2:(package-at ?pack1 ? airp2) g 3:(package-at ?pack ? loc2 ? city2) Constraints: g 1 < g 3, g 2 < g 3 Figure 1: Example of an HGN method in the logistics transportation domain. The question marks indicate variables. The goal achieved by the method is the last subgoal, g 3. It recursively decomposes the goal of delivering? pack into? loc2 into three subgoals: delivering? pack1 to the airport? airp1 in the same city as its current location? loc1, delivering? pack to the airport? airp2 in the same city as the destination location? loc2, and g 3 is to be achieved after g 1 and g 2 are achieved. HGNs planners BID57 BID56 maintain a list G = g 1,..., g n of open goals (i.e., goals to achieve). Planning follows a recursive procedure, starting with π =, choosing the first element, g 1, in G, and either applying an HGN method m decomposing g 1 into m's subgoals g 1,..., g k, concatenating m's subgoals into G (i.e, G = g 1, . . ., g k, g 1, . . ., g n are the new open goals), or applying an action a ∈ A achieving g, appending a to π (i.e., π ← π · a) and removing g from G. In either case it will check if the preconditions of m (respectively, a) are satisfied in the current state. When a is applied, the current state is transformed in the usual way BID18. When G = ∅, π is returned. HGN planners extend this basic procedure to allow the use of standard planning techniques to achieve open goals and to enable a partial ordering between the methods' subgoals. the planner picks the first goal in G without predecessors. For example, in Figure 1, the user may define the constraints: g 1 < g 3, g 2 < g 3, and the planner instead of always picking the first subgoal in G, it picks the first subgoal without predecessors. We propose transforming the problem of identifying the goals and learning their hierarchical relation into the problem of finding relations between word embeddings extracted from text. Specifically, we propose viewing the collection of input traces Π as text: each plan trace π = s 0 a 0 s 1 a 1... a n s n+1 is viewed as a sentence w 1 w 2... w m; each action a i and each atom in s j is viewed as a word w k in the sentence. The order of the plan elements in each trace is preserved (we use the term plan element to refer to both atoms and actions): the word w j = a i appears before the word w j = p, for every p ∈ s i+1. In turn, every w j appears before w j = a i+1.Word embeddings are vectors representing words in a multi-dimensional vector space BID3 BID2. There are a number of algorithms to do this translation BID37 BID51. They have in common that they represent vector similarity based on the cooccurrence of words in the text. That is, words that tend to occur near one another will have similar vector representations. In our preliminary work we used Word2Vec BID37 ) (i.e., Word-Neighboring Word), a widely used algorithm for generating word embeddings. Word2Vec uses a shallow neural network, consisting of a single hidden layer, to compute these vector representation; it computes a context window W consisting of k contiguous words and trains the network using each word w ∈ W (i.e., W is w's context). The window W is "moved" one word at the time through the text further training the network each time. Training is repeated with windows of size i = {1, 2, . . . k}. For this reason, Word2Vec is said to use "dynamic windows". In Word2Vec, similarity is computed with the cosine similarity, sim C, because it measures how close is the orientation of the ing vectors, which are distributed in such a way that words frequently co-occurring in the context windows have similar orientation whereas those that co-occur less frequently will have a dissimilar orientation. There are two particularities of the change of representation from plan elements to word embeddings that is particularly suitable for our purposes: first the procedure is unsupervised. This means in our case that we do not have to annotate the traces with additional information such as where the goals are been achieved in the traces. Second, vector representations are generated based on the context in which they occur (e.g., the dynamic window W in Word2Vec). In our case, the vector representations of the plan elements will be generated based on their proximity to other plan elements in the traces. These vectors can be clustered together into plan elements that are close to one another. Our working hypothesis, supported by previous work BID21, is that what we call bridge atoms, are ideal candidates for goals. Given two clusters of plan element embeddings, A and B, a bridge atom, bridge AB, is an atom in either A or B that is most similar to the plan elements in the other set. Establishing a bridge atom hierarchy is a recursive process that first requires calculating the bridge atom of a corpus, splitting each text around the bridge atom so that each text in the corpus becomes two new texts, before and after the bridge atom, and then repeating the procedure on the ing sub-corpora. The procedure for find a bridge atom for a corpora is as follows. We train a Word2Vec model on the corpus to determine the word vectors and cluster them with Hierarchical Agglomerative Clustering. Currently we limit the number of clusters to two, although later research may explore how to determine the number of clusters from the structure of the traces. We determine the cosine distance of each atom in a cluster to each atom in the other and average them together for each atom, selecting the word with the shortest average distance, DISPLAYFORM0 where dist C is the cosine distance between the vector representations of two atoms. If an action is selected as the bridge atom, we instead use in its place the atom describing one of its goals. As previously stated, by splitting each trace around the bridge atom, we can form two new sub-corpora, one from the section of each trace before the bridge atom and one from the section after the bridge atom. Then we recursively perform the procedure for bridge atom selection on each new corpora, keeping track of the hierarchical relationship of each subcorpora to the other corpora. If during the division process, a section of a trace becomes shorter than some threshold, we discard it from the sub-corpora. Progress along any branch of recursion halts once there are insufficient traces in a subcorpus for training. We use the hierarchy of bridge atoms as a guide for building a set of hierarchical methods. At the lowest level of division are single-action or short multi-action sections of the traces. Each of these sections will become a method with a single goal (an effect of an action) or a method with multiple goals (one for each of the actions). Each of these methods have two subgoals: one for the subsection of trace before a bridge atom and another one for the trace after that bridge atom. Each action is annotated with its preconditions. The preconditions of a method can be extrapolated from the preconditions of the actions into which it decomposes by regressing over the actions of that section of the plan trace in reverse, collecting the action preconditions and removing from the preconditions any atom which is in the effects of chronologically-preceding action. We are using a variant of the Pyhop HTN planner (https: //bitbucket.org/dananau/pyhop). Our variant introduces nondeterminism in the actions and generates solution policies as described in BID33.Our experiments use a nondeterministic variant of the logistics domain BID64 ). In the domain, packages must be relocated from one location to another. Trucks transport packages within cities, and airplanes transport packages between cities via airports. Nondeterminism is introduced via the load and unload operators, which have two outcomes, success (the package is loaded onto/unload from the specified vehicle) or failure (the package does not move from its original location). We have also added rockets that transport packages between cities on different planets via launchpads. All traces demonstrate a plan for achieving the same goal, the relocation of a package from a location in one city on the starting planet to a location in a city on the destination planet. To ensure that Word2Vec can identify common bridge atoms across the corpus, the package and each location must have the same name in all traces. Although Word2Vec typically works best on a corpus of thousands of texts or more, we are able to learn reasonable bridge atoms from hundreds of texts by increasing the number of epochs and lowering learning rate. For our problem design, a reasonable first bridge atom is one that involves the package and a rocket or launchpad, as transporting the package from the start planet to the destination planet marks the halfway point in the traces. From a corpus of 700 traces, with 1000 epochs and a learning rate of 0.00025, our first bridge atom is the action unload(package, rocket).Because word embeddings are sensitive to word context, the trace structure influences the bridge atom hierarchy. Which atoms are included in the trace and where they are included is important. We are experimenting with two different variants of state expression within traces. In one variant, we list each action preceded by its deletelist and followed by its addlist. If an atom occurs in the addlist of one action and the deletelist of the subsequent action, that atom will only appear in the addlist of the first action. In another variant, we list actions preceded by their preconditions and followed by their effects. In both variants, atoms are listed alphabetically. it matches with Learning HGNs, ND domains but not with This paper introduces partial grounding to tackle the problem that arises when the full grounding process, i.e., the translation of a PDDL input task into a ground representation like STRIPS, is infeasible due to memory or time constraints.

Question: Neural program embeddings have shown much promise recently for a variety of program analysis tasks, including program synthesis, program repair, code completion, and fault localization. However, most existing program embeddings are based on syntactic features of programs, such as token sequences or abstract syntax trees. Unlike images and text, a program has well-deﬁned semantics that can be difﬁcult to capture by only considering its syntax (i.e. syntactically similar programs can exhibit vastly different run-time behavior), which makes syntax-based program embeddings fundamentally limited. We propose a novel semantic program embedding that is learned from program execution traces. Our key insight is that program states expressed as sequential tuples of live variable values not only capture program semantics more precisely, but also offer a more natural ﬁt for Recurrent Neural Networks to model. We evaluate different syntactic and semantic program embeddings on the task of classifying the types of errors that students make in their submissions to an introductory programming class and on the CodeHunt education platform. Our evaluation show that the semantic program embeddings signiﬁcantly outperform the syntactic program embeddings based on token sequences and abstract syntax trees. In addition, we augment a search-based program repair system with predictions made from our semantic embedding and demonstrate signiﬁcantly improved search efﬁciency. Recent breakthroughs in deep learning techniques for computer vision and natural language processing have led to a growing interest in their applications in programming languages and software engineering. Several well-explored areas include program classification, similarity detection, program repair, and program synthesis. One of the key steps in using neural networks for such tasks is to design suitable program representations for the networks to exploit. Most existing approaches in the neural program analysis literature have used syntax-based program representations. BID6 proposed a convolutional neural network over abstract syntax trees (ASTs) as the program representation to classify programs based on their functionalities and detecting different sorting routines. DeepFix BID4, SynFix BID1, and sk p BID9 are recent neural program repair techniques for correcting errors in student programs for MOOC assignments, and they all represent programs as sequences of tokens. Even program synthesis techniques that generate programs as output, such as RobustFill BID3, also adopt a token-based program representation for the output decoder. The only exception is BID8, which introduces a novel perspective of representing programs using input-output pairs. However, such representations are too coarse-grained to accurately capture program properties -programs with the same input-output behavior may have very different syntactic characteristics. Consequently, the embeddings learned from input-output pairs are not precise enough for many program analysis tasks. Although these pioneering efforts have made significant contributions to bridge the gap between deep learning techniques and program analysis tasks, syntax-based program representations are fundamentally limited due to the enormous gap between program syntax (i.e. static expression) and Bubble Insertion Figure 1: Bubble sort and insertion sort (code highlighted in shadow box are the only syntactic differences between the two algorithms). Their execution traces for the input vector A = are displayed on the right, where, for brevity, only values for variable A are shown. semantics (i.e. dynamic execution). This gap can be illustrated as follows. First, when a program is executed at runtime, its statements are almost never interpreted in the order in which the corresponding token sequence is presented to the deep learning models (the only exception being straightline programs, i.e., ones without any control-flow statements). For example, a conditional statement only executes one branch each time, but its token sequence is expressed sequentially as multiple branches. Similarly, when iterating over a looping structure at runtime, it is unclear in which order any two tokens are executed when considering different loop iterations. Second, program dependency (i.e. data and control) is not exploited in token sequences and ASTs despite its essential role in defining program semantics. FIG0 shows an example using a simple max function. On line 8, the assignment statement means variable max val is data-dependent on item. In addition, the execution of this statement depends on the evaluation of the if condition on line 7, i.e., max val is also control-dependent on item as well as itself. Third, from a pure program analysis standpoint, the gap between program syntax and semantics is manifested in that similar program syntax may lead to vastly different program semantics. For example, consider the two sorting functions shown in Figure 1. Both functions sort the array via two nested loops, compare the current element to its successor, and swap them if the order is incorrect. However, the two functions implement different algorithms, namely Bubble Sort and Insertion Sort. Therefore minor syntactic discrepancies can lead to significant semantic differences. This intrinsic weakness will be inherited by any deep learning technique that adopts a syntax-based program representation. We have evaluated our dynamic program embeddings in the context of automated program repair. In particular, we use the program embeddings to classify the type of mistakes students made to their programming assignments based on a set of common error patterns (described in the appendix). The dataset for the experiments consists of the programming submissions made to Module 2 assignment in Microsoft-DEV204.1X and two additional problems from the Microsoft CodeHunt platform. The show that our dynamic embeddings significantly outperform syntax-based program embeddings, including those trained on token sequences and abstract syntax trees. In addition, we show that our dynamic embeddings can be leveraged to significantly improve the efficiency of a searchbased program corrector SARFGEN 1 BID13 ) (the algorithm is presented in the appendix). More importantly, we believe that our dynamic program embeddings can be useful for many other program analysis tasks, such as program synthesis, fault localization, and similarity detection. To summarize, the main contributions of this paper are: we show the fundamental limitation of representing programs using syntax-level features; we propose dynamic program embeddings learned from runtime execution traces to overcome key issues with syntactic program representations; we evaluate our dynamic program embeddings for predicting common mistake patterns students make in program assignments, and show that the dynamic program embeddings outperform state-of-the-art syntactic program embeddings; and we show how the dynamic program embeddings can be utilized to improve an existing production program repair system. This section briefly reviews dynamic program analysis BID0, an influential program analysis technique that lays the foundation for constructing our new program embeddings. Unlike static analysis BID7, i.e., the analysis of program source code, dynamic analysis focuses on program executions. An execution is modeled by a set of atomic actions, or events, organized as a trace (or event history). For simplicity, this paper considers sequential executions only (as opposed to parallel executions) which lead to a single sequence of events, specifically, the executions of statements in the program. Detailed information about executions is often not readily available, and separate mechanisms are needed to capture the tracing information. An often adopted approach is to instrument a program's source code (i.e., by adding additional monitoring code) to record the execution of statements of interest. In particular, those inserted instrumentation statements act as a monitoring window through which the values of variables are inspected. This instrumentation process can occur in a fully automated manner, e.g., a common approach is to traverse a program's abstract syntax tree and insert "write" statements right after each program statement that causes a side-effect (i.e., changing the values of some variables).Consider the two sorting algorithms depicted in Figure 1. If we assume A to be the only variable of interest and subject to monitoring, we can instrument the two algorithms with Console. WriteLine(A) after each program location in the code whenever A is modified 2 (i.e. the lines marked by comments). Given the input vector A =, the execution traces of the two sorting routines are shown on the right in Figure 1.One of the key benefits of dynamic analysis is its ability to easily and precisely identify relevant parts of the program that affect execution behavior. As shown in the example above, despite the very similar program syntax of bubble sort and insertion sort, dynamic analysis is able to discover their distinct program semantics by exposing their execution traces. Since understanding program semantics is a central issue in program analysis, dynamic analysis has seen remarkable success over the past several decades and has ed in many successful program analysis tools such as debuggers, profilers, monitors, or explanation generators. We now present an overview of our approach. Given a program and the execution traces extracted for all its variables, we introduce three neural network models to learn dynamic program embeddings. To demonstrate the utility of these embeddings, we apply them to predict common error patterns (detailed in Section 5) that students make in their submissions to an online introductory programming course. Variable Trace Embedding As shown in TAB1, each row denotes a new program point where a variable gets updated. 3 The entire variable trace consists of those variable values at all program points. As a subsequent step, we split the complete trace into a list of sub-traces (one for each variable). We use one single RNN to encode each sub-trace independently and then perform max pooling on the final states of the same RNN to obtain the program embedding. Finally, we add a one layer softmax regression to make the predictions. The entire workflow is show in FIG1.State Trace Embedding Because each variable trace is handled individually in the previous approach, variable dependencies/interactions are not precisely captured. To address this issue, we propose the state trace embedding. As depicted in TAB1, each program point l introduces a new program state expressed by the latest variable valuations at l. The entire state trace is a sequence of program states. To learn the state trace embedding, we first use one RNN to encode each program state (i.e., a tuple of values) and feed the ing RNN states as a sequence to another RNN. Note that we do not assume that the order in which variables values are encoded by the RNN for each program state but rather maintain a consistent order throughout all program states for a given trace. Finally, we feed a softmax regression layer with the final state of the second RNN (shown in FIG2 . The benefit of state trace embedding is its ability to capture dependencies among variables in each program state as well as the relationship among program states. Dependency Enforcement for Variable Trace Embedding Although state trace embedding can better capture program dependencies, it also comes with some challenges, the most significant of which is redundancy. Consider a looping structure in a program. During an iteration, whenever one variable gets modified, a new program state will be created containing the values of all variables, even of those unmodified by the loop. This issue becomes more severe for loops with larger numbers of iterations. To tackle this challenge, we propose the third and final approach, dependency enforcement for variable trace embedding (hereinafter referred as dependency enforcement embedding), that combines the advantages of variable trace embedding (i.e., compact representation of execution traces) and state trace embedding (i.e., precise capturing of program dependencies). In dependency enforcement embedding, a program is represented by separate variable traces, with each variable being handled by a different RNN. In order to enforce program dependencies, the hidden states from different RNNs will be interleaved in a way that simulates the needed data and control dependencies. Unlike variable trace embedding, we perform an average pooling on the final states of all RNNs to obtain the program embedding on which we build the final layer of softmax regression. FIG3 describes the workflow. We now formally define the three program embedding models. Given a program P, and its variable set V (v 0, v 1,. .., v n ∈ V), a variable trace is a sequence of values a variable has been assigned during the execution of P.4 Let x t vn denote the value from the variable trace of v n that is fed to the RNN encoder (Gated Recurrent Unit) at time t as the input, and h t vn as the ing RNN's hidden state. We compute the variable trace embedding for P in Equation as follows (h T vn denotes the last hidden state of the encoder): DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 We compute the representation of the program trace by performing max pooling over the last hidden state representation of each variable trace embedding. The hidden states h t v1,..., h t vn, h P ∈ R k where k denotes the size of hidden layers of the RNN encoder. Evidence denotes the output of a linear model through the program embedding vector h P, and we obtain the predicted error pattern class Y by using a softmax operation. The key idea in state trace model is to embed each program state as a numerical vector first and then feed all program state embeddings as a sequence to another RNN encoder to obtain the program embedding. Suppose x t vn is the value of variable v n at t-th program state, and h t vn is the ing hidden state of the program state encoder. Equation computes the t-th program state embedding. Equations encode the sequence of all program state embeddings (i.e., h t vn, h t+1 vn, . . ., h t+m vn) with another RNN to compute the program embedding. DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 h t+1 vn = GRU(h t vn, h t+1 vn)... h P = GRU(h t+m−1 vn, x t+m vn) h t v1,..., h t vn ∈ R k1; h t vn,..., h P ∈ R k2 where k 1 and k 2 denote, respectively, the sizes of hidden layers of the first and second RNN encoders. The motivation behind this model is to combine the advantages of the previous two approaches, i.e. representing the execution trace compactly while enforcing the dependency relationship among variables as much as possible. In this model, each variable trace is handled with a different RNN. A potential issue to be addressed is variable matching/renaming (i.e., α-renaming). In other words same variables may be named differently in different programs. Processing each variable id with a single RNN among all programs in the dataset will not only cause memory issues, but more importantly the loss of precision. Our solution is to execute all programs to collect traces for all variables, perform dynamic time wrapping on the variable traces across all programs to find the top-n most used variables that account for the vast majority of variable usage, and rename the top-n most used variables consistently across all programs, and rename all other variables to a same special variable. Given the same set of variables among all programs, the mechanism of dependency enforcement on the top ones is to fuse the hidden states of multiple RNNs based on how a new value of a variable is produced. For example, in FIG0 at line 8, the new value of max val is data-dependent on item, and control-dependent on both item and itself. So at the time step when the new value of max val is produced, the latest hidden states of the RNNs encode variable item as well as itself; they together determine the previous state of the RNN upon which the new value of max val is produced. If a value is produced without any dependencies, this mechanism will not take effect. In other words, the RNN will act normally to handle data sequences on its own. In this work we enforce the data-dependency in assignment statement, declaration statement and method calls; and control-dependency in control statements such as if, f or and while statements. Equations (11 and 12) expose the inner workflow. h LT vm denotes the latest hidden state of the RNN encoding variable trace of v m up to the point of time t when x t vn is the input of the RNN encoding variable trace of v n. denotes element-wise matrix product. DISPLAYFORM0 Given v n depends on v 1 and v m We train our dynamic program embeddings on the programming submissions obtained from Assignment 2 from Microsoft-DEV204.1X: "Introduction to C#" offered on edx and two other problems on Microsoft CodeHunt platform. DISPLAYFORM1 • Print Chessboard: Print the chessboard pattern using "X" and "O" to represent the squares as shown in FIG4.• Count Parentheses: Count the depth of nesting parentheses in a given string.• Generate Binary Digits: Generate the string of binary digits for a given integer. Regarding the three programming problems, the errors students made in their submissions can be roughly classified into low-level technical issues (e.g., list indexing, branching conditions or looping bounds) and high-level conceptual issues (e.g., mishandling corner case, misunderstanding problem requirement or misconceptions on the underlying data structure of test inputs). In order to have sufficient data for training our models to predict the error patterns, we convert each incorrect program into multiple programs such that each new program will have only one error, and mutate all the correct programs to generate synthetic incorrect programs such that they exhibit similar errors that students made in real program submissions. These two steps allow us to set up a dataset depicted in TAB4. Based on the same set of training data, we evaluate the dynamic embeddings trained with the three network models and compare them with the syntax-based program embeddings (on the same error prediction task) on the same testing data. The syntax-based models include one trained with a RNN that encodes the run-time syntactic traces of programs BID10; another trained with a RNN that encodes token sequences of programs; and the third trained with a RNN on abstract syntax trees of programs BID11 5 Please refer to the Appendix for a detailed summary of the error patterns for each problem. All models are implemented in TensorFlow. All encoders in each of the trace model have two stacked GRU layers with 200 hidden units in each layer except that the state encoder in the state trace model has one single layer of 100 hidden units. We adopt random initialization for weight initialization. Our vocabulary has 5,568 unique tokens (i.e., the values of all variables at each time step), each of which is embedded into a 100-dimensional vector. All networks are trained using the Adam optimizer BID5 with the learning and the decay rates set to their default values (learning rate = 0.0001, beta1 = 0.9, beta2 = 0.999) and a mini-batch size of 500. For the variable trace and dependency enforcement models, each trace is padded to have the same length across each batch; for the state trace model, both the number of variables in each program state as well as the length of the entire state trace are padded. During the training of the dependency enforcement model, we have observed that when dependencies become complex, the network suffers from optimization issues, such as diminishing and exploding gradients. This is likely due to the complex nature of fusing hidden states among RNNs, echoing the errors back and forth through the network. We resolve this issue by truncating each trace into multiple sub-sequences and only back-propagate on the last sub-sequence while only feedforwarding on the rest. Regarding the baseline network trained on syntactic traces/token sequences, we use the same encoder architecture (i.e., two layer GRU of 200 hidden units) processing the same 100-dimension embedding vector for each statement/token. As for the AST model, we learn an embedding (100-dimension) for each type of the syntax node by propagating the leaf (a simple look up) to the root through the learned production rules. Finally, we use the root embeddings to represent programs. Table 3: Comparing dynamic program embeddings with syntax-based program embedding in predicting common error patterns made by students. As shown in Table 3, our embeddings trained on execution traces significantly outperform those trained on program syntax (greater than 92% accuracy compared to less than 27% for syntax-based embeddings). We conjecture this is because of the fact that minor syntactic discrepancies can lead to major semantic differences as shown in Figure 1. In our dataset, there are a large number of programs with distinct labels that differ by only a few number of tokens or AST nodes, which causes difficulty for the syntax models to generalize. Even for the simpler syntax-level errors, they are buried in large number of other syntactic variations and the size of the training dataset is relatively small for the syntax-based models to learn precise patterns. In contrast, dynamic embeddings are able to canonicalize the syntactical variations and pinpoint the underlying semantic differences, which in the trace-based models learning the correct error patterns more effectively even with relatively smaller size of the training data. In addition, we incorporated our dynamic program embeddings into SARFGEN BID13 -a program repair system -to demonstrate their benefit in producing fixes to correct students errors in programming assignments. Given a set of potential repair candidates, SARFGEN uses an enumerative search-based technique to find minimal changes to an incorrect program. We use the dynamic embeddings to learn a distribution over the corrections to prioritize the search for the repair algorithm. 6 To establish the baseline, we obtain the set of all corrections from SARFGEN for each of the real incorrect program to all three problems and enumerate each subset until we find the minimum fixes. On the contrary, we also run another experiment where we prioritize each correction according to the prediction of errors with the dynamic embeddings. It is worth mentioning that one incorrect program may be caused by multiple errors. Therefore, we only predict the top-1 error each time and repair the program with the corresponding corrections. If the program is still incorrect, we repeat this procedure till the program is fixed. The comparison between the two approaches is based on how long it takes them to repair the programs. Enumerative Search Table 4: Comparing the enumerative search with those guided by dynamic program embeddings in finding the minimum fixes. Time is measured in seconds. As shown in Table 4, the more fixes required, the more speedups dynamic program embeddings yield -more than an order of magnitude speedups when the number of fixes is four or greater. When the number of fixes is greater than seven, the performance gain drops significantly due to poor prediction accuracy for programs with too many errors. In other words, our dynamic embeddings are not viewed by the network as capturing incorrect execution traces, but rather new execution traces. Therefore, the predictions become unreliable. Note that we ignored incorrect programs having greater than 10 errors when most experiments run out of memory for the baseline approach. There has been significant recent interest in learning neural program representations for various applications, such as program induction and synthesis, program repair, and program completion. Specifically for neural program repair techniques, none of the existing techniques, such as DeepFix BID4, SynFix BID1 and sk p BID9, have considered dynamic embeddings proposed in this paper. In fact, dynamic embeddings can be naturally extended to be a new feature dimension for these existing neural program repair techniques. BID8 is a notable recent effort targeting program representation. Piech et al. explore the possibility of using input-output pairs to represent a program. Despite their new perspective, the direct mapping between input and output of programs usually are not precise enough, i.e., the same input-output pair may correspond to two completely different programs, such as the two sorting algorithms in Figure 1. As we often observe in our own dataset, programs with the same error patterns can also in different input-output pairs. Their approach is clearly ineffective for these scenarios. BID10 introduced the novel approach of using execution traces to induce and execute algorithms, such as addition and sorting, from very few examples. The differences from our work are they use a sequence of instructions to represent dynamic execution trace as opposed to using dynamic program states; their goal is to synthesize a neural controller to execute a program as a sequence of actions rather than learning a semantic program representation; and they deal with programs in a language with low-level primitives such as function stack push/pop actions rather than a high-level programming language. As for learning representations, there are several related efforts in modeling semantics in sentence or symbolic expressions BID11 BID14 BID2. These approaches are similar to our work in spirit, but target different domains than programs. We have presented a new program embedding that learns program representations from runtime execution traces. We have used the new embeddings to predict error patterns that students make in their online programming submissions. Our evaluation shows that the dynamic program embeddings significantly outperform those learned via program syntax. We also demonstrate, via an additional application, that our dynamic program embeddings yield more than 10x speedups compared to an enumerative baseline for search-based program repair. Beyond neural program repair, we believe that our dynamic program embeddings can be fruitfully utilized for many other neural program analysis tasks such as program induction and synthesis. for Pc ∈ Pcs do // Generates the syntactic discrepencies w.r.t. each Pc 7 C(P, Pc) ← DiscrepenciesGeneration(P, Ps) // Executing P to extract the dynamic execution trace 8 T (P) ← DynamicTraceExtraction(P) // Prioritizing subsets of C(P, Pc) through pre-trained model 9 C subs (P, Pc) ← Prioritization(C(P, Pc), T (P), M) 10 for C sub (P, Pc) ∈ C subs (P, Pc) do

represent the tldr for finding the corresponding scientific article

A new way of learning semantic program embedding

Represent the TLDR for finding the corresponding scientific article.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we propose a generalization of the BN algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way. Batch normalization (BN) is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm remains the overall structure of the original BN algorithm while introduces a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to trainable parameters. Our analysis can be easily generalized for original BN algorithm by setting some parameters to constant. To the best knowledge of authors, this analysis is the first of its kind for convergence with Batch Normalization introduced. We analyze a two-layer model with arbitrary activation function. The primary challenge of the analysis is the fact that some parameters are updated by gradient while others are not. The convergence analysis applies to any activation function that satisfies our common assumptions. For the analysis, we also show the sufficient and necessary conditions for the stepsizes and diminishing weights to ensure the convergence. In the numerical experiments, we use more complex models with more layers and ReLU activation. We observe that DBN outperforms the original BN algorithm on Imagenet, MNIST, NI and CIFAR-10 datasets with reasonable complex FNN and CNN models. Deep neural networks (DNN) have shown unprecedented success in various applications such as object detection. However, it still takes a long time to train a DNN until it converges. Ioffe & Szegedy identified a critical problem involved in training deep networks, internal covariate shift, and then proposed batch normalization (BN) to decrease this phenomenon. BN addresses this problem by normalizing the distribution of every hidden layer's input. In order to do so, it calculates the preactivation mean and standard deviation using mini-batch statistics at each iteration of training and uses these estimates to normalize the input to the next layer. The output of a layer is normalized by using the batch statistics, and two new trainable parameters per neuron are introduced that capture the inverse operation. It is now a standard practice;. While this approach leads to a significant performance jump, to the best of our knowledge, there is no known theoretical guarantee for the convergence of an algorithm with BN. The difficulty of analyzing the convergence of the BN algorithm comes from the fact that not all of the BN parameters are updated by gradients. Thus, it invalidates most of the classical studies of convergence for gradient methods. In this paper, we propose a generalization of the BN algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way. It essentially means that the BN layer adjusts its output according to all past mini-batches instead of only the current one. It helps to reduce the problem of the original BN that the output of a BN layer on a particular training pattern depends on the other patterns in the current mini-batch, which is pointed out by Bottou et al.. By setting the layer parameter we introduce into DBN to a specific value, we recover the original BN algorithm. We give a convergence analysis of the algorithm with a two-layer batch-normalized neural network and diminishing stepsizes. We assume two layers (the generalization to multiple layers can be made by using the same approach but substantially complicating the notation) and an arbitrary loss function. The convergence analysis applies to any activation function that follows our common assumption. The main shows that under diminishing stepsizes on gradient updates and updates on mini-batch statistics, and standard Lipschitz conditions on loss functions DBN converges to a stationary point. As already pointed out the primary challenge is the fact that some trainable parameters are updated by gradient while others are updated by a minor recalculation. Contributions. The main contribution of this paper is in providing a general convergence guarantee for DBN. Specifically, we make the following contributions.• In section 4, we show the sufficient and necessary conditions for the stepsizes and diminishing weights to ensure the convergence of BN parameters.• We show that the algorithm converges to a stationary point under a general nonconvex objective function. This paper is organized as follows. In Section 2, we review the related works and the development of the BN algorithm. We formally state our model and algorithm in Section 3. We present our main in Sections 4. In Section 5, we numerically show that the DBN algorithm outperforms the original BN algorithm. Proofs for main steps are collected in the Appendix. Before the introduction of BN, it has long been known in the deep learning community that input whitening and decorrelation help to speed up the training process. In fact, Orr & Müller show that preprocessing the data by subtracting the mean, normalizing the variance, and decorrelating the input has various beneficial effects for back-propagation. Krizhevsky et al. propose a method called local response normalization which is inspired by computational neuroscience and acts as a form of lateral inhibition, i.e., the capacity of an excited neuron to reduce the activity of its neighbors. Gülçehre & Bengio propose a standardization layer that bears significant resemblance to batch normalization, except that the two methods are motivated by very different goals and perform different tasks. Inspired by BN, several new works are taking BN as a basis for further improvements. Layer normalization BID2 is much like the BN except that it uses all of the summed inputs to compute the mean and variance instead of the mini-batch statistics. Besides, unlike BN, layer normalization performs precisely the same computation at training and test times. Normalization propagation that Arpit et al. uses data-independent estimations for the mean and standard deviation in every layer to reduce the internal covariate shift and make the estimation more accurate for the validation phase. Weight normalization also removes the dependencies between the examples in a minibatch so that it can be applied to recurrent models, reinforcement learning or generative models. Cooijmans et al. propose a new way to apply batch normalization to RNN and LSTM models. Given all these flavors, the original BN method is the most popular technique and for this reason our choice of the analysis. To the best of our knowledge, we are not aware of any prior analysis of BN.BN has the gradient and non-gradient updates. Thus, nonconvex convergence do not immediately transfer. Our analysis explicitly considers the workings of BN. However, nonconvex convergence proofs are relevant since some small portions of our analysis rely on known proofs and approaches. The optimization problem for a network is an objective function consisting of a large number of component functions, that reads: DISPLAYFORM0 where DISPLAYFORM1.., N, are real-valued functions for any data record X i. Index i associates with data record X i and target response y i (hidden behind the dependency of f on i) in the training set. Parameters θ include the common parameters updated by gradients directly associated with the loss function, i.e., behind the part that we have a parametric model, while BN parameters λ are introduced by the BN algorithm and not updated by gradient methods but by the mini-batch statistics. We define that the derivative of f i is always taken with respect to θ: DISPLAYFORM2 The deep network we analyze has 2 fully-connected layers with D 1 neurons each. The techniques presented can be extended to more layers with additional notation. Each hidden layer computes y = a(W u) with activation function a(·) and u is the input vector of the layer. We do not need to include an intercept term since the BN algorithm automatically adjusts for it. BN is applied to the output of the first hidden layer. We next describe the computation in each layer to show how we obtain the output of the network. The notations introduced here is used in the analysis. FIG0 shows the full structure of the network. The input data is vector X, which is one of DISPLAYFORM3 is the set of all BN parameters and vector θ = W 1, W 2, (β DISPLAYFORM4 is the set of all trainable parameters which are updated by gradients. Matrices W 1, W 2 are the actual model parameters and β, γ are introduced by BN. The value of j th neuron of the first hidden layer is DISPLAYFORM5 where W 1,j,· denotes the weights of the linear transformations for the j th neuron. The j th entry of batch-normalized output of the first layer is DISPLAYFORM6 DISPLAYFORM7 The objective function for the i th sample is DISPLAYFORM8 where l i (·) is the loss function associated with the target response y i. For sample i, we have the following complete expression for the objective function: DISPLAYFORM9 Function f i (X i : θ, λ) is nonconvex with respect to θ and λ. Algorithm 1 shows the algorithm studied herein. There are two deviations from the standard BN algorithm, one of them actually being a generalization. We use the full gradient instead of the more popular stochastic gradient (SG) method. It essentially means that each batch contains the entire training set instead of a randomly chosen subset of the training set. An analysis of SG is potential future research. Although the primary motivation for full gradient update is to reduce the burdensome in showing the convergence, the full gradient method is similar to SG in the sense that both of them go through the entire training set, while full gradient goes through it deterministically and the SG goes through it in expectation. Therefore, it is reasonable to speculate that the SG method has similar convergence property as the full algorithm studied herein. Algorithm 1 DBN: Diminishing Batch-Normalized Network Update Algorithm 1: Initialize θ ∈ R n1 and λ ∈ R n2 2: for iteration m=1,2,... do 3: DISPLAYFORM0 The second difference is that we update the BN parameters (θ, λ) by their moving averages with respect to diminishing α (m). The original BN algorithm can be recovered by setting α (m) = 1 for every m. After introducing diminishing α (m), λ (m) and hence the output of the BN layer is determined by the history of all past data records, instead of those solely in the last batch. Thus, the output of the BN layer becomes more general that better reflects the distribution of the entire dataset. We use two strategies to decide the values of α (m). One is to use a constant smaller than 1 for all m, and the other one is to decay the α (m) gradually, such as α (m) = 1/m. In our numerical experiment, we show that Algorithm 1 outperforms the original BN algorithm, where both are based on SG and non-linear activation functions with many layers FNN and CNN models. The main purpose of our work is to show that Algorithm 1 converges. In the general case, we focus on the nonconvex objective function. Here are the assumptions we used for the convergence analysis. Assumption 1 (Lipschitz continuity on θ and λ). For every i we have DISPLAYFORM0 Noted that the Lipschitz constants associated with each of the above inequalities are not necessarily the same. HereL is an upper bound for these Lipschitz constants for simplicity. Assumption 2 (bounded parameters). Sets P and Q are compact set, where θ ∈ P and λ ∈ Q. Thus, there exists a constant M that weights W and parameters λ are bounded element-wise by this constant M. DISPLAYFORM1 This also implies that the updated θ, λ in Algorithm 1 remain in P and Q, respectively. Assumption 3 (diminishing update on θ). The stepsizes of θ update satisfy DISPLAYFORM2 This is a common assumption for diminishing stepsizes in optimization problems. Assumption 4 (Lipschitz continuity of l i (·)). Assume the loss functions l i (·) for every i is continuously differentiable. It implies that there existsM such that DISPLAYFORM3 Assumption 5 (existence of a stationary point). There exists a stationary point (θ DISPLAYFORM4 We note that all these are standard assumptions in convergence proofs. We also stress that Assumption 4 does not directly imply 1. Since we assume that P and Q are compact, then Assumptions 1, 4 and 5 hold for many standard loss function such as softmax and MSE.Assumption 6 (Lipschitz at activation function). The activation function a(·) is Lipschitz with constant k: DISPLAYFORM5 Since for all activation function there is a = 0, the condition is equivalent to |a(x) − a| ≤ k x − 0. We note that this assumption works for many popular choices of activation functions, such as ReLU and LeakyReLu. We first have the following lemma specifying sufficient conditions for λ to converge. Proofs for main steps are given in the Appendix. Theorem 7 Under Assumptions 1, 2, 3 and 6, if {α (m) } satisfies DISPLAYFORM0 We give a discussion of the above conditions for α (m) and η (m) at the end of this section. With the help of Theorem 7, we can show the following convergence . Lemma 8 Under Assumptions 4, 5 and the assumptions of Theorem 7, when DISPLAYFORM1 we have DISPLAYFORM2 This is similar to the classical convergence rate analysis for the non-convex objective function with diminishing stepsizes, which can be found in.Lemma 9 Under the assumptions of Lemma 8, we have DISPLAYFORM3 This theorem states that for the full gradient method with diminishing stepsizes the gradient norms cannot stay bounded away from zero. The following characterizes more precisely the convergence property of Algorithm 1.Lemma 10 Under the assumptions stated in Lemma 8, we have DISPLAYFORM4 Our main is listed next. Theorem 11 Under the assumptions stated in Lemma 8, we have DISPLAYFORM5 We cannot show that {θ (m) }'s converges (standard convergence proofs are also unable to show such a stronger statement). For this reason, Theorem 11 does not immediately follow from Lemma 10 together with Theorem 7. The statement of Theorem 11 would easily follow from Lemma 10 if the convergence of {θ (m) } is established and the gradient being continuous. We show in the Appendix that the set of sufficient and necessary conditions to satisfy the assumptions of Theorem 7 are h > 1 and k ≥ 1. The set of sufficient and necessary conditions to satisfy the assumptions of Lemma 8 are h > 2 and k ≥ 1. For example, we can pick DISPLAYFORM0 ) to achieve the above convergence in Theorem 11. We conduct the computational experiments with Theano and Lasagne on a Linux server with a Nvidia Titan-X GPU. We use , CIFAR-10 and Network Intrusion (NI) kdd datasets to compare the performance between DBN and the original BN algorithm. For the MNIST dataset, we use a four-layer fully connected FNN (784 × 300 × 300 × 10) with the ReLU activation function and for the NI dataset, we use a four-layer fully connected FNN (784 × 50 × 50 × 10) with the ReLU activation function. For the CIFAR-10 dataset, we use a reasonably complex CNN network that has a structure of (Conv-Conv-MaxPool-DropoutConv-Conv-MaxPool-Dropout-FC-Dropout-FC), where all four convolution layers and the first fully connected layers are batch normalized. We use the softmax loss function and l 2 regularization with for all three models. All the trainable parameters are randomly initialized before training. For all 3 datasets, we use the standard epoch/minibatch setting with the minibatch size of 100, i.e., we do not compute the full gradient and the statistics are over the minibatch. We use to update the learning rates η (m) for trainable parameters, starting from 0.01. We test all the choices of α (m) with the performances presented in Figure 2. Figure 2 shows that all the non-zero choices of α (m) converge properly. The algorithms converge without much difference even when α (m) in DBN is very small, e.g., 1/m 2. However, if we select α (m) = 0, the algorithm is erratic. Besides, we observe that all the non-zero choices of α (m) converge at a similar rate. The fact that DBN keeps the batch normalization layer stable with a very small α (m) suggests that the BN parameters do not have to be depended on the latest minibatch, i.e., the original BN.We compare a selected set of the most efficient choices of α TAB2 shows the best obtained from each choice of α (m). Most importantly, it suggests that the choices of α (m) = 1/m and 1/m 2 perform better than the original BN algorithm. Besides, all the constant less-than-one choices of α (m) perform better than the original BN, showing the importance of considering the mini-batch history for the update of the BN parameters. The BN algorithm in each figure converges to similar error rates on test datasets with different choices of α (m) except for the α (m) = 0 case. Among all the models we tested, α (m) = 0.25 is the only one that performs top 3 for all three datasets, thus the most robust choice. To summarize, our numerical experiments show that the DBN algorithm outperforms the original BN algorithm on the MNIST, NI and CIFAT-10 datasets with typical deep FNN and CNN models. On the analytical side, we believe an extension to more than 2 layers is doable with significant augmentations of the notation. The following proofs are shortened to corporate with AAAI submission page limit. Proposition 12 There exists a constant M such that, for any θ and fixed λ, we have DISPLAYFORM0 Proof. By Assumption 5, we know there exists (θ *, λ *) such that ∇f (θ *, λ *) 2 = 0. Then we have DISPLAYFORM1 where the last inequality is by Assumption 1. We then have ∇f (θ, λ) DISPLAYFORM2 because sets P and Q are compact by Assumption 2. Proof. This is a known of the Lipschitz-continuous condition that can be found in. We have this together with Assumption 1. Lemma 14 When DISPLAYFORM0 is a Cauchy series. Proof. By Algorithm 1, we have DISPLAYFORM1 We defineα DISPLAYFORM2 Then we have DISPLAYFORM3 It remains to show that DISPLAYFORM4 DISPLAYFORM5 implies the convergence of {μ (m) }. By, we have Π DISPLAYFORM6 It is also easy to show that there exists C and Mc such that for all m ≥ Mc, we have DISPLAYFORM7 Therefore, lim DISPLAYFORM8 Thus the following holds: DISPLAYFORM9 and DISPLAYFORM10 From equation 29 and equation 32 it follows that the sequence {μ DISPLAYFORM11} is a Cauchy series. Lemma 15 Since {μ DISPLAYFORM12} is a Cauchy series, {µ DISPLAYFORM13} is a Cauchy series. Proof. We know that µ } is a Cauchy series. Proof. We define σ DISPLAYFORM0 Since {µ (m) j } is convergent, there exists c1, c2 and N1 such that for any m > N1, −∞ < c1 < µ DISPLAYFORM1 Inequality equation 35 is by the following fact: DISPLAYFORM2 where b and ai for every i are arbitrary real scalars. Besides, equation 39 is due to −2aic ≤ max{−2|ai|c, 2|ai|c}.Inequality equation 36 follow from the square function being increasing for nonnegative numbers. Besides these facts, equation 36 is also by the same techniques we used in equation 23-equation 25 where we bound the derivatives with the Lipschitz continuity in the following inequality: DISPLAYFORM3 Inequality equation 37 is by collecting the bounded terms into a single boundML,M. Therefore, DISPLAYFORM4 Using the similar methods in deriving equation 28 and equation 29, it can be seen that a set of sufficient conditions ensuring the convergence for {σ DISPLAYFORM5 Therefore, the convergence conditions for {σ (m) j } are the same as for {µ DISPLAYFORM6 It is clear that these lemmas establish the proof of Theorem 7. Proposition 17 Under the assumptions of Theorem 7, we have |λ (m) −λ|∞ ≤ am, where DISPLAYFORM0 M1 and M2 are constants. Proof. For the upper bound of σ DISPLAYFORM1, by equation 38, we have DISPLAYFORM2 We defineσj:=σ DISPLAYFORM3 The first inequality comes by substituting p by m and by taking lim as q → ∞ in equation 41. The second inequality comes from equation 30. We then obtain, DISPLAYFORM4 The second inequality is by (1 − α )...(1 − α (m) ) < 1, the third inequality is by equation 30 and the last inequality can be easily seen by induction. By equation 44, we obtain DISPLAYFORM5 Therefore, we have DISPLAYFORM6 The first inequality is by equation 45, the second inequality is by equation 41, the third inequality is by equation 31 and the fourth inequality is by adding the nonnegative termσ j C α (m) to the right-hand side. For the upper bound of µ DISPLAYFORM7 Let us define Am:= μ (m) −μ (∞) and Bm:= μj DISPLAYFORM8 } is a Cauchy series, by equation 27, |μ DISPLAYFORM9. Therefore, the first term in equation 47 is bounded by DISPLAYFORM10 For the second term in equation 47, recall that C: DISPLAYFORM11 where the inequality can be easily seen by induction. Therefore, the second term in equation 47 is bounded by DISPLAYFORM12 From these we obtain DISPLAYFORM13 DISPLAYFORM14 where M1 and M2 are constants defined as DISPLAYFORM15 Proposition 18 Under the assumptions of Theorem 7, DISPLAYFORM16, where am is defined in Proposition 17.Proof. For simplicity of the proof, let us define DISPLAYFORM17 where √ n2 is the dimension of λ. The second inequality is by Assumption 1 and the fourth inequality is by Proposition 17. Inequality equation 51 implies that for all m and i, we have |x DISPLAYFORM18 This is established by the following four cases. DISPLAYFORM0 by Proposition 12. DISPLAYFORM1 The last inequality is by Proposition 12.All these four cases yield equation 52.Proposition 19 Under the assumptions of Theorem 7, we havē DISPLAYFORM2 where M is a constant and am is defined in Proposition 17.Proof. By Proposition 13, DISPLAYFORM3 2. Therefore, we can sum it over the entire training set from i = 1 to N to obtain DISPLAYFORM4 In Algorithm 1, we define the update of θ in the following full gradient way: DISPLAYFORM5 which implies DISPLAYFORM6 By equation 56 we haveθ DISPLAYFORM7. We now substituteθ:= θ (m+1), θ:= θ (m) and λ:=λ into equation 54 to obtain DISPLAYFORM8 The first inequality is by plugging equation 56 into equation 54, the second inequality comes from Proposition 12 and the third inequality comes from Proposition 18. Here we show Theorem 11 as the consequence of Theorem 7 and Lemmas 8, 9 and 10.6.4.1 PROOF OF LEMMA 8Here we show Lemma 8 as the consequence of Lemmas 20, 21 and 22.Lemma 20 DISPLAYFORM0 Proof. By plugging equation 45 and equation 43 into equation 58, we have the following for all j: DISPLAYFORM1 It is easy to see that the the following conditions are sufficient for right-hand side of equation 59 to be finite: DISPLAYFORM2 Therefore, we obtain DISPLAYFORM3 Lemma 21 Under Assumption 4, DISPLAYFORM4 is a set of sufficient conditions to ensure DISPLAYFORM5 Proof. By Assumption 4, we have DISPLAYFORM6 By the definition of fi(·), we then have DISPLAYFORM7 The first inequality is by the Cauchy-Schwarz inequality, and the second one is by equation 60. To show the finiteness of equation 64, we only need to show the following two statements: DISPLAYFORM8 and DISPLAYFORM9 Proof of equation 65: For all j we have DISPLAYFORM10 The inequality comes from |W Finally, we invoke Lemma 14 to assert that DISPLAYFORM11 Proof of equation 66: For all j we have DISPLAYFORM12 The first term in equation 68 is finite since {µ Noted that function f (σ) = 1 σ + B is Lipschitz continuous since its gradient is bounded by 1 Next we show that each of the four terms in the right-hand side of equation 75 is finite, respectively. For the first term, DISPLAYFORM13 is by the fact that the parameters {θ, λ} are in compact sets, which implies that the image of fi(·) is in a bounded set. For the second term, we showed its finiteness in Lemma 21. The right-hand side of equation 77 is finite because DISPLAYFORM14 and DISPLAYFORM15 The second inequalities in equation 78 and equation 79 come from the stated assumptions of this lemma. For the fourth term, DISPLAYFORM16 holds, because we have In Lemmas 20, 21 and 22, we show that {σ (m) } and {µ (m) } are Cauchy series, hence Lemma 8 holds. This proof is similar to the the proof by.Proof. By Theorem 8, we have DISPLAYFORM0 If there exists a > 0 and an integerm such that ∇f (θ (m),λ) 2 ≥ Since k ≥ 1 due to Assumption 3, we conclude that k + h > 2. Therefore, the conditions for η (m) and α (m) to satisfy the assumptions of Theorem 7 are h > 1 and k ≥ 1. For the assumptions of Theorem 7, the first condition DISPLAYFORM0 requires h > 2.Besides, the second condition is DISPLAYFORM1 The inequality holds because for any p > 1, we have DISPLAYFORM2 Therefore, the conditions for η (m) and α (m) to satisfy the assumptions of Lemma 8 are h > 2 and k ≥ 1.

represent the short summary for finding the corresponding scientific article (long).

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.