Question: represent the scientific article to find a short tldr! E.g. given 'A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multi-agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment. Many real-world scenarios that require cooperation among multiple autonomous agents are multi-goal multi-agent control problems: each agent needs to achieve its own individual goal, but the global optimum where all agents succeed is only attained when agents cooperate to allow the success of other agents. In autonomous driving, multiple vehicles must execute cooperative maneuvers when their individual goal locations and nominal trajectories are in conflict (e.g., double lane merges) . In social dilemmas, mutual cooperation has higher global payoff but agents' individual goals may lead to defection out of fear or greed . Even settings with a global objective that seem unfactorizable can be formulated as multi-goal problems: in Starcraft II micromanagement, a unit that gathers resources must not accidentally jeopardize a teammate's attempt to scout the opponent base ; in traffic flow optimization, different intersection controllers may have local throughput goals but must cooperate for high global performance . While the framework of multi-agent reinforcement learning (MARL) (; ;) has been equipped with methods in deep reinforcement learning (RL) and shown promise on high-dimensional problems with complex agent interactions (; ; ; ;), learning multi-agent cooperation in the multi-goal scenario involves significant open challenges. First, given that exploration is crucial for RL and even more so in MARL with larger state and joint action spaces, how should agents explore to learn both individual goal attainment and cooperation for others' success? Uniform random exploration is common in deep MARL but can be highly inefficient as the value of cooperative actions may be discoverable only in small regions of state space where cooperation is needed. Furthermore, the conceptual difference between attaining one's own goal and cooperating for others' success calls for more modularized and targeted approaches. Second, while there are methods for multi-agent credit assignment when all agents share a single goal (i.e., a global reward) (; ;), and while one could treat the cooperative multi-goal scenario as a problem with a single joint goal, this coarse approach makes it extremely difficult to evaluate the impact of an agent's action on another agent's success. Instead, the multi-goal scenario can benefit from fine-grained credit assignment that leverages available structure in action-goal interactions, such as local interactions where only few agents affect another agent's goal attainment at any time. Given these open challenges, our paper focuses on the cooperative multi-goal multi-agent setting where each agent is assigned a goal 1 and must learn to cooperate with other agents with possibly different goals. To tackle the problems of efficient exploration and credit assignment in this complex problem setting, we develop CM3, a novel general framework involving three synergistic components: 1. We approach the difficulty of multi-agent exploration from a novel curriculum learning perspective, by first training an actor-critic pair to achieve different goals in an induced single-agent setting (Stage 1), then using them to initialize all agents in the multi-agent environment (Stage 2). The key insight is that agents who can already act toward individual objectives are better prepared for discovery of cooperative solutions with additional exploration once other agents are introduced. In contrast to hierarchical learning where sub-goals are selected sequentially in time , all agents act toward their goals simultaneously in Stage 2 of our curriculum. 2. Observing that a wide array of complex MARL problems permit a decomposition of agents' observations and state vectors into components of self, others, and non-agent specific environment information , we employ function augmentation to bridge Stages 1-2: we reduce the number of trainable parameters of the actor-critic in Stage 1 by limiting their input space to the part that is sufficient for single-agent training, then augment the architecture in Stage 2 with additional inputs and trainable parameters for learning in the multi-agent environment. 3. We propose a credit function, which is an action-value function that specifically evaluates actiongoal pairs, for localized credit assignment in multi-goal MARL. We use it to derive a multi-goal multi-agent policy gradient for Stage 2. In synergy with the curriculum, the credit function is constructed via function augmentation from the critic in Stage 1. We evaluate our method on challenging multi-goal multi-agent environments with high-dimensional state spaces: cooperative navigation with difficult formations, double lane merges in the SUMO simulator , and strategic teamwork in a Checkers game. CM3 solved all domains significantly faster than IAC and COMA , and solved four out of five environments significantly faster than QMIX . Exhaustive ablation experiments show that the combination of all three components is crucial for CM3's overall high performance. While early theoretical work analyzed Markov games in discrete state and action spaces (; ;), recent literature have leveraged techniques from deep RL to develop general algorithms for high dimensional environments with complex agent interactions (; ;), which pose difficulty for traditional methods that do not generalize by learning interactions . Cooperative multi-agent learning is important since many real-world problems can be formulated as distributed systems in which decentralized agents must coordinate to achieve shared objectives . The multi-agent credit assignment problem arises when agents share a global reward . While credit assignment be resolved when independent individual rewards are available , this may not be suitable for the fully cooperative setting: showed that agents whose rewards depend on the success of other agents can cooperate better than agents who optimize for their own success. In the special case when all agents have a single goal and share a global reward, COMA et al., 2019) apply to agents with different rewards, they do not address multi-goal cooperation as they do not distinguish between cooperation and competition, despite the fundamental difference. Multi-goal MARL was considered in , who analyzed convergence in a special networked setting restricted to fully-decentralized training, while we conduct centralized training with decentralized execution . In contrast to multi-task MARL, which aims for generalization among non-simultaneous tasks , and in contrast to hierarchical methods that sequentially select subtasks , our decentralized agents must cooperate concurrently to attain all goals. Methods for optimizing high-level agent-task assignment policies in a hierarchical framework are complementary to our work, as we focus on learning low-level cooperation after goals are assigned. Prior application of curriculum learning to MARL include a single cooperative task defined by the number of agents and the probability of agent appearance , without explicit individual goals. instantiate new neural network columns for task transfer in single-agent RL. Techniques in transfer learning are complementary to our novel curriculum approach to MARL. In multi-goal MARL, each agent should achieve a goal drawn from a finite set, cooperate with other agents for collective success, and act independently with limited local observations. We formalize the problem as an episodic multi-goal Markov game, review an actor-critic approach to centralized training of decentralized policies, and summarize counterfactual-based multi-agent credit assignment. Multi-goal Markov games. A multi-goal Markov game is a tuple S, {O n}, {A n}, P, R, G, N, γ with N agents labeled by n ∈ [N]. In each episode, each agent n has one fixed goal g n ∈ G that is known only to itself. At time t and global state s t ∈ S, each agent n receives an observation o n t:= o n (s t) ∈ O n and chooses an action a n t ∈ A n. The environment moves to s t+1 due to joint action a t:= {a 1 t, . . ., a N t}, according to transition probability P (s t+1 |s t, a t). Each agent receives a reward R n t:= R(s t, a t, g n), and the learning task is to find stochastic decentralized policies π n: O n × G × A n →, conditioned only on local observations and goals, to maximize where γ ∈ and joint policy π factorizes as π(a|s, g):= N n=1 π n (a n |o n, g n) due to decentralization. Let a −n and g −n denote all agents' actions and goals, respectively, except that of agent n. Let boldface a and g denote the joint action and joint goals, respectively. For brevity, let π(a n):= π n (a n |o n, g n). This model covers a diverse set of cooperation problems in the literature , without constraining how the attainability of a goal depends on other agents: at a traffic intersection, each vehicle can easily reach its target location if not for the presence of other vehicles; in contrast, agents in a strategic game may not be able to maximize their rewards in the absence of cooperators . Centralized learning of decentralized policies. A centralized critic that receives full state-action information can speed up training of decentralized actors that receive only local information . Directly extending the single-goal case, for each n ∈ [1..N] in a multigoal Markov game, critics are represented by the value function V π n (s):= E π ∞ t=0 γ t R n t s 0 = s and the action-value function Q π n (s, a):= E π ∞ t=0 γ t R n t s 0 = s, a 0 = a, which evaluate the joint policy π against the reward R n for each goal g n. Multi-agent credit assignment. In MARL with a single team objective, COMA addresses credit assignment by using a counterfactual baseline in an advantage function (, Lemma 1), which evaluates the contribution of a chosen action a n versus the average of all possible counterfactualsâ n, keeping a −n fixed. The analysis in for a formally equivalent action-dependent baseline in RL suggests that COMA is a low-variance estimator for single-goal MARL. We derive its variance in Appendix C.1. However, COMA is unsuitable for credit assignment in multi-goal MARL, as it would treat the collection of goals g as a global goal and only learn from total reward, making it extremely difficult to disentangle each agent's impact on other agents' goal attainment. Furthermore, a global Q-function does not explicitly capture structure in agents' interactions, such as local interactions involving a limited number of agents. We substantiate these arguments by experimental in Section 6. We describe the complete CM3 learning framework as follows. First we define a credit function as a mechanism for credit assignment in multi-goal MARL, then derive a new cooperative multi-goal policy gradient with localized credit assignment. Next we motivate the possibility of significant training speedup via a curriculum for multi-goal MARL. We describe function augmentation as a mechanism for efficiently bridging policy and value functions across the curriculum stages, and finally synthesize all three components into a synergistic learning framework. If all agents take greedy goal-directed actions that are individually optimal in the absence of other agents, the joint action can be sub-optimal (e.g. straight-line trajectory towards target in traffic). Instead rewarding agents for both individual and collective success can avoid such bad local optima. A naïve approach based on previous works would evaluate the joint action a via a global Q-function Q π n (s, a) for each agent's goal g n, but this does not precisely capture each agent's contribution to another agent's attainment of its goal. Instead, we propose an explicit mechanism for credit assignment by learning an additional function Q π n (s, a m) that evaluates pairs of action a m and goal g n, for use in a multi-goal actor-critic algorithm. We define this function and show that it satisfies the classical relation needed for sample-based model-free learning. Definition 1. For n, m ∈ [N], s ∈ S, the credit function for goal g n and a m ∈ A m by agent m is:, the credit function satisfies the following relations: Derivations are given in Appendix B.1, including the relation between Q π n (s, a m) and Q π n (s, a). Equation takes the form of the Bellman expectation equation, which justifies learning the credit function, parameterized by θ Qc, by optimizing the standard loss function in deep RL: While centralized training means the input space scales linearly with agent count, many practical environments involving only local interactions between agents allows centralized training with few agents while retaining decentralized performance when deployed at scale (evidenced in Appendix E). We use the credit function as a critic within a policy gradient for multi-goal MARL. Letting θ parameterize π, the overall objective J(π) is maximized by ascending the following gradient: Proposition 2. The cooperative multi-goal credit function based MARL policy gradient is This is derived in Appendix B.2. For a fixed agent m, the inner summation over n considers all agents' goals g n and updates m's policy based on the advantage of a m over all counterfactual actionŝ a m, as measured by the credit function for g n. The strength of interaction between action-goal pairs is captured by the extent to which Q π n (s,â m) varies withâ m, which directly impacts the magnitude of the gradient on agent m's policy. For example, strong interaction in non-constant Q π n (s, ·), which implies larger magnitude of A π n,m and larger weight on ∇ θ log π(a m). The double summation accounts for first-order interaction between all action-goal pairs, but complexity can be reduced by omitting terms when interactions are known to be sparse, and our empirical runtimes are on par with other methods due to efficient batch computation (Appendix F). As the second term in A π n,m is a baseline, the reduction of variance can be analyzed similarly to that for COMA, given in Appendix C.2.), ablation show stability improvement due to the credit function (Section 6). As the credit function takes in a single agent's action, it synergizes with both CM3's curriculum and function augmentation as described in Section 4.5. Multi-goal MARL poses a significant challenge for exploration. Random exploration can be highly inefficient for concurrently learning both individual task completion and cooperative behavior. Agents who cannot make progress toward individual goals may rarely encounter the region of state space where cooperation is needed, rendering any exploration useless for learning cooperative behavior. On the other extreme, exploratory actions taken in situations that require precise coordination can easily lead to penalties that cause agents to avoid the coordination problem and fail to achieve individual goals. Instead, we hypothesize and confirm in experiments that agents who can achieve individual goals in the absence of other agents can more reliably produce state configurations where cooperative solutions are easily discovered with additional exploration in the multi-agent environment 2. We propose a MARL curriculum that first solves a single-agent Markov decision process (MDP), as preparation for subsequent exploration speedup. Given a cooperative multi-goal Markov game MG, we induce an MDP M to be the tuple S n, O n, A n, P n, R, γ, where an agent n is selected to be the single agent in M. Entities S n, P n, and R are defined by removing all dependencies on agent interactions, so that only components depending on agent n remain. This reduction to M is possible in almost all fully cooperative multi-agent environments used in a large body of work 3 , precisely because they support a variable number of agents, including N = 1. Important real-world settings that allow this reduction include autonomous driving, multi traffic light control, and warehouse commissioning (removing all but one car/controller/robot, respectively, from the environment). Given a full Markov game implementation, the reduction involves only deletion of components associated with all other agents from state vectors (since an agent is uniquely defined by its attributes), deletion of if-else conditions from the reward function corresponding to agent interactions, and likewise from the transition function if a simulation is used. Appendix G provides practical guidelines for the reduction. Based on M, we define a greedy policy for MG. Definition 2. A greedy policy π n by agent n for cooperative multi-goal MG is defined as the optimal policy π * for the induced MDP M where only agent n is present. This naturally leads to our proposed curriculum: Stage 1 trains a single agent in M to achieve a greedy policy, which is then used for initialization in MG in Stage 2. Next we explain in detail how to leverage the structure of decentralized MARL to bridge the two curriculum stages. In Markov games with decentralized execution, an agent's observation space decomposes into self captures the agent's own properties, which must be observable by the agent for closed-loop control, while o n others ∈ O n others is the agent's egocentric observation of other agents. In our work, egocentric observations are private and not accessible by other agents . Similarly, global state s decomposes into s:= (s env, s n, s −n), where Function augmentation π π 1 Figure 1: In Stage 1, Q 1 and π 1 learn to achieve multiple goals in a single-agent environment. Between Stage 1 and 2, π is constructed from the trained π 1 and a new module π 2 according to (same construction is done for Q n (s, a) and Q n (s, a m), not shown). In the multi-agent environment of Stage 2, these augmented functions are instantiated for each of N agents (with parameter-sharing). s env is environment information not specific to any agent (e.g., position of a landmark), and s n captures agent n's information. While this decomposition is implicitly available in a wide range of complex multi-agent environments (; ; ; ; ;), we explicitly use it to implement our curriculum. In Stage 1, as the ability to process o n others and s −n is unnecessary, we reduce the input space of policy and value functions, thereby reducing the number of trainable parameters and lowering the computation cost. In Stage 2, we restore Stage 1 parameters and activate new modules to process additional inputs o n others and s −n. This augmentation is especially suitable for efficiently learning the credit function and global Q-function, since Q(s, a) can be augmented into both Q π n (s, a) and Q π n (s, a m), as explained below. We combine the preceding components to create CM3, using deep neural networks for function approximation (Figure 1 and Algorithm 1). Without loss of generality, we assume parameter-sharing among homogeneous agents with goals as input . The inhomogeneous case can be addressed by N actor-critics. Drawing from multi-task learning , we sample goal(s) in each episode for the agent(s), to train one model for all goals. Stage 1. We train an actor π 1 (a|o, g) and critic Q 1 (s 1, a, g) to convergence according to and Stage 2. The Markov game is instantiated with all N agents. We restore the trained π 1 parameters, instantiate a second neural network π 2 for agents to process o n others, and connect the output of π 2 to a selected hidden layer of π 1. Concretely, let h Being restored from Stage 1, not re-initialized, hidden layers i < i * begin with the ability to process (o n self, g n), while the new weights in π 2 and W 1:2 specifically learn the effect of surrounding agents. Higher layers i ≥ i * that already take greedy actions to achieve goals in Stage 1 must now do so while cooperating to allow other agents' success. This augmentation scheme is simplest for deep policy and value networks using fully-connected or convolutional layers. The middle panel of Figure 1 depicts the construction of π from π 1 and π 2. The global Q π (s, a, g n) is constructed from Q 1 similarly: when the input to Q 1 is (s env, s n, a n, g n), a new module takes input (s −n, a −n) and connects to a chosen hidden layer of 4 Setting i * to be the last hidden layer worked well in our experiments, without needing to tune. augmented from a copy of Q 1, such that when We train the policy using, train the credit function with loss, and train the global Q-function with the joint-action analogue of. We investigated the performance and robustness of CM3 versus existing methods on diverse and challenging multi-goal MARL environments: cooperative navigation in difficult formations, double lane merge in autonomous driving, and strategic cooperation in a Checkers game. We evaluated ablations of CM3 on all domains. We describe key setup here, with full details in Appendices G to J. Cooperative navigation: We created three variants of the cooperative navigation scenario in , where N agents cooperate to reach a set of targets. We increased the difficulty by giving each agent only an individual reward based on distance to its designated target, not a global team reward, but initial and target positions require complex cooperative maneuvers to avoid collision penalties (Figure 3). Agents observe relative positions and velocities (details in Appendix G.1). SUMO: Previous work modeled autonomous driving tasks as MDPs in which all other vehicles do not learn to respond to a single learning agent . However, real-world driving requires cooperation among different drivers' with personal goals. Built in the SUMO traffic simulator with sublane resolution , this experiment requires agent vehicles to learn double-merge maneuvers to reach goal lane assignments (Figure 4). Agents have limited field of view and receive sparse rewards (Appendix G.2). Checkers: We implemented a challenging strategic game (Appendix G.3, an extension of), to investigate whether CM3 is beneficial even when an agent cannot maximize its reward in the absence of another agent. In a gridworld with red and yellow squares that disappear when collected (Figure 2), Agent A receives +1 for red and -0.5 for yellow; Agent B receives -0.5 for red and +1 for yellow. Both have a limited 5x5 field of view. The global optimum requires each agent to clear the path for the other. Algorithm implementations. We describe key points here, leaving complete architecture details and hyperparameter tables to Appendices H and I. CM3: Stage 1 is defined for each environment as follows (Appendix G): in cooperative navigation, a single particle learns to reach any specified landmark; in SUMO, a car learns to reach any specified goal lane; in Checkers, we alternate between training one agent as A and B. Appendix H describes function augmentation in Stage 2 of CM3. COMA : the joint goal g and total reward n R n can be used to train COMA's global Q function, which receives input (s, o n, g n, n, a −n, g −n). Each output node i represents Q(s, a n = i, a −n, g). IAC : IAC trains each agent's actor and critic independently, using the agent's own observation. The TD error of value function V (o n, g n) is used in a standard policy gradient . QMIX : we used the original hypernetwork, giving all goals to the mixer and individual goals to each agent network. We used a manual coordinate descent on exploration and learning rate hyperparameters, including values reported in the original works. We ensured the number of trainable parameters are similar among all methods, up to method-specific architecture requirements for COMA and QMIX. Ablations. We conducted ablation experiments in all domains. To discover the speedup from the curriculum with function augmentation, we trained the full Stage 2 architecture of CM3 (labeled as Direct) without first training components π 1 and Q 1 in an induced MDP. To investigate the benefit of the new credit function and multi-goal policy gradient, we trained an ablation (labeled QV) with, where credit assignment between action-goal pairs is lost. QV uses the same π 1, Q 1, and function augmentation as CM3. CM3 finds optimal or near-optimal policies significantly faster than IAC and COMA on all domains, and performs significantly higher than QMIX in four out of five. We report absolute runtime in Appendix F and account for CM3's Stage 1 episodes (Appendix J) when comparing sample efficiency. Main comparison. Over all cooperative navigation scenarios (Figures 5a to 5c), CM3 (with 1k episodes in Stage 1) converged more than 15k episodes faster than IAC. IAC reached the same final performance as CM3 because dense individual rewards simplifies the learning problem for IAC's fully decentralized approach, but CM3 benefited significantly from curriculum learning, as evidenced by comparison to "Direct" in Figure 5f. QMIX and COMA settled at suboptimal behavior. Both learn global critics that use all goals as input, in contrast to CM3 and IAC that process each goal separately. This indicates the difficulty of training agents for individual goals under a purely global approach. While COMA was shown to outperform IAC in SC2 micromanagement where IAC must learn from a single team reward , our IAC agents have access to individual rewards that resolve the credit assignment issue and improve performance . In SUMO (Figure 5d), CM3 and QMIX found cooperative solutions with performances within the margin of error, while COMA and IAC could not break out of local optima where vehicles move straight but do not perform merge maneuvers. Since initial states force agents into the region of state space requiring cooperation, credit assignment rather than exploration is the dominant challenge, which CM3 addressed via the credit function, as evidenced in Figure 5i. IAC underperformed because SUMO requires a longer sequence of cooperative actions and gave much sparser rewards than the "Merge" scenario in cooperative navigation. We also show that centralized training of merely two decentralized agents allows them to generalize to settings with much heavier traffic (Appendix E). In Checkers (Figure 5e), CM3 (with 5k episodes in Stage 1) converged 10k episodes faster than COMA and QMIX to the global optimum with score 24. Both exploration of the combinatorially large joint trajectory space and credit assignment for path clearing are challenges that CM3 successfully addressed. COMA only solved Checkers among all domains, possibly because the small bounded environment alleviates COMA's difficulty with individual goals in large state spaces. IAC underperformed all centralized learning methods because cooperative actions that give no instantaneous reward are hard for selfish agents to discover in Checkers. These demonstrate CM3's ability to attain individual goals and find cooperative solutions in diverse multi-agent systems. Ablations. The significantly better performance of CM3 versus "Direct" (Figures 5f to 5j) shows that learning individual goal attainment prior to learning multi-agent cooperation, and initializing Stage 2 with Stage 1 parameters, are crucial for improving learning speed and stability. It gives evidence that while global action-value and credit functions may be difficult to train from scratch, function augmentation significantly eases the learning problem. While "QV" initially learns quickly to attain individual goals, it does so at the cost of frequent collisions, higher variance, and inability to maintain a cooperative solution, giving clear evidence for the necessity of the credit function. We presented CM3, a general framework for cooperative multi-goal MARL. CM3 addresses the need for efficient exploration to learn both individual goal attainment and cooperation, via a two-stage curriculum bridged by function augmentation. It achieves local credit assignment between action and goals using a credit function in a multi-goal policy gradient. In diverse experimental domains, CM3 attains significantly higher performance, faster learning, and overall robustness than existing MARL methods, displaying strengths of both independent learning and centralized credit assignment while avoiding shortcomings of existing methods. Ablations demonstrate each component is crucial to the whole framework. Our motivate future work on analyzing CM3's theoretical properties and generalizing to inhomogeneous systems or settings without known goal assignments. Hernandez-Leal, P., Kartal, B., and Taylor Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R. Instantiate N > 1 agents 8: Set all target network weights to equal main networks weights 13: Initialize exploration parameter = start and empty replay buffer B for each training episode e = 1 to E do for t = 1 to T do // execute policies in environment Sample action a n t ∼ π(a n t |o n t ; θ π,) for each agent. Compute global target for all n: Gradient descent on L(θ Qg) = if c = 1 then 29: end if 35: Update policy: Update all target network parameters using: Reset buffer B Off-policy training with a large replay buffer allows RL algorithms to benefit from less correlated transitions . The algorithmic modification for off-policy training is to maintain a circular replay buffer that does not reset (i.e. remove line 38), and conduct training (lines 24-41) while executing policies in the environment (lines 17-22). Despite introducing bias in MARL, we found that off-policy training benefited CM3 in SUMO and Checkers. By stationarity and relabeling t, the credit function can be written: Using the law of iterated expectation, the credit function satisfies the Bellman expectation equation: The goal-specific joint value function is the marginal of the credit function: The credit function can be expressed in terms of the goal-specific action-value function: First we state some elementary relations between global functions V π n (s) and Q π n (s, a). These carry over directly from the case of an MDP, by treating the joint policy π as as an effective "single-agent" policy and restricting attention to a single goal g n (standard derivations are included at the end of this section). We follow the proof of the policy gradient theorem : We can replace Q π n (s, a) by the advantage function A π n (s, a):= Q π n (s, a) − V π n (s), which does not change the expectation in Equation because: So the gradient can be written Recall that from, for any choice of agent label k ∈ [1..N]: Then substituting into: Now notice that the choice of k in is completely arbitrary, since holds for any k ∈ [1..N]. Therefore, it is valid to distribute A π n,k (s, a) into the summation in using the summation index m instead of k. Further summing over all n, we arrive at the of Proposition 2: The relation between V π n (s) and Q π n (s, a) in and are derived as follows: Let Q:= Q π (s, a, g) denote the centralized Q function, let π(a n):= π(a n |o n, g n) denote a single agent's policy, and let π(a −n):= π(a −n |o −n, g −n) denote the other agents' joint policy. In cooperative multi-goal MARL, the direct application of COMA has the following gradient. Define the following: f n ]. its variance can be derived to be : The CM3 gradient can be rewritten as As before, z m:= ∇ θ log π(a m). Define h nm:= z m (Q n − b nm (s)) and let h n:= m h nm. Then the variance is A greedy initialization can provide significant improvement in multi-agent exploration versus naïve random exploration, as shown by a simple thought experiment. Consider a two-player MG defined by a 4 × 3 gridworld with unit actions (up, down, left, right). Agent A starts at with goal, while agent B starts at with goal. The greedy policy for each agent in MG is to move horizontally toward its target, since this is optimal in the induced M (when the other agent is absent). Case 1: Suppose that for ∈, A and B follow greedy policies with probability 1 −, and take random actions (p(a) = 1/4) with probability. Then the probability of a symmetric optimal trajectory is P (cooperate) = 2 2 ((1 −) + /4) 8. For = 0.5, P (cooperate) ≈ 0.01. Case 2: If agents execute uniform random exploration, then P (cooperate) = 3.05e-5 0.01. We investigated whether policies trained with few agent vehicles (N = 2) on an empty road can generalize to situations with heavy SUMO-controlled traffic. We also tested on initial and goal lane configurations (C3 and C4) which occur with low probability when training with configurations C1 and C2. Table 1 shows the sum of agents' reward, averaged over 100 test episodes, on these configurations that require cooperation with each other and with minimally-interactive SUMOcontrolled vehicles for success. CM3's higher performance than IAC and COMA in training is reflected by better generalization performance on these test configurations. There is almost negligible decrase in performance from train Figure 5d to test, giving evidence to our hypothesis that centralized training with few agents is feasible even for deployment in situations with many agents, for certain applications where local interactions are dominant. F ABSOLUTE RUNTIME CM3's higher sample efficiency does not come at greater computational cost, as all methods' runtimes are within an order of magnitude of one another. Test times have no significant difference as all neural networks were similar. The full Markov game for each experimental domain, along with the single-agent MDP induced from the Markov game, are defined in this section. In all domains, each agent's observation in the Markov game consists of two components, o self and o others. CM3 leverages this decomposition for faster training, while IAC, COMA and QMIX do not. This domain is adapted from the multi-agent particle environment in. Movable agents and static landmarks are represented as circular objects located in a 2D unbounded world with real-valued position and velocity. Agents experience contact forces during collisions. A simple model of inertia and friction is involved. State. The global state vector is the concatenation of all agents' absolute position (x, y) ∈ R 2 and velocity (v x, v y) ∈ R 2. Observation. Each agent's observation of itself, o self, is its own absolute position and velocity. Each agent's observation of others, o others, is the concatenation of the relative positions and velocities of all other agents with respect to itself. Actions. Agents take actions from the discrete set do nothing, up, down, left, right, where the movement actions produce an instantaneous velocity (with inertia effects). Goals and initial state assignment. With probability 0.2, landmarks are given uniform random locations in the set (−1, 1) 2, and agents are assigned initial positions uniformly at random within the set (−1, 1) 2. With probability 0.8, they are predefined as follows (see Figure 3). In "Antipodal", landmarks for agents 1 to 4 have (x, y) coordinates [(0.9,0.9), (-0.9,-0.9), (0.9,-0.9), (-0.9,0.9)], while agents 1 to 4 are placed at [(-0.9,-0.9), (0.9,0.9), (-0.9,0.9), (0.9,-0.9)]. In "Intersection", landmark coordinates are [(0.9,-0.15), (-0.9,0.15), (0.15,0.9), (-0.15,-0.9 Reward. At each time step, each agent's individual reward is the negative distance between its position and the position of its assigned landmark. If a collision occurs between any pair of agents, both agents receive an additional -1 penalty. A collision occurs when two agents' distance is less than the sum of their radius. Termination. Episode terminates when all agents are less than 0.05 distance from assigned landmarks. Induced MDP. This is the N = 1 case of the Markov game, used by Stage 1 of CM3. The single agent only receives o self . In each episode, its initial position and the assigned landmark's initial position are both uniform randomly chosen from (−1, 1) 2. We constructed a straight road of total length 200m and width 12.8m, consisting of four lanes. All lanes have width 3.2m, and vehicles can be aligned along any of four sub-lanes within a lane, with lateral spacing 0.8m. Vehicles are emitted at average speed 30m/s with small deviation. Simulation time resolution was 0.2s per step. SUMO file merge_stage3_dense.rou.xml contains all vehicle parameters, and merge.net.xml defines the complete road architecture. State. The global state vector s is the concatenation of all agents' absolute position (x, y), normalized respectively by the total length and width of the road, and horizontal speed v normalized by 29m/s. Observation. Each agent observation of itself o n self is a vector consisting of: agent speed normalized by 29m/s, normalized number of sub-lanes between agent's current sub-lane and center sub-lane of goal lane, and normalized longitudinal distance to goal position. Each agent's observation of others o n others is a discretized observation tensor of shape centered on the agent, with two channels: binary indicator of vehicle occupancy, and normalized relative speed between agent and other vehicles. Each channel is a matrix with shape, corresponding to visibility of 15m forward and backward (with resolution 2.5m) and four sub-lanes to the left and right. Actions. All agents have the same discrete action space, consisting of five options: no-op (maintain current speed and lane), accelerate (2.5m/s 2), decelerate (−2.5m/s 2), shift one sub-lane to the left, shift one sub-lane to the right. Each agent's action a n is represented as a one-hot vector of length 5. Goals and initial state assignment. Each goal vector g n is a one-hot vector of length 4, indicating the goal lane at which agent n should arrive once it crosses position x=190m. With probability 0.2, agents are assigned goals uniformly at random, and agents are assigned initial lanes uniformly at random at position x=0. With probability 0.8, agent 1's goal is lane 2 and agent 2's goal is lane 1, while agent 1 is initialized at lane 1 and agent 2 is initialized at lane 2 (see Figure 4). Departure times were drawn from a normal distribution with mean 0s and standard deviation 0.5s for each agent. Reward. The reward R(s t, a t, g n) for agent n with goal g n is given according to the conditions: -1 for a collision; -10 for time-out (exceed 33 simulation steps during an episode); 10(1 − ∆) for reaching the end of the road and having a normalized sub-lane difference of ∆ from the center of the goal lane; and -0.1 if current speed exceeds 35.7m/s. Termination. Episode terminates when 33 simulation steps have elapsed or all agents have x >190m. Induced MDP. This is the N = 1 case of the Markov game defined above, used by Stage 1 of CM3. The single agent receives only o self. For each episode, agent initial and goal lanes are assigned uniformly at random from the available lanes. This domain is adapted from the Checkers environment in. It is a gridworld with 5 rows and 13 columns (Figure 2). Agents cannot move to the two highest and lowest rows and the two highest and lowest columns, which are placed for agents' finite observation grid to be well-defined. Agents cannot be in the same grid location. Red and yellow collectible reward are placed in a checkered pattern in the middle 3x8 region, and they disappear when any agent moves to their location. State. The global state s consists of two components. The first is s T, a tensor of shape, where the two "channels" in the last dimension represents the presence/absence of red and yellow rewards as 1-hot matrices. The second is s V, the concatenation of all agents' (x, y) location (integer-valued) and the number of red and yellow each agent has collected so far. Observation. Each agent's obsevation of others, o n others, is the concatenation of all other agents' normalized coordinates (normalized by total size of grid). An agent's observation of itself, o n self, consists of two components. First, o n self,V is a vector concatenation of agent n's normalized coordinate and the number of red and yellow it has collected so far. Second, o n self,T is a tensor of shape, centered on its current location in the grid. The tensor has three "channels", where the first two represent presence/absence of red and yellow rewards as 1-hot matrices, and the last channel indicates the invalid locations as a 1-hot matrix. The agent's own grid location is a valid location, while other agents' locations are invalid. Actions. Agents choose from a discrete set of actions do-nothing, up, down, left, right. Movement actions transport the agent one grid cell in the chosen direction. Goals. Agent A's goal is to collect all red rewards without touching yellow. Agent B's goal is to collect all yellow without touching red. The goal is represented as a 1-hot vector of length 2. Reward. Agent A gets +1 for red, -0.5 for yellow. Agent B gets -0.5 for red, +1 for yellow. For all experiment domains, ReLU nonlinearity was used for all neural network layers unless otherwise specified. All layers are fully-connected feedforward layers, unless otherwise specified. All experiment domains have a discrete action space (with |A| = 5 actions), and action probabilities were computed by lower-bounding softmax outputs of all policy networks by P (a n = i) = (1 −)softmax(i) + /|A|, where is a decaying exploration parameter. To keep neural network architectures as similar as possible among all algorithms, our neural networks for COMA differ from those of in that we do not use recurrent networks, and we do not feed previous actions into the Q function. For the Q network in all implementations of COMA, the value of each output node i is interpreted as the action-value Q(s, a −n, a n = i, g) for agent n taking action i and all other agents taking action a −n. Also for COMA, agent n's label vector (one-hot indicator vector) and observation o self were used as input to COMA's global Q function, to differentiate between evaluations of the Q-function for different agents. These were choices in COMA. COMA uses the same policy network as Stage 2 of CM3. The global Q function of COMA computes Q(s, (a n, a −n)) for each agent n as follows. Input is the concatenation of state s, all other agents' 1-hot actions a −n, agent n's goal g n, all other agent goals g −n, agent label n, and agent n's observation o n self. This is passed through two layers of 128 units each, then connected to a linear output layer with 5 units. The Q 1 function in Stage 1 feeds the concatenation of state s, goal g, and 1-hot action a to one layer with 256 units, which is connected to the special layer h The Q 1 function in Stage 1 is defined as: state tensor s T is fed to a convolutional layer with 4 filters of size 3x5 and stride 1x1 and flattened. o n self,T is given to a convolution layer with 6 filters of size 3x3 and stride 1x1 and flattened. Both are concatenated with s n (agent n part of the s V vector), goal g n, action a n and o COMA. COMA uses the same policy network as Stage 2 of CM3. The global Q(s, (a n, a −n)) function of COMA is defined as follows for each agent n. Tensor part of global state s T is given to a convolutional layer with 4 filters of size 3x5 and stride 1x1. Tensor part of agent n's observation o n self,T is given to a convolutional layer with 6 filters of size 3x3 and stride 1x1. Outputs of both convolutional layers are flattened, then concatenated with s V, all other agents' actions a −n, agent n's goal g n, other agents' goals g −n, agent n's label vector, and agent n's vector observation o n self,V. The concatenation is passed through two layers with 256 units each, then to a linear output layer with 5 units. QMIX. Individual value functions are defined as: o n self,T is passed through the same convolutional layer as above, connected to hidden layer with 32 units, then concatenated with o n self,V, a n t−1, and g n. This is connected to layer h 2 with 64 units. o n others is connected to a layer with 64 units then connectd to h 2. h 2 is fully-connected to an output layer. The mixing network feeds s T into the same convolutional network as above and follows the exact architecture of with embedding dimension 128. We used the Adam optimizer in Tensorflow with hyperparameters in Tables 3 to 5. div is used to compute the exploration decrement step:= (start − end)/ div. 5e2 1e3 1e3 1e4 2e4 1e4 1e4 Replay buffer 1e4 1e4 1e4 1e4 1e4 1e4 1e4 Minibatch size 128 128 128 128 128 128 128 Steps per Stage 1 training curves for all three experimental domains are shown in Figure 6.' it should be close to 'A modular method for fully cooperative multi-goal multi-agent reinforcement learning, based on curriculum learning for efficient exploration and credit assignment for action-goal interactions.' but not to 'We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.'.

Question: Catastrophic forgetting in neural networks is one of the most well-known problems in continual learning. Previous attempts on addressing the problem focus on preventing important weights from changing. Such methods often require task boundaries to learn effectively and do not support backward transfer learning. In this paper, we propose a meta-learning algorithm which learns to reconstruct the gradients of old tasks w.r.t. the current parameters and combines these reconstructed gradients with the current gradient to enable continual learning and backward transfer learning from the current task to previous tasks. Experiments on standard continual learning benchmarks show that our algorithm can effectively prevent catastrophic forgetting and supports backward transfer learning. The ability to learn continually without forgetting previously learned skills is crucial to artificial general intelligence (AGI) BID3. Addressing catastrophic forgetting in artificial neural networks (ANNs) has been the top priority of continual learning research. Notable attempts on solving the problem include Elastic Weight Consolidation (EWC) by BID2 and the follow up work on Synaptic Intelligence (SI) by BID6, and Memory Aware Synapse (MAS) by BID0. These algorithms share the same core idea: preventing important parameters from deviating from their old (presumably better) values. In order to achieve that, EWC-like algorithms compute the importance of each parameter w.r.t. each task in the sequence and for each old task, a regularization term is added to the loss of the new task to prevent that task from being catastrophically forgotten. The regular-Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute. ization term for task T (i) in EWC-like algorithms takes the following form: DISPLAYFORM0 where λ (i) controls the relative importance of task i to the current task, θ is the current parameters, θ (i) * is the parameters found at the end of the training of T (i), and ω DISPLAYFORM1 j is the importance of parameter θ 1. The regularizer in Eqn. 1 prevent changes to important parameters regardless of the effect of these changes. Unless θ DISPLAYFORM2 is the optimal value for the j-th parameter, either increasing or decreasing its value will in better performance on task i. Keeping θ close to θ (i) * only prevent the network from catastrophically forgetting T (i) but cannot help the network to leverage the information from the current task T (k), k > i to improve its performance on T (i) and other previous tasks. In other words, regularizers of the form in Eqn. 1 do not support backward transfer learning.2. The number of old parameter and importance vectors, θ * and ω, grows linearly with the number of tasks, making EWC-like algorithms not scalable to a large number of tasks. BID5 proposed the online EWC algorithm which maintains only one copy of θ * and ω. The sizes of θ * and ω are equal to that of the network. Therefore, the memory requirement of online EWC is still considerably large for large networks. To address these limitations of EWC-like algorithms, we propose a meta learning algorithm which:1. Learns to approximate the gradient of a task w.r.t. the current parameters from the current parameters 2. Combines the approximated gradients of old tasks w.r.t. the current parameters and the current task's gradient to in an update that improves the performance of the network on all tasks. By combining the gradients, our algorithm exploits the similarity between the current task and previous tasks to enable backward transfer learning. As described in section 2.2 and 5.2, the size of a meta-network is typically orders of magnitude smaller than that of the main network and metanetworks for different tasks can be distilled into a single meta-network in an online manner. That significantly reduces the memory requirement of our method. In the next section, we introduce our learning to learn algorithm for continual learning. Experiments are presented in section 3. Conclusions and future work are located in section 4 and 5, respectively. Let us consider a continual learning problem with a learner f (x; θ): DISPLAYFORM0, DISPLAYFORM1, and T loss functions DISPLAYFORM2 To avoid clutter, we remove the input of the loss function L, and DISPLAYFORM3 In joint learning settings, data from all tasks is available to the learner. The parameter θ is updated using the average of gradients from all tasks: DISPLAYFORM4 where α is the learning rate, and DISPLAYFORM5 Generally, updating θ with δ will improve the performance of f on all T tasks. In continual learning settings, at task t+1, the learner cannot access to D (i), i = 1,..., t and cannot compute ∇ θ L (i), i = 1,..., t. The update at task t + 1 is computed from ∇ θ L (t+1)only. When θ is updated with this gradient, f's performance on T (t+1) will be improved while f's performance on tasks T (i), i = 1,..., t might be catastrophically damaged.1 The analysis here still applies to the case where mini-batches are used because the expectation of DISPLAYFORM6 To address this problem, we propose the following meta learning algorithm. During task i, we train meta-network DISPLAYFORM7 In subsequent tasks, h (i) is used to reconstruct the gradient of task i w.r.t. the current parameters without having to access to D (i). More concretely, h (i) learns to map the parameter to the corresponding gradient: DISPLAYFORM8 When the main network f is trained on a new task DISPLAYFORM9 Section 2.3 introduces several ways to combine predicted gradients with the current gradient to prevent catastrophic forgetting and enable backward transfer learning. For our method to work when optimizers other than SGD is used to train the main network, ∇ θ L (i) should be replaced with the update vector produced by the optimizer. Because a real world neural network typically contains tens of thousands to billions of parameters, the naive way of training h would require an astronomically large number of samples of θ and ∇ to cover a very high dimensional space. A fully connected meta-network h also need to be extremely large to receive a very high dimensional input and produce a very high dimensional output. To circumvent the problem, we follow the coordinate-wise approach proposed by BID1 where each coordinate is processed independently. h is a neural network that takes in a 1-dimensional input and produces 1-dimensional output DISPLAYFORM0 The procedure is applied to all coordinates in θ. In our experiments, hs are MLPs and are trained to minimize the Euclidean distance between h(θ j ; φ) and ∇ j for all θ j in θ. h could be modified to process more inputs such as the position of parameter in the network or the previous values of θ j. It is also possible for h to process a small set of related parameters simultaneously, e.g. parameters in the same filter of a CNN. However, we leave these variations for future work. Let us consider a pair of gradients ∇ (k) and DISPLAYFORM0 and ∇ j have different signs and α is small enough, then updating f with ∇ (k) j will improve the network's performance on task k and damage its performance on task i. If they have the same sign then the update will improve the performance on both tasks. That intuition leads to the following rule to create an update vector from a pair of gradients: DISPLAYFORM0 At task t+1, an update vector δ can be produced by applying the above rule the pair between ∇ (t+1) and all other gradients∇ (i), i = 1,..., t. When t is large, that method usually in a sparse update vector. In practice, we apply the rule to the pair ∇ (t+1),∇ 1:t where∇ 1:t = 1 t t i=1∇(i). Updating the main network with δ will improve the performance on task t + 1 and will likely to improve the performance on tasks 1,..., t. The update vector δ contains information that are common between task t + 1 and previous tasks. Updating f with δ transfers information from the current task to previous tasks. δ is the medium for backward transfer learning in our algorithm. We tested our algorithm on the Permuted MNIST dataset BID2 ). To better demonstrate the effect of backward transfer learning, we train each task for only 2000 iterations to prevent the main network from reaching its maximum performance. The is shown in FIG1.The network in FIG1 suffers from catastrophic forgetting problem: the performance on old tasks decrease rapidly when new tasks are trained. The network trained with our algorithm FIG1 ) does not suffer from catastrophic forgetting: the performance on old tasks is maintained or even improved when new tasks are trained. The performance improvement on old tasks suggests that our algorithm has backward transfer learning capability. We also note the forward transfer learning phenomenon in FIG1: the starting accuracy of a later task is higher than that of former ones. In this paper, we present a meta learning algorithm for continual learning. Experiments on Permuted MNIST dataset show that our algorithm is effective in preventing catastrophic forgetting and is capable of supporting backward transfer learning. To make our algorithm works without task boundaries, we need to detect boundaries automatically. The simplest way to detect task boundaries is to look at the loss of the main network. That way, however, does not work well when different tasks uses different loss functions or have different input, output scales. We propose to use detect task boundaries using the loss of the meta-networks. Different tasks

Represent the TLDR for finding the corresponding scientific article

We propose a meta learning algorithm for continual learning which can effectively prevent catastrophic forgetting problem and support backward transfer learning.

Represent the short summary for finding the corresponding scientific article (long)

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the scientific article to find a short TLDR

Question: We give a formal procedure for computing preimages of convolutional network outputs using the dual basis defined from the set of hyperplanes associated with the layers of the network. We point out the special symmetry associated with arrangements of hyperplanes of convolutional networks that take the form of regular multidimensional polyhedral cones. We discuss the efficiency of of large number of layers of nested cones that from incremental small size convolutions in order to give a good compromise between efficient contraction of data to low dimensions and shaping of preimage manifolds. We demonstrate how a specific network flattens a non linear input manifold to an affine output manifold and discuss it's relevance to understanding classification properties of deep networks. Deep convolutional networks for classification map input data domains to output domains that ideally correspond to various classes. The ability of deep networks to construct various mappings has been the subject of several studies over the years (1; 3; 10) and in general ed in various estimates of capacity given a network structure. The actual mappings that are learnt by training a specific network however, often raise a set of questions such as why are increasingly deeper networks advantageous (13; 14)? What are the mechanisms responsible for the successful generalisation properties of deep networks? Also the basic question why deep learning over large datasets is so much more effective than earlier machine learning approaches is still essentially open, BID6. These questions are not in general answered by studies of capacity. A more direct approach based on actual trained networks and the mappings they are efficiently able to produce seems needed in order to answer these questions. It seems ever more likely e.g that the ability of deep networks to generalize is connected with some sort of restriction of mappings that they theoretically can produce and that these mappings are ideally adapted to the problem for which deep learning has proven successful, Due to the complexity of deep networks the actual computation of how input domains are mapped to output classifiers has been considered prohibitively difficult. From general considerations of networks with rectifier (ReLU) non linearities we know that these functions must be piecewise linear BID9 but the relation between network parameters such has convolutional filter weights and fully connected layer parameters and the actual functions remains largely obscure. In general, work has therefore been concentrated on empirical studies of actual trained networks (6; 8; 9) Recently however there have been attempts to understand the relation between networks and their mapping properties from a more general and theoretical point of view. This has included specific procedures for generating preimages of network outputs BID3 and more systematic studies of the nature of piecewise linear functions and mappings involved in deep networks, (2; 11; 15).In this work we will make the assertion that understanding the geometry of deep networks and the manifolds of data they process is an effective way to understand the comparative success of deep networks. We will consider convolutional networks with ReLU non linearities. These can be completely characterised by the corresponding hyperplanes associated with individual convolutional kernels. We will demonstrate that the individual arrangement of hyperplanes inside a layer and the relative arrangement between layers is crucial to the understanding the success of various deep network structures and how they map data from input domains to output classifiers. We will consider only the convolutional part of a deep network with a single channel. We will assume no subsampling or max pooling. This will allow us to get a clear understanding of the role of the convolutional part. A more complete analysis involving multiple channels and fully connected layers is possible but more complex and will be left to future work. The focus of our study is to analyse how domains of input data are mapped through a deep network. A complete understanding of this mapping and its inverse or preimage will give a detailed description of the workings of the network. Since we are not considering the final fully connected layers we will demonstrate how to compute in detail the structure of input data manifold that can be mapped to a specified reduced dimensionality affine manifold in the activity space of the final convolutional output layer. This flattening of input data is often considered as a necessary preprocessing step for efficient classification. The understanding of mappings between layers will be based on the specific understanding of how to compute preimages for networks activities. We will recapitulate and extend the work in based on the construction of a dual basis from an arrangement of hyperplanes. By specialising to convolutional networks we will demonstrate that the arrangement of hyperplanes associated with a specific layer can be effectively described by a regular multidimensional polyhedral cone oriented in the identity direction in the input space of the layer. Cones associated with successive layers are then in general partly nested inside their predecessor. This leads to efficient contraction and shaping of the input domain data manifold. In general however contraction and shaping are in conflict in the sense that efficient contraction implies less efficient shaping. We will argue that this' conflict is resolved by extending the number of layers of the network with small incremental updates of filters at each layer. The main contribution of the paper is the exploitation of the properties of nested cones in order to explain how non linear manifolds can be shaped and contracted in order to comply with the distribution of actual class manifolds and to enable efficient preprocessing for the final classifier stages of the network. We will specifically demonstrate the capability of the convolutional part of the network to flatten non linear input manifolds which has previously been suggested as an important preprocessing step in object recognition, (5; 12) Transformations between layers in a network with ReLU as nonlinear elements can be written as DISPLAYFORM0 Where [] + denotes the ReLU function max(0, x i). applied component wise to the elements of the input vector x which will be confined to the positive orthant of the d-dimensional Euclidean input space. It divides the components of the output vector y into two classes depending on the location of the input x: DISPLAYFORM1 In order to analyse the way domains are mapped through the network we will be interested in the set of inputs x that can generate a specific output y. DISPLAYFORM2 This set, known as the preimage of y can be empty, contain a unique element x or consist of a whole domain of the input space. This last case is quite obvious by considering the ReLU nonlinearity that maps whole half spaces of the input domain to 0 components of the output y. The preimage will depend on the location of the input relative to the arrangement of the hyperplanes defined by the affine part of the mapping: DISPLAYFORM3 These hyperplanes divides the input space into a maximum of 2 d number of different cells with the maximum attained if all hyperplanes cut through the input space which we take as the non negative orthant of the d-dimensional Euclidean space R d +. Understanding the arrangement of these hyperplanes in general and especially in the case of convolutional mappings will be central to our understanding of how input domains are contracted and collapsed through the network. The preimage problem can be treated geometrically using these hyperplanes as well as the constraint input domains defined by these. For a given output y we can denote the components where y j > 0 as y j1, y j1... y jq and the complementary index set where y i = 0 as i 1, i 1... i p With each positive component of y we can associate a hyperplane: DISPLAYFORM4 which is just the hyperplane Π j translated with the output y j For the 0-components of y we can define the half spaces DISPLAYFORM5 I.e the half space cut out by the negative side of the plane Π i. These planes and half spaces together with the general input domain constraint of being inside R + d define the preimage constraints given the output y. If we define the affine intersection subspace: DISPLAYFORM6 and the intersection of half spaces: DISPLAYFORM7 the preimage of y can be defined as: DISPLAYFORM8 The constraint sets and the preimage set is illustrated in FIG0 for the case of d = 3 and various outputs y with different number of 0-components. For fully connected networks, computing the preimage set amounts to finding the intersection of an affine subspace with a polytope in d− dimensional space. This problem is known to be exponential in d and therefore intractable. However, we will see that this situation is changed substantially when we consider convolutional instead of fully connected networks. In order to get more insight into the nature of preimages we will devise a general method of computing that highlights the nature of the arrangement of hyperplanes. The set of hyperplanes Π i, i = 1... d will be assumed to be in general position, i.e. no two planes are parallel. The intersection of all hyperplanes excluding plane i: DISPLAYFORM0 is then a one-dimensional affine subspace S i that is contained in all hyperplanes Π j excluding j = i. For all i we can define vectors e i in R d parallel to S i. The general position of the hyperplanes then guarantees that the set e i is complete in R d. By translating all vectors e i to the point in R d which is the mutual intersection of all planes Π i DISPLAYFORM1 they can therefore be used as a basis that spans R d. This construction also has the property that the intersection of the subset of hyperplanes: DISPLAYFORM2 is spanned by the complementary dual basis set DISPLAYFORM3 The dual basis can now be used to express the solution to the preimage problem. The affine intersection subspace P * associated with the positive components j 1 j 2... j q of the output y is spanned by the complementary vectors associated with the negative components i 1 i 2... i p. These indices also define the hyperplanes Π 1, Π 2... Π p that constrain the preimage to lie in the intersections of half spaces associated with the negative sides. We now define the positive direction of the vector e i as that associated with the negative side of the plane Π i. If we consider the intersection of the subspace P * and the subspace generated by the intersections of the hyperplanes Π i associated with the negative components of y we get: DISPLAYFORM4 Due to complementarity of the positive and negative indices, this is a unique element x * ∈ R d (marked "output" in figure 1 which lies in the affine subspace of the positive output components P * as well as on the intersection of the boundary hyperplanes Π i that make up the half space intersection constraint X − for the preimage. if we take the subset of the dual basis vectors with e i1, e i2 . . . e ip and move them to this intersection element, they will span the part of the negative constraint region X − associated with the preimage. I.e. the preimage of the output y is given by: DISPLAYFORM5 We will now specialise to the standard case of convolutional networks. In order to emphasize the basic role of geometric properties we will consider only a single channel with no subsampling. Most of what we state will generalize to the more general case of multiple channels with different convolutional kernels but needs a more careful analysis we will exploit the fact that convolutional matrices are in most respects asymptotically equivalent to those of circulant matrices where each new row is a one element cyclic shift of the previous. For any convolution matrix we will consider the corresponding circulant matrix that appends rows at the end to make it square and circulant. Especially when the support of the convolution is small relative to the dimension d, typically the order of 10 in relation to 1000, this approximation will be negligible. Except for special cases the corresponding circulant will be full rank, which means that properties about dual basis etc. derived previously will apply also here. As is standard we will assume that the bias b is the same for all applications of the convolution kernels. The first thing to note about hyperplanes associated with circulant matrices is that they all intersect on the identity line going through the origin and the point (1, 1, . . . 1). Denote the circulant matrix as C with elements c i,j. The circulant property implies c i+1, DISPLAYFORM0. Each row is shifted one step cyclically relative to the previous. For the hyperplane corresponding to row i we have: DISPLAYFORM1 It is easy to see that the circulant property implies that the sum of all elements along a row is the same for all rows. Let the sum of the row be a. We then get: x j = −b/a for j = 1... d as a solution for this system of equations which is a point on the identity line in R d.The arrangement of the set of hyperplanes: DISPLAYFORM2 with w T i the i:th row of the circulant augmented convolutional matrix W, will be highly regular. Consider a cyclic permutation P x of the components of the input x described by the single shift matrix P i.e x i is mapped to x i+1 for i = 1... d − 1 and x d is mapped to x 1. We then get: DISPLAYFORM3 which states that points on the hyperplane associated with weights w i are mapped to hyperplane associated with weights w i+1. The hyperplanes associated with the weights w i i = 1... d therefore form a regular multidimensional polyhedral cone in R d around the identity line, with the apex located at x T = (−b/a, −b/a . . . − b/a) controlled by the bias b and the sum of filter weights a. Geometrically, the cone is determined by the apex location, the angle of the planes to the central identity line and its rotation in d-dimensional space. Apex location and angle are two parameters which leaves d − 2 parameters for the multidimensional rotation in R d. This maximum degree of freedom is however attained only for unrestricted circulant transformations. The finite support of the convolution weights in CNN:s will heavily restrict rotations of the cone. The implications of this will be discussed later. Any transformation between two layers in a convolutional network can now be considered as a mapping between two regular multidimensional polyhedral cones that are symmetric around the identity line in R d. The coordinate planes of the input space R d + can be modelled as such a cone as well as the output space given by the convolution. The strong regularity of these cones will of course impose strong regularities on the geometric description of the mapping between layers. Just as in the general case, this transformation will be broken down to transformations between intersection subspaces of the two cones. In order to get an idea of this we will start with a simple multi layer network with two dimensional input and output and a circulant transformation: FIG1 illustrates the mapping of data from the input space (x 1, x 2) to the output space (y 1, y 2) for two networks with 3 and 6 layers respectively. The dashed lines represent successive preimages of data that maps to a specific location at a layer. By connecting them we get domains of input data mapped to the same output at the final layer, i.e they are contraction flows depicting how data is moved through the network. Note that in both layers the major part of the input domain is mapped to output. This is illustrated for the first trivial bias only layer with a = 1, b = 0. The domain of the input that is mapped to output domain is just quite trivial planar manifolds. DISPLAYFORM0 The second network with more varied weights illustrates how input domain manifolds with more structure can be created. It also demonstrates the importance of the concept of "nested cones" and how this affects the input data manifolds. The red lines represent data that is associated with layer cones that are completely nested inside its predecessors, while the black lines represent data where the succeeding cone has a wider angle than its predecessor. When this happens, the hyperplanes associated with the output cone will intersect the hyperplanes of the input cone and input data beyond this intersection is just transformed linearly. Since all data in figure 2 is remapped to the input space this has the effect that data is not transformed at all. This has no effect at all at the shaping of the input manifold. One could say that these layers are "wasted" beyond the location of the intersection as far as network properties are concerned since they neither contribute to the shaping or the contraction of input data manifolds. The effect of this on the input manifold can be seen as a less diverse variation of its shape (black) compared to the previous part associated with the completely nested part of the layer cones. In higher dimensions the effects of nested vs. partially nested cones appear in the same way but more elaborate. In addition to the 2d case we also have to consider rotations of the cone, which as was pointed out earlier, has d − 2 degrees of freedom for cones in d dimensional space. The effects of contraction of data from higher to lower dimensions also become more intricate as the number of different subspace dimensionalities increases. Most of these effects can be illustrated with 3 dimensional input and output spaces. For d = 3 the generic circulant matrix can be used to define a layer transformation: DISPLAYFORM1 The transformation properties of this network are most easily illustrated if we start with the pure bias case with transformation W = I, i.e a = 1, b = 0, c = 0. A specific element in input space is mapped according to its position relative to the hyperplanes. If we use the dual basis to define the coordinates of the output data, the mapping for the input element will be the same in input cells with the same relation to all hyperplanes. In d dimensions, the hyperplanes divide the input space into 2 d cells where elements are mapped to a specific associated intersection subspace in the output domain. In FIG2, the grey shaded boxes indicate two cells with different numbers of negative constraints 1 and 2 respectively. The content of the upper one with one negative constraint including all its bounding faces and their intersections is mapped to a specific 2d hyperplane in the output domain while the content of the lower one with two negative constraints is mapped to the 1d intersection of two hyperplanes. This illustrates the most important property of the nesting of the cones associated with the input and output layer: For a range of transformations in the vicinity of the identity mapping, the input space, properly normalised in range, is divided into cells where the elements of the cells including their bounding faces an their intersections are mapped to output intersection subspaces with equal or lower dimension. This means that the content of the cell is irreversibly contracted to lower dimensions. and the components of the dual bases used to span these. Note that these are affected by changing the angle of the output cone. It introduces a limit of the nesting beyond which the mapping properties of the transformation are changed so that data no longer maps to a manifold of equal or lower dimensionality. I.e the contraction property is lost in those regions of the input space where nesting of cones ceases. We will formally define this important property of nested cones as:Let R We see that this definition implies that the cone formed by planes Π is completely contained in that formed by the planes Π 0 but also that its relative rotation is restricted. We will have reason to relax the condition of inclusion of all elements i 1... i p in the intersection subset and talk about cones with restricted nesting. Complete nesting implies contraction of data from one layer to the next which can be seen from the fact that all elements of the complete intersection subset Π 0 i1 ∩... ∩ Π 0 ip and thereby in each subset are mapped to the intersection subset M i1...ip = Π i1 ∩... ∩ Π ip with same dimensionality. In addition elements from intersection subsets formed by subsets of indexes i 1... i p will also be mapped to this same intersection subset. The subsets associated with these indexes are however of higher dimension. Consequently, mapping of data between layers will be from intersection subsets to intersection subsets with equal or lower dimension. This is the crucial property connecting degree of nesting with degree of contraction. By going further through the network to higher layers this contraction is iterated and data is increasingly concentrated on intersection subspaces with lower dimension which is reflected by the increased sparsity of the nodes of the network. The convolutional part of a deep network can therefore be seen as a component in a metric learning system where the purpose of the training will be to create domains in input space associated with different classes that are mapped to separate low dimensional outputs as a preprocessing for the final fully connected layers that will make possible efficient separation of classes. There is therefore a conflict between the diversity of input manifolds that contract to low dimensional outputs and the degree of contraction that can be generated in a convolutional network. The efficient resolution of this conflict seems to lie in increasing the number of layers in the network in order to be able to shape diverse input manifolds but with small incremental convolution filters that retain the nesting property of the cone in order to preserve the proper degree of contraction. Empirically, this is exactly what has been demonstrated to be the most efficient way to increase performance of deep networks (13; 14). We are now in a position to give a general characterizaton of the preimage corresponding to a specified output domain at the final convolutional output layer assuming the property of nested layer cones. Ideally we would like to include the final fully connected layers in this analysis but it will require a special study since we cannot assume the nesting property to be valid for these. In the end the network should map different classes to linearly separable domains in order to enable efficient classification. It is generally suggested that the preprocessing part of a network corresponds to flattening nonlinear input manifolds in order to achieve this final separation at the output. In order to be able to draw as general as possible we shall demonstrate the exact structure of a nonlinear input manifold that maps to a prespecified affine manifold at the final convolutional layer. We denote this manifold M and the output at the final convolutional layer by x (l). The final layer can be characterised by the set of hyperplanes: DISPLAYFORM0 Let the zero components of the output x (l) be i 1, i 2... i q. It can then be associated with the intersection of the output manifold and the corresponding hyperplanes DISPLAYFORM1 The degree of intersection q will depend on the dimensionality of M. If M is a d − 1 dimensional hyperplane in general position it will intersect any combination of hyperplanes at output level l. This is the maximum complexity situation that will generate a d − 1 -dimensional input manifold. Reducing dimensionality of M means reducing the possible intersection with combinations of hyperplanes. Note that if M intersects the set the intersection of planes i 1, i 2... i p it also intersects the intersection of any subset of these. Intersecting M with each of these subsets will generate pieces of intersections linked together. These are affine subsets with different dimensionality and the preimage of each piece will be generated by complementary dual basis components. This is illustrated by figure 4 for the case of an affine plane in R 3 + intersecting to give a triangular domain. In this case we have three points on the coordinate axis, and three lines connecting these. The three points will all span 2d planes bases on different pairs of complementary dual basis components. In addition to these, the points on the lines of the triangle generated by intersecting M with each of the three individual output planes will generate 1d lines that jointly will span a 2d plane. This plane will connect continuously with the planes spanned from the points on the axis to yield a piecewise planar input manifold to the final layer. Continuing through the network, this piecewise planar manifold will intersect with the planes of layer l − 1 and the procedure is iterated until we reach the input layer. This procedure generalises to arbitrary dimensions but the complexity of course grows with the increasing combinatorics. The basic principle of layer by layer recursively generating piecewise affine manifolds still holds. The complexity lies in the fact that each intersection of the manifold M with every subset of possible hyperplane intersections will generate a seeding hyperplane and and each of these will act as a new manifold M at the next layer. Note however that the nested cone property substantially reduces complexity compared to the general case of arbitrary hyperplanes. Figure 4: Piecewise planar manifold in 3d input space that maps to affine manifold (blue triangle) at the final convolutional layer in a 3-node 3-layer network with circulant transformations. All data is remapped to the input space. Left: Red patches are mapped to 0 dimensional red points at the three output coordinate axis Blue patches are mapped to 1d lines connecting the points. (dark red is outside light red is inside of the manifold Right: Patches that are generated by selective components of the dual basis at each layer. The positive span generated by selective components of the dual basis emanating from the red output points on the triangle as well as from each intersection with coordinate lines in early layers, intersects with the arrangement of hyperplanes representing the preceding layer. The 1-d intersections are then used as seed points for new spans that intersect next preceding layer etc. The 2d intersections together with selective edges from the spans generate linking patches that ensures the continuity of the input manifold. as It should be pointed out that these manifold do not necessarily correspond to actual class manifolds since we are not considering the complete network with fully connected layers. They can however be considered as more elaborate and specific building blocks in order to construct the actual class manifolds of a trained network. We have defined a formal procedure for computing preimages of deep linear transformation networks with ReLU non linearities using the dual basis extracted from the set of hyperplanes representing the transformation. Specialising to convolutional networks we demonstrate that the complexity and the symmetry of the arrangement of corresponding hyperplanes is substantially reduced and we show that these arrangements can be modelled closely with multidimensisional regular polyhedral cones around the identity line in input space. We point out the crucial property of nested cones which guarantees efficient contraction of data to lower dimensions and argue that this property could be relevant in the design of real networks. By increasing the number of layers to shape input manifolds in the form of preimages we can retain the nested cone property that most efficiently exploits network data in order to construct input manifolds that comply with manifolds corresponding to real classes and would explain the success of ever deeper networks for deep learning. The retaining of the nested cone property can be expressed as a limitation of the degrees of freedom of multidimensional rotation of the cones. Since convolutional networks essentially always have limited spatial support convolutions, this is to a high degree built in to existing systems. The desire to retain the property of nesting could however act as an extra constraint to further reduce the complexity of the convolutions. This of course means that the degrees of freedom are reduced for a network which could act as a regularization constraint and potentially explain the puzzling efficiency of generalisation of deep networks in spite of a high number of parameters. We demonstrate that it is in principle possible to compute non linear input manifolds that map to affine output manifolds. This demonstrates the possibility of deep convolutional networks to achieve flattening of input data which is generally considered as an important preprocessing step for classification. Since we do not consider a complete network with fully connected layers at the end we cannot give details how classification is achieved. The explicit demonstration of non linear manifolds that map to affine outputs however indicates a possible basic structure of input manifolds for classes. It is easy to see that a parallel translation of the affine output manifold would in two linearly separable manifolds that would be generated by essentially parallel translated non linear manifolds in the input space. This demonstrates that convolutional networks can be designed to exactly separate sufficiently "covariant " classes. and that this could be the reason for the relative success of convolutional networks over previous machine learning approaches to classification and explain why using a large number of classes for training is advantageous since they all contribute to very similar individual manifolds. Disregarding these speculations the fact remains that these manifolds will always exist since they are derived on purely formal grounds from the structure of the network. If they have no role in classification their presence will have to be explained in other ways.

Analysis of deep convolutional networks in terms of associated arrangement of hyperplanes

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that neural networks operate by changing topologly of a data set and explore how architectural choices effect this change.